Unit 15: Bias, representation and accessibility

Reading time: about 8 minutes. Activity: about 20 minutes.

Checking without an oracle, one more time

Unit 9 ran the module's method on a case with no computable answer, and the reference was what the sources said. This unit runs it on a harder object still, where the reference is not a document at all.

The question is not "is this output biased", which invites a yes-or-no answer to something that does not have one. It is the question this module always asks:

What would I compare this against, and what would that comparison establish?

That has real answers here. They are less comfortable than a value in a table, and they are the same kind of work.

Where the distributions come from

A model's output reflects the statistical patterns of its training data, shaped further by post-training (Unit 3). Two consequences, stated at the level this module can defend.

What is common in the data tends to be common in the output. Ask for "a nurse" and "a surgeon" and the outputs will tend towards whatever was common in the material — in pronouns, in names, in described appearance. "Tends" is doing real work in that sentence: post-training, system instructions and filtering all sit between the training distribution and what you see, and any of them can damp a pattern or introduce one. So the training data is where the tendency comes from and not a formula for the output. Nothing in the system decided this, and describing it as a decision would be the anthropomorphism the module rules out.

What is rare in the data is thin in the output. This is the less-discussed half and often the more consequential. Material about smaller languages, smaller places, minority practices and less-documented conditions is thinner, so output about them is more likely to be generic, out of date, or wrong in ways that are harder to catch, because there is also less to check it against.

Those two are why "bias" is a poor single word for it. One is a skew you can sometimes see in a single output. The other is an absence, and absences do not announce themselves.

Three places it shows up in your work

In summaries. A summary reflects what is prominent in what it summarised. If you ask for "the main criticisms of X" and one strand of criticism is well-represented online and another is not, the shape of the answer is a fact about the corpus rather than about the criticisms.

In examples. Generated examples, names, scenarios and case studies carry the distribution. In teaching material, recruitment material or anything with an audience, that becomes a fact about who sees themselves in your work.

In descriptions of people. Anything that classifies, ranks, or predicts something about people is the highest-consequence case, and Unit 12's escalation row usually applies.

What a check looks like

You cannot compare against ground truth, because there is no table of what the answer should have been. There are three things you can do, and they establish different things.

1. Vary one thing. Produce the same output with a single attribute changed — a name, a pronoun, a place, a language, and compare. This is the closest thing to a controlled comparison available, and it is the one to reach for.

It is worth being exact about what one such pair shows, because the obvious reading of it is wrong. Two fresh conversations differ in two ways, not one: the attribute you changed, and the ordinary run-to-run variation of a system that does not produce the same words twice. So a difference between them is not yet evidence about the attribute. It is one observation with two candidate explanations.

What separates them is repetition. Run the pair several times, in both orders, and look at whether the difference persists. A feature that appears in four runs out of five on one side and once out of five on the other is telling you something; a feature that appears once is telling you the system varies.

What a repeated comparison establishes: that outputs differed in these specific ways, at this frequency, on this system, on this date. What it does not establish: that no difference exists elsewhere, that the size of the difference is stable, or that you have found the mechanism. You have found a pattern in some outputs, which is a real finding and a modest one.

2. Compare against a real distribution. Where a published statistic exists — who actually works in a profession, which languages are spoken in a place — you can compare the output's spread against it.

What it establishes: a specific, citable mismatch. What it does not establish is more subtle: a match is not a licence, because the real distribution may be the thing you were trying not to reproduce. A recruitment description matching current demographics may be exactly the problem.

3. Ask who is missing. Not a comparison but a prompt to yourself. Who is not in this output, and would I notice if they were absent? Absence is the failure mode that no comparison catches, because you cannot compare against something you did not think of.

Accessibility is the checkable part

Some of this is genuinely hard to establish. Accessibility mostly is not, and it is where a check produces a definite answer.

If you use generated material in anything others read, the ordinary requirements apply and are testable:

• Alt text that conveys what the image conveys for the purpose it is used for. Generated alt text is a description of an image by something that cannot know why you included it, so it tends to describe contents rather than what the image is doing in your document. A purely decorative image takes empty alt text, so a screen reader skips it — describing it is the wrong answer, not a thorough one. A chart needs the figure's point in the alt text and its numbers reachable some other way, in a table or a text summary.
• Headings that reflect the structure, without skipping levels where the structure does not skip. This is a check on whether the outline is real, not a rule that a jumped level is automatically a failure.
• Contrast that meets the standard, checkable with a tool in seconds.
• Meaningful link text — not "click here", which tells a screen-reader user nothing when links are listed out of context.
• Colour never used alone to carry meaning.
• Keyboard access, with a focus indicator you can see, in an order that matches the reading order.

The point is not that generated material is worse at this. It is that the requirements are checkable and often unchecked, and that unlike most of this unit you can settle them today, with a tool, for free.

This module runs those checks on itself. scripts/check_accessibility.py is in make test, and it fails the build, and its output ends with a list of the things a machine cannot decide, which is the honest shape for this whole subject.

What this unit is not saying

It is not saying generated material is uniquely biased. Human-written material carries distributions too, and the module makes no comparative claim. What is different is scale and invisibility: an output arrives without the provenance a human author's work usually carries.

It is not saying a varied output is a fair one. Varying an output changes the output. Whether the underlying task was appropriate is a separate question, and it is the one worth asking first when the task is about people.

It is not asking you to solve this. It is asking you to notice it, check what can be checked, and escalate what cannot, which is what the module asks everywhere.

Summary

• Output tends to reflect the distribution of its training data, as later training and system design allow: what was common tends to be common, and what was rare is thin. The second is harder to catch because absences do not announce themselves.
• It shows up in summaries, in examples, and most consequentially in anything describing or classifying people.
• Three checks: vary one attribute and compare over repeated runs, compare against a published distribution, and ask who is missing. Each establishes something narrow, and none establishes fairness. A single pair differs in the attribute and in ordinary run-to-run variation, so it settles nothing on its own.
• A match to the real distribution is not a licence — sometimes the real distribution is the problem.
• Accessibility is the part with definite answers. Check it, because you can.

Then answer cp15-what-a-check-shows and cp15-alt-text at the foot of this page.