Unit 15: Bias, representation and accessibility

Reading time: about 9 minutes. Activity: about 20 minutes.

1. Checking without an oracle, one more time

Unit 9 ran the course's method on a case with no computable answer, and the reference was what the sources said. This unit runs it on a harder object still, where the reference is not a document at all.

The question is not "is this output biased", which invites a yes-or-no answer to something that does not have one. It is the question this course always asks:

What would I compare this against, and what would that comparison establish?

That has real answers here. They are less comfortable than a value in a table, and they are the same kind of work.

2. Where the distributions come from

A model's output reflects the statistical patterns of its training data, shaped further by post-training (Unit 3). Two consequences, stated at the level this course can defend.

What is common in the data tends to be common in the output. Ask for "a nurse" and "a surgeon" and the outputs will tend towards whatever was common in the material — in pronouns, in names, in described appearance. "Tends" is doing real work in that sentence. Post-training, standing instructions (the instructions a product adds, which you do not normally see) and filtering all sit between the training distribution and what you see. Any of them can damp (weaken) a pattern or introduce one. So the training data is where the tendency comes from, not a formula for the output. Nothing in the system decided this. Describing it as a decision would be anthropomorphism, treating the system as a person making a choice, which the course rules out.

Image generators show the same tendency, and more visibly. In a 2023 study, prompts that named no group at all produced stereotyped pictures: 99% of one widely used generator's "software developer" images showed people represented as white, against 56% of software developers in US statistics, and asking explicitly for the opposite did not reliably undo it [V6]. Those were 2022 systems; the point is the mechanism, not the figure, and it is checkable in the same way as text: vary one thing and look at many outputs.

What is rare in the data is thin in the output. This is the less-discussed half and often the more consequential. Material about smaller languages, smaller places, minority practices and less-documented conditions is thinner, so output about them is more likely to be generic, out of date, or wrong in ways that are harder to catch, because there is also less to check it against.

Those two are why "bias" is a poor single word for it. One is a skew you can sometimes see in a single output. The other is an absence, and absences do not announce themselves.

3. Three places it shows up in your work

In summaries. A summary reflects what is prominent in what it summarised. If you ask for "the main criticisms of X" and one strand of criticism is well-represented online and another is not, the shape of the answer is a fact about the corpus rather than about the criticisms.

In examples. Generated examples, names, scenarios and case studies carry the distribution. In teaching material, recruitment material or anything with an audience, that becomes a fact about who sees themselves in your work.

In descriptions of people. Anything that classifies, ranks, or predicts something about people is the highest-consequence case, and Unit 12's advice to escalate usually applies.

4. What a check looks like

You cannot compare against ground truth, because there is no table of what the answer should have been. There are three things you can do, and they establish different things. The first two are comparisons, and they can catch the skew; the third is for the absence, which needs a different kind of check.

1. Vary one thing. Produce the same output with a single attribute changed — a name, a pronoun, a place, a language, and compare. This is the closest thing to a controlled comparison available, and it is the one to reach for. Software testers call this metamorphic testing: say how the outputs should relate when one input changes, then try many cases, looking for one where that fails [E20, sec. 2.2].

It is worth being exact about what one such pair shows, because the obvious reading of it is wrong. Two fresh conversations differ in two ways, not one: the attribute you changed, and the ordinary run-to-run variation of a system that does not produce the same words twice. So a difference between them is not yet evidence about the attribute. It is one observation with two candidate explanations.

What separates them is repetition. Run the pair several times, in both orders, and look at whether the difference persists. A feature that appears in four runs out of five on one side and once out of five on the other is worth following up with more runs; a feature that appears once is telling you the system varies.

Take three pairs in alternating order. One feature appeared in every run on one side and in none on the other; a second appeared once on each side, which is the system varying. What that establishes: a pattern in these outputs, at this frequency, on this system, on this date. What it does not establish: a rate, a harm, the mechanism, that the size of the difference is stable, or that no difference exists anywhere else. Six runs are six runs, and with so few even that pattern could be chance; more runs would test it. It is a real finding and a modest one. And the opposite mistake, saying there is no problem, is the easier one to make without noticing: "the difference was small, so there is no problem" is a general claim from one comparison, exactly as much as the accusing version is.

2. Compare against a real distribution. Where a published statistic exists — who actually works in a profession, which languages are spoken in a place — you can compare the output's spread against it.

What it establishes: a specific, citable mismatch. What it does not establish is more subtle: a match is not a licence, because the real distribution may be the thing you were trying not to reproduce. A recruitment description matching current demographics may be exactly the problem.

3. Ask who is missing. Both checks so far compared the output with something: the same prompt with one attribute changed, or a published figure. This one is for the other half, the absence, and it starts differently, as a prompt to yourself. Who is not in this output, and would I have noticed if I had not been asked? Absences are easy to miss when your reference omits the same groups, so where you can, also compare against a list of relevant groups assembled independently of the output.

5. Accessibility is the checkable part

Some of this is genuinely hard to establish. Accessibility is easier: part of it a tool settles with a definite answer, and the rest is a judgement you can still make carefully.

If you use generated material in anything others read, the ordinary requirements apply and are testable:

• Alt text that conveys what the image conveys for the purpose it is used for. Alt text is a short written description attached to an image. A screen reader reads it aloud for someone who cannot see the image, and a browser shows it when the image does not load. It matters most to blind and low-vision readers. It also serves search engines and anyone on a slow connection. WCAG 2.2, the main web accessibility standard, asks for a text alternative "that serves the equivalent purpose" [A1, SC 1.1.1]. So a good one describes function: what the image is for in that place, not just what it looks like.

Generated alt text is produced from what is in the request. The image is there; your reason for including it is not. So it tends to describe contents rather than what the image is doing in your document.

A purely decorative image takes empty alt text, so a screen reader skips it. Describing it is the wrong answer, not a thorough one. A chart needs the figure's point in the alt text, and its numbers somewhere else, in a table or a text summary. - Headings that reflect the structure, without skipping levels where the structure does not skip. This is a check on whether the outline is real, not a rule that a jumped level is automatically a failure. - Contrast that meets the standard, checkable with a tool in seconds. - Meaningful link text — not "click here", which tells a screen-reader user nothing when links are listed out of context. - Colour never used alone to carry meaning. - Keyboard access, with a focus indicator you can see, in an order that matches the reading order.

The point is not that generated material is worse at this. It is that the requirements are checkable and often unchecked. Some a tool settles in seconds, for free: a contrast ratio, a missing alt attribute, a skipped heading level. Others need your judgement: whether the alt text serves the image's purpose, whether the headings reflect the real structure, whether the focus order matches the reading order.

This course runs those checks on itself. scripts/check_accessibility.py is in make test, the command that runs the course's own tests; a failure stops the build, and its output ends with a list of the things a machine cannot decide, which is the honest shape for this whole subject. Run on 2 November 2026, it checked 121 pages, 64 animations and 65 contrast pairs, passed, and listed six checks still to be done by a human, the first being to read every alt text against what is on screen, because a script can confirm that alt text is present but not that it is accurate.

6. What this unit is not saying

It is not saying generated material is uniquely biased. Human-written material carries distributions too, and the course makes no comparative claim: biased, unrepresentative and inaccessible material long predates AI, in textbooks, stock images and forms. What is different is scale and invisibility: generation reproduces and amplifies what it was trained on at a speed and volume no human author matches, and its output often arrives without provenance, as much human material also does.

It is not saying a varied output is a fair one. Varying an output changes the output. Whether the underlying task was appropriate is a separate question, and it is the one worth asking first when the task is about people.

It is not asking you to solve this. It is asking you to notice it, check what can be checked, and escalate what cannot, which is what the course asks everywhere.

Summary

• Output tends to reflect the distribution of its training data, as later training and system design allow: what was common tends to be common, and what was rare is thin. The second is harder to catch because absences do not announce themselves.
• It shows up in summaries, in examples, and most consequentially in anything describing or classifying people.
• Three checks: vary one attribute and compare over repeated runs, compare against a published distribution, and ask who is missing. Each establishes something narrow, and none establishes fairness. A single pair differs in the attribute and in ordinary run-to-run variation, so it settles nothing on its own.
• A match to the real distribution is not a licence — sometimes the real distribution is the problem.
• Accessibility is the part with the most definite answers. A tool settles contrast and some structure; whether alt text serves its purpose needs your judgement. Check both, because you can.

Then answer cp15-what-a-check-shows and cp15-alt-text at the foot of this page.