Unit 8 checks one artefact against one reference and gets a decidable answer: your routine agrees with the oracle, or the first index where it does not is named. That is a check, and where a check is available it is the strongest thing you can do.
Some questions do not have that shape. How often does this go wrong? Is the new prompt better than the old one? What kinds of error does it make? Is it good enough to use? None of those is settled by one comparison. They need an evaluation: a sample, a measure, a protocol, and a result that comes with an honest statement of how much it can bear.
Unit 8 teaches the six moves an evaluation is made of. This page is what sits behind them: which approach suits which question, the established methods on each side, and the handful of mistakes that recur so reliably they are worth recognising before you meet them. It is an overview and does not pretend to be more — each section names the literature to go to if you are going to do the thing properly.
The principle that carries over
Unit 8 names the external-reference principle: a check has to come from a reference the thing being checked does not control. Everything below is that principle under load.
Scale is where it gets hard to honour, because at scale it is tempting to let the thing under test supply the machinery that tests it — the examples, the measure, or the judgement. If a system generates the test cases it is scored on, scores itself, or is tuned until the number improves, you have a number and not a finding. That is one error wearing several costumes, and most of the rest of this page is a list of the costumes.
Which question you are asking
| your question | what you need |
|---|---|
| Is this particular answer right? | A check. Unit 8. No evaluation, no statistics. |
| How often is this kind of answer right? | Quantitative evaluation on a sample. |
| Is A better than B? | A comparison with a decision rule fixed before you look. |
| What goes wrong, and in what ways? | Qualitative evaluation: coding, not counting. |
| Is it good enough to use for this? | A threshold, written down before the run, and a named person who decides. |
The last row is the one people skip. "Good enough" is not a property of a system; it is a judgement about a use, and it needs a number chosen in advance by somebody willing to be held to it. Unit 12 is where that judgement lives.
The protocol, in six moves
Unit 8 states these; here they are with the reasoning. This is the shape shared by most serious evaluation, quantitative or qualitative, and the order matters more than any individual step.
1. State the question so that an answer could disappoint you. "Does the model do well at summarising?" cannot disappoint anyone. "Do at least 90% of generated summaries of these 50 papers avoid stating a finding the paper does not contain?" can. If no plausible result would change what you do, do not run the evaluation.
2. Choose the measure before you see any output. A measure chosen after looking is a measure chosen to give the result you already saw. And note what a measure becomes once people know it is the measure: it stops describing the thing and starts being the target [E1]. Ask directly whether your measure measures what you care about, or a proxy that happens to be easy to compute.
3. Pilot small. Five to ten items, run end to end. The pilot exists to find out that your instrument is broken, not to find out the answer. You will discover that two of your categories overlap, that the measure cannot be scored consistently, that a third of the sample does not apply. Fix those. Then discard the pilot data — it was collected under a design that no longer exists.
4. Freeze. Write down, before the full run: the question, the measure, the sample and how it was selected, the threshold or decision rule, the stopping rule, and what counts as a failure. Date it and put it under version control. This is preregistration, and it is the single cheapest thing on this page; posting the plan to an independent registry is the usual and stronger form [E2, abstract]. Its entire power is that it makes it impossible to move the goalposts without leaving a record — including impossible for you, later, honestly, without noticing.
5. Run the full evaluation without adjusting anything. If you must change something, you have started a new evaluation: say so, and report both.
6. Score, then report. Score blind to condition where you can — a rater who knows which output came from the new prompt is not a neutral instrument. Report the result, the uncertainty, the sample size, and the thing you would have accepted as the opposite finding.
Unit 8 gives this course's own example of move 2 going wrong: a measure that was accurate about motion and brightness and said nothing about the brief it was supposed to be judging. The general form is worth stating on its own. A measure can be perfectly correct and still be measuring the wrong thing, and nothing in the number tells you which case you are in. Only the written brief does, which is why it is written first.
Quantitative evaluation
Use it when you already know what counts as failure and want to know how often it happens, or which of two options fails less.
- Have a baseline, always. The trivial one: do nothing, always answer the most common class, keep the previous version. An accuracy figure with no baseline beside it is uninterpretable, and a surprising number of impressive numbers do not beat a trivial baseline.
- Report an effect size and an interval, not just whether something reached significance [E3, abstract]. "Significant" answers a question about noise, not about whether the difference is big enough to care about. A tiny improvement can be statistically real and practically irrelevant.
- Small samples are not just imprecise; selecting on significance inflates what they report. A small sample reaches significance only when its estimate happens to be large, so an underpowered study that finds an effect has probably overestimated it, and the same design run again often finds nothing [E4, abstract], [E5, p. 697]. If you can only afford twenty items, say what twenty items can and cannot show.
- Test many things and some will look significant by chance. If you compare eight prompts, expect one to look good for no reason. Decide in advance which comparison is the one, or correct for the others.
- The system's own variance is part of the measurement. Generative systems are not deterministic by default: temperature, sampling and seeds all move the output. Run each condition more than once, record the settings and the seed, and report the spread and not only the best run.
- Hold data out, and know where it has been. Anything the system was tuned on cannot measure it. In machine-learning work this leaks in subtle ways — duplicates across splits, preprocessing applied before splitting, choosing a model on the test set — and it is a documented cause of results that fail to replicate [E6, "Toward a solution: A taxonomy of data leakage"]. For a widely published benchmark, assume the answers may already be in the training data unless you know otherwise.
Qualitative evaluation
Use it when you do not yet know what the failure modes are, or when the question is about meaning, quality or effect on people rather than about a count. It is not the soft option; done properly it is more laborious than counting.
The list below is guidance for structured content analysis and coding-reliability approaches. Reflexive thematic analysis works differently: it treats a codebook, consensus coding and inter-rater reliability as procedures that do not belong to it, so go to its own quality criteria if you use it [E23, question 12].
- A rubric, with worked examples. Each level needs an example of what it looks like, or two raters will not mean the same thing by "adequate".
- A coding scheme, developed on the pilot and then fixed. Categories should be exhaustive and, where possible, not overlapping [E7].
- Two independent raters, at least on a subset. One rater's judgement is one person's opinion with a table around it.
- Measure agreement, do not assume it. Raw percentage agreement flatters itself when one category dominates; Cohen's κ and Krippendorff's α correct for agreement by chance [E8], [E7]. Report the figure.
- Adjudicate disagreements; do not average them. Where raters disagree, the disagreement is data — it usually means the rubric is ambiguous at exactly that point. Resolve it by discussion, record the resolution, and say how many there were.
- Sample until nothing new appears (saturation, a grounded-theory notion that reflexive thematic analysis does not assume [E23, question 7]), then go looking specifically for the case that would break your account. An evaluation that only collects confirming examples has demonstrated the diligence of its collector.
Thematic analysis, content analysis and grounded-theory coding are the established named methods here [E9], [E7]; each has a literature and a set of rules about what it licenses you to say.
Mixed, and in which order
Qualitative first, then quantitative, is usually right when the failure modes are unknown: read fifty outputs closely, let the categories emerge, fix the scheme, then count on a fresh sample. Counting first and interpreting afterwards means counting categories you invented after seeing the data.
The reverse order is right when the categories are already settled — you know the nine diagnostic codes and want to know which is most common, then read the examples of the commonest to understand why.
The standard software tests
Relevant if you are on the Python route, and the place where verification can be exact: where an exact answer exists, you can check rather than evaluate.
| approach | what it is | where it suits |
|---|---|---|
| Unit tests | Fixed inputs, expected outputs, asserted [E10]. | Anything with a decidable answer you can write down. |
| Property-based tests | Assert a property over many generated inputs rather than one case [E11]. | Invariants: "odd-index Bernoulli numbers above 1 are zero", "sorting twice changes nothing". |
| Differential testing | Two independent implementations, same inputs, compare outputs [E12, p. 101]; the first divergence localises the fault. | Exactly what check.py does to your routine in Unit 8. Strongest form: two languages, two authors. |
| Metamorphic testing | No oracle; instead assert relations between outputs — this input should give twice that output [E13, sec. 3.3]. | Where no correct answer is available to compare against, which is most generative work. |
| Regression and golden tests | Freeze current output; fail when it changes. | Guarding behaviour you have already established. Says nothing about whether it was right to begin with. |
| Mutation testing | Deliberately break the code; a test suite that still passes is not testing much [E14]. | Judging your tests rather than your code. |
| Coverage | Which lines the tests execute. | A floor, not a ceiling: 100% coverage of assertions that assert nothing is 100% coverage. |
The optional extension ex05_write_a_test.py is mutation testing by hand — four
deliberately faulty implementations, and your test has to separate them from the
correct one without rejecting it.
Where evaluations go wrong
Almost every failure below is a version of letting the thing under test certify itself.
- Tuning on the test set. You try six prompts, keep the one that scored best, and report its score. That score is now an estimate of how well you chose, not of how well the prompt works. You need fresh data.
- Letting the system judge itself. Using a generative system to score generative output is convenient and widely done, and it carries known biases — towards whichever answer came first and towards longer answers, and possibly towards its own outputs (a 2023 study) [E15, sec. 3.3]. It can be a useful screen. It is not an external reference, and it must not be the only judge on anything that matters.
- Cherry-picking the run. Five attempts, one good screenshot. If you ran it five times, the finding is about five runs.
- Moving the threshold after seeing the score. This is why move 4 exists.
- No baseline, so nothing to compare against.
- Ceiling and floor effects. If everything scores 98%, your measure has stopped discriminating and the differences you are reading are noise.
- The demonstration mistaken for evidence. One impressive example is a hypothesis. It is the reason to run an evaluation, not the result of one.
The minimum honest report
Even a small evaluation should record enough for someone else to disagree with it on the evidence:
- the question, and when it was fixed;
- what was evaluated, at which version, with which settings and seeds;
- how the sample was selected, and how large it is;
- the measure or rubric, and when it was fixed relative to seeing output;
- who scored, whether they were blind to condition, and the agreement between raters if there were more than one;
- the result, with its uncertainty;
- what was changed mid-run, if anything;
- what the evaluation does not establish.
That last line is the one that makes the rest trustworthy, and it is the same move Unit 17 asks of you when you state what your evidence supports.
Further reading
The references are grouped under Evaluation and experimental method in References and source policy. Each entry says what it is for, so that this page can be checked and so that you know which literature to enter for the part you need. [E10], [E11], [E12], [E13] and [E14] are the software-testing side; [E2], [E3], [E4] and [E5] the quantitative-protocol side; [E7], [E8] and [E9] the qualitative side; and [E16] the standard treatment of experimental and quasi-experimental design, which is where to start if you are designing a study involving people.