Required work: about 16 minutes. Optional extensions at the end add more if you want them.
Ten things a text-generating system has produced. For each one, decide which kind of checking it needs before you could rely on it, and name the check you would actually run.
The three categories, from the reading:
- E — needs exact checking. There is a right answer you can compare against, by running it or by recomputing it against values you already know.
- S — needs source checking. It is a claim about the world. It cannot be executed, only looked up in the record.
- L — low-risk drafting. An error would be cheap and visible, and you were going to rework the text anyway.
Part 1 — sort them (7 minutes)
Copy the table and fill in the last two columns. One or two words for the category; one clause for the check.
| # | The output | E / S / L | The check I would run |
|---|---|---|---|
| 1 | A regular expression it wrote to validate UK postcodes in a form — a regular expression is a pattern a computer uses to test whether text has the right shape. | ||
| 2 | A statement that a named professional body has required a particular qualification since 2019. | ||
| 3 | A rewrite of your own lecture notes as a bulleted revision list. | ||
| 4 | A unit conversion in a lab report: 2.3 kg expressed in pounds. | ||
| 5 | A summary of a journal article you have not read. | ||
| 6 | Five candidate titles for your dissertation. | ||
| 7 | A database query that deletes every row older than a given date — an instruction to a database that permanently deletes records. | ||
| 8 | A claim that a piece of software behaves a certain way unless you change a setting — checkable in its manual or by trying it. | ||
| 9 | A rewritten opening paragraph for an essay you are still drafting. | ||
| 10 | A list of six references at the end of a generated literature review. |
Do not agonise. If an item feels like it sits in two categories, write both letters and move on — several of them genuinely do, and the answer key says which and why.
Part 2 — two follow-up questions (3 minutes)
A. What would stop a low-risk item being low-risk?
Take one item you marked L and write one condition that would move it out of that category — "if I were not going to reread it", or "if this went straight to a client". Low risk is a property of your situation, not of the text.
B. Which action is not independent verification?
For one item, write down something a person might do that feels like independent verification but is not. Examples: asking the system whether it is sure; regenerating and seeing whether you get the same thing; noticing that the output is very specific. Say what the action can reveal, and what it cannot establish.
Part 3 — one real item of your own (2 minutes)
Add an eleventh row: something you have personally used or are about to use an AI system for. Sort it, and name the check.
If your honest answer is "I would not have checked that", write that down too. That is more useful than an aspirational answer, and Unit 8 asks you to look back at it.
Part 4 — locate the system layer (4 minutes)
Write one layer beside each description: attention, post-training, retrieval, next-token scoring, or pre-training.
| Description | Layer |
|---|---|
| Combines information from relevant token positions in the available context. | |
| Shapes the base model into an assistant that follows instructions. | |
| Supplies external documents, whose relevance and support still need checking. | |
| Ranks possible continuations; its scores are not a truth predicate. | |
| Learns statistical patterns for next-token prediction from a large corpus. |
Check that your labels describe different functions. In particular, attention does not fetch a source, retrieval does not certify a source, and pre-training is not the complete deployed product.
Now take the checkpoints
cp02-risk-triage gives you three outputs and asks you to match each to the
kind of checking it needs — the same judgement you made in Parts 1 to 3.
cp02-system-layers checks the five distinctions from Part 4.
python3 selfcheck.py run cp02-risk-triage
python3 selfcheck.py run cp02-system-layers
Run the Python route from course/lab/, or use the browser self-check. Each
checkpoint names the specific mix-up behind the most likely wrong answers.
Keep your table. Units 4 and 5 hand you an output that spans two of these categories at once, and you will want to have made the distinction on easy cases first.
Optional extensions
Optional. These sit outside the core study time, are not required for the checkpoint, and nothing later in the module depends on them.
Finish Part 2A across the whole table (about 3 minutes). Part 2A asked for one L item. Do the rest: for every item you marked L, name the condition that would move it out of that category. The pattern that emerges is the useful part — the conditions are all about where the output goes next, never about the text.
Build the full list of false verification (about 4 minutes). Part 2B asked for one item. Write down as many actions as you can that feel like independent verification and are not, and for each say what it can reveal and what it cannot establish. The answer key lists six; getting to four unaided is a good result.
Rewrite the three weakest checks (about 5 minutes). Go back through your Part 1 table and find the three entries where you wrote something closer to an intention than a method — "check it carefully", "look at it again". Replace each with a check somebody else could run and get the same answer from. If nothing about a check could fail, it is not a check.
Marking guidance — open this once you have done the activity, to check your own work
Mark the reason, not the letter. A learner who writes S for item 1 and justifies it by "the postcode rules are published, so I would check the specification before testing anything" has understood more than one who writes E because code is usually E.
Part 1 — the ten items
| # | Intended | Why | The check |
|---|---|---|---|
| 1 | E (with S first) | A regular expression has a right answer relative to a specification — but the specification itself is a published claim about the world. | Look up the published postcode format, then test the expression against a list of addresses you know to be valid and a list you know to be invalid. |
| 2 | S | A claim about what an organisation requires, and since when. Nothing here can be executed. | Find the requirement on the body's own published material. A secondary summary is not the record. |
| 3 | L, conditionally | Low risk because you hold the original. The failure mode is silent omission: material dropped in the rewrite, then revised from. | Read the rewrite against your own notes once. If you would not do that, it is not low-risk. |
| 4 | E | A conversion with one correct value. | Recompute it yourself and compare. Check the direction of the conversion as well as the number. |
| 5 | S | A summary is a set of claims about a document. | Read the article. Note the awkward consequence: the only real check costs the labour the summary was meant to save. |
| 6 | L | Candidate titles are options you will choose between; a bad one is visible and free to discard. | None needed beyond reading them. |
| 7 | E, with the stakes raised | Correctness is exactly checkable, but the error is not recoverable once it has run. | Run it against a copy, or run the equivalent selection first and inspect what would have been deleted. Never test destructive operations on the live data. |
| 8 | S or E | Both routes are legitimate. The manual for the version you are using is the record; trying the software is a direct observation. | Whichever is cheaper — but check the version. A default setting that changed between versions is a common and quiet failure. |
| 9 | L | You are still drafting; an error is yours to fix on the next pass. | None, provided you actually reread it. |
| 10 | S, and the highest-value check here | Two separate failures: a reference may not exist, and an existing reference may not support the claim attached to it. | Look up each reference in a catalogue or database, then confirm that each source says what the review claims it says. |
Items 1, 3 and 8 are the productive ones to discuss. Item 8 in particular demonstrates that the categories are about how an error would reach you rather than about what kind of object the output is: the same claim about a default setting can be checked by reading the manual or by trying the software, and both are real verification.
Part 2A — what stops something being low-risk
The required work asks for one item marked L; covering the rest of them is an optional extension, so do not expect to see it. Good answers name a change in the learner's situation, not in the text:
- the output goes somewhere you will not reread it;
- it is passed to someone else who will assume it was checked;
- it becomes the basis of later work, so an error propagates instead of sitting still;
- the original you would have compared against is no longer available;
- the cost of being wrong shifts from your marks to someone else's decision.
Weak answers say "if it were more important". Importance is not the axis. Push for the mechanism by which the error would survive undetected.
Part 2B — actions that are not independent verification
The required work asks for one item, so expect one row rather than a list; compiling the full set is an optional extension. All six are worth naming aloud in feedback whichever way round it was done.
| Action | What it cannot establish |
|---|---|
| Asking the system whether it is sure | The revision may be useful, but it is generated by the same system and is not independent evidence. |
| Regenerating and comparing | Agreement or variation can be diagnostic, but neither establishes correspondence with the world. |
| The output being very specific | Specificity is a property of the text, produced whether or not the content is right. A fabricated page number looks exactly like a real one. |
| The output being long and well organised | Same argument. Formatting is generated, not earned. |
| The code running without error | Running is not the same as being correct. A routine can execute cleanly and return the wrong number at every index — Unit 5 gives you exactly that. |
| Recognising the answer as familiar | Familiarity tracks how often something is said, which is close to what the generation step optimises. It is the least reliable cue on this list. |
Why plausibility is not correctness
This is the point the activity exists to establish, and it is worth stating explicitly in feedback.
Plausibility and correctness are different properties. Plausibility is a property of the text: whether it reads like the sort of thing that follows. That is what the generation step is doing, and it does it well and uniformly. Correctness is a relation between the text and something outside it — a specification, a calculation, a document, a fact about the world. Nothing in the generation step evaluates that relation.
Two consequences follow, and they are the ones learners need.
First, surface quality is insufficient evidence. Fluency, specificity, and confident formatting can accompany correct or incorrect content. They may help a reader notice a poor answer, but they do not establish the underlying claim.
Second, verification needs evidence separable from the assertion. A revision can improve an answer [T9], but it does not provide the independence of a value you derived, a document you opened, or a test you ran. This is why the next three units build an independent reference before any generated code is looked at, rather than after.
For an instructor
Expected time: 16 minutes for the required work. Formative. The three optional extensions at the end of the activity add roughly 3, 4 and 5 minutes and are outside the core budget.
Strong responses get at least eight of the ten broadly right, name a check that could fail for each, and flag at least one item as legitimately spanning two categories.
What to watch for:
| Observation | Diagnosis | Response |
|---|---|---|
| Everything marked E or S; nothing L | The learner has heard "check everything" and produced a policy they will not keep. | The L category is not permission to be careless; it is what makes the other two affordable. Ask which items they would actually check next Tuesday. |
| Everything sorted by stakes | The most common error, and the one cp02-risk-triage targets. |
Item 7 versus item 5: both matter, and they need completely different checks. |
| "I would check it carefully" as the check | Intention offered as method. | Ask what a failing result would look like. If nothing could fail, it is not a check. |
| Item 5 marked E | A summary treated as something with a computable right answer. | It is a set of claims about a document. The only check is the document. |
| Anthropomorphic reasoning in the justifications ("it wouldn't lie about a date") | Intent attributed to the system. | Return to the mechanism: there is no step at which truth is evaluated, so there is nothing to be honest or dishonest about. |
Part 3 — the learner's own item — is the part worth reading if you read only one. It shows what they actually do rather than what they can classify, and a learner who honestly writes "I would not have checked that" has given you the most useful sentence in the exercise. Do not penalise it.
The checkpoint
This unit's checkpoints are cp02-risk-triage and cp02-system-layers. Their
prompts, options, correct matching and diagnostics are defined once, in the
checkpoint blocks at the end of this file; ../lab/checkpoints.py and the
browser self-check both read those blocks, so there is a single source and the
routes cannot drift apart.
It uses three items distinct from the ten above and tests the same judgement. The expected failure is a swap driven by sorting on apparent seriousness rather than on failure mode; the framework's feedback names that directly, so let learners read it rather than correcting them.
python3 selfcheck.py run cp02-risk-triage # or use the browser self-check
python3 selfcheck.py run cp02-system-layers
Your private activity record
Browser storage is not a permanent copy
Progress is kept only in this browser, profile and device. Private browsing, clearing site data, removing the profile, a browser reset, storage eviction or device loss can erase it. Keep important answers and contributions separately.
These notes stay in this browser unless you download a backup or activity log.