Time: about 14 minutes. Read page.md first.
Three short experiments on the thing you cannot see. You need one document of your own that is longer than a page or two — a handbook, a long article, a report, a set of lecture notes.
Part 1 — the same question, two contexts (4 minutes)
Ask a question you know the answer to, twice: once at the start of a fresh conversation, and once at the end of a conversation you have already been having about something else.
Record both answers and write one line on how they differ. Length, format, confidence, assumptions carried over from the earlier topic.
Then answer this in a sentence:
If I had only seen the second answer, what would I have concluded about the system?
The point is not that one answer is better. It is that "the same question" described two different inputs, and nothing on screen said so.
Part 2 — the buried clause (6 minutes)
This is the experiment worth doing carefully.
- Take your document and find something stated late in it — the last quarter — that is specific and checkable. A number, a deadline, a condition, a named exception.
- Start a fresh conversation, attach the whole document, and ask a question that can only be answered from that late passage.
- Ask for the locator: the section or page, and the sentence it comes from.
- Go and look.
Record three things: what it said, whether the locator was real, and whether the quoted sentence is at that location and says what the answer said it says.
There are four outcomes and all of them are informative:
| What happened | What you have found |
|---|---|
| Correct answer, correct locator, sentence matches | The passage was in the context, and you now know how to confirm that cheaply |
| Correct answer, no locator offered | You have an answer you cannot check without reading the document. Ask again and insist |
| Confident answer, locator points somewhere else | The characteristic failure of this unit. Note whether anything in the wording warned you |
| The system says the document does not address it | The most useful outcome, and the rarest, because it is the one that required saying no |
If your document is short enough that everything fits, say so — that is a result too, and it tells you the bound is above the size you work with.
Part 3 — before and after (4 minutes)
Take the vaguest question you asked in Parts 1 or 2 and rewrite it the way the reading's example does: state what would count as correct, ask for locators, and say what to do about absence.
Ask it. Then write two or three sentences on what changed — not about which answer is better written, but about what you can now check that you could not check before.
Keep all three parts. Unit 7 asks you to record a prompt, and after this unit that means recording the whole input: fresh conversation or not, what was attached, what was retrieved.
Now take the checkpoints
cp05-what-reaches-the-system is about what the context actually contains.
cp05-locators is about Part 2's outcome.
python3 selfcheck.py run cp05-what-reaches-the-system
python3 selfcheck.py run cp05-locators # or in the browser
Run the Python route from course/lab/, or use the browser self-check.
Optional extensions
Optional. These sit outside the core study time, are not required for the checkpoints, and nothing later in the module depends on them.
Find the edge deliberately (about 8 minutes). Repeat Part 2 with progressively longer documents until the late-passage question starts failing. You will not get a clean threshold — systems handle overflow differently and the behaviour is not documented — and that is the finding. Record the sizes at which it worked and stopped working, and note that you now know something about your tool that its documentation does not tell you.
Test the format trap (about 4 minutes). Ask for an answer in a very specific format — a table with named columns — about something you can check, and see whether a perfectly formatted answer can be wrong. Compliance with a format is the cheapest kind of compliance to obtain and the easiest to mistake for care.
Marking guidance — open this once you have done the activity, to check your own work
Read this after attempting the activity.
Part 1 — what the difference means
The common finding is that the second answer carries something over: an assumption about your situation, a format from the earlier topic, a level of detail calibrated to a different question. That is the accumulated context doing what it does.
The question at the end of Part 1 is the one that transfers. If you had seen only the second answer, you would probably have concluded something about the system — that it is verbose, or that it assumes too much, or that it is good at this. All of those would have been conclusions about a system drawn from evidence about an input. This is a small instance of a large error, and it is worth having made once in a controlled way.
If the two answers were near-identical, that is a real result and not a failed experiment. A short earlier conversation on an unrelated topic may change little. Try again after a long conversation with strong assumptions in it.
Part 2 — the four outcomes
Nothing here is a pass or a fail. What is being tested is whether you can tell which outcome you got, which requires having gone and looked.
The third row is the one to sit with. When a confident answer cites a location that does not support it, look back at the wording and ask honestly whether anything in it warned you. Usually nothing did — and that is the unit's point rather than a complaint about the tool. The register of an answer generated from a partial document is the register of an answer generated from a whole one.
The fourth row deserves comment because it is rare. An answer that says "the document does not address this" has declined to fill a gap, and gap-filling is the more fluent option. If you got one, notice what you asked for: the reading's example prompt explicitly says what to do about absence, and requests that do not distinguish silence from absence tend to receive the more confident of the two.
If everything fitted, your document was inside the bound. Say so and move on. Reporting "I could not reproduce the failure at this size" is a finding with a condition attached, which is exactly the form this module wants.
Part 3 — what changed
The answer being looked for is checkability, not quality.
A good response says something like: before, confirming the answer meant reading the document; after, it means opening three named sections. That is the same claim the reading makes about the handbook example, arrived at on your own material.
A weaker response says the second answer was more detailed, or better organised, or more professional. Those are properties of the text. The second answer may also be wrong — the exercise does not establish otherwise — and if it is, you can now find out in two minutes.
The checkpoints
cp05-what-reaches-the-system tests the model of the context window. The
tempting wrong answers all treat the typed prompt as the input, which is the
assumption behind both "prompt engineering as magic words" and "it gave me a
different answer, so the model changed".
cp05-locators tests Part 2's judgement. The trap is treating a plausible
locator as a check. A section number is not evidence until you have opened the
section — it is a claim like any other in the answer, and it is the one claim in
the answer that is cheap to falsify.
Your private activity record
Browser storage is not a permanent copy
Progress is kept only in this browser, profile and device. Private browsing, clearing site data, removing the profile, a browser reset, storage eviction or device loss can erase it. Keep important answers and contributions separately.
These notes stay in this browser unless you download a backup or activity log.