Unit 10 video transcript: Re-prompting, specification, and overreliance

Release candidate. The unit videos are added in the first release.

The words spoken in the Unit 10 video, scene by scene, as its captions show them, with what the animations show marked [On screen: ...] where they play. Each speaker is named where the speaker changes. The Unit 10 reading covers the same material.

The narration is spoken by synthetic voices made for this course, each shown on screen as a drawn robot. No person is speaking.

Unit 10 — Re-prompting, specification, and overreliance

Narrator: A widespread idea holds that better results come from better wording — that somewhere a phrase unlocks the correct answer. This unit argues something less exciting and more useful.

The claim

Better prompting is precise requirements work — the same skill as writing a specification, and if your route took you through Units 7 and 8, you have already been doing it. Every time the checker named a failure mode in Unit 8, it told you which requirement you had not written down.

The vague prompt, and what it left open

If your route skipped Units 6 to 8, here is what you need. The Bernoulli numbers are a fixed sequence of fractions: B-zero, B-one, B-two, and so on. The number after B is the index, the position in the sequence. Two published conventions differ only at B-one. Every odd index above one has the value zero. And Lovelace's Note G labelled only the non-zero values. On the routes that include it, Unit 7 started with this request. "Write a Python function that returns Bernoulli numbers." Everything in that sentence is true, and it leaves the decisions open. Which Bernoulli convention? Which indexing scheme — modern, or the nineteenth-century one Lovelace used? What type comes back? What happens at index zero? Are the zero terms returned, or skipped? Nothing here settles it, so the output does, according to patterns in what the system was trained and tuned on. The result is plausible. Whether it is correct is not a question the prompt asked.

The failure modes were the missing requirements

Now put the diagnoses from Unit 8 beside that prompt: the codes its checker returns, one for each way a generated routine went wrong.

Checker: b1-sign says: you never stated the convention. ada-indexing says: you never stated the indexing scheme. not-fraction says: you never stated the type of value to return. raises says: you never said which inputs it must accept. odd-terms says: you never said the odd indices above one must return exactly zero. no-function says: you never gave the function's name, and what it takes and returns. off-by-one says: you never named a value at an index. import-error says: you never said the file must load cleanly.

Questioner: Could better wording have fixed any of those?

Narrator: Not one of those is a wording problem. Each one points at a requirement you had not stated.

The interface, quoted exactly

Questioner: What does the precise prompt actually add?

Narrator: So the precise prompt is not a cleverer sentence. It is the same request with the decisions written down, as nine clauses. They cover the interface, the convention and the indexing, the inputs it accepts and the zero terms, the arithmetic and some required values, how the file loads, and comments. The prompt is on this unit's page in full, about two hundred words long. The clause that matters most is the interface, quoted exactly: bernoulli, taking n as an int, returning a fractions.Fraction.

Checker: That is the same line that appears in the checker's contract.

Narrator: Not every clause can be enforced by the checker. Named anchor values can be met by a lookup table, which is why it compares thirty-one indices. Exact arithmetic inside the routine is invisible in its outputs, and so is a side effect on import. The error a negative index should raise lies outside the range it tries. And no code in the lab reads a comment. A specification precise enough to test against is precise enough to prompt with — and if you cannot write the prompt, that is because you did not yet know what you wanted.

What the checker settles

Questioner: If a routine passes the checker, is it the right routine?

Narrator: The checker verifies. It shows that a routine meets the contract, over the range it tested. No checker can decide, for every program it might be given, what that program does on all its inputs; to decide means always to give the correct yes or no. That is Rice's theorem, from nineteen fifty-three. In plain terms, no general tool can answer every interesting question about what programs do. The best-known example is the halting problem: whether a program ever stops, or runs for ever. In the nineteen-thirties, Turing proved that no program can answer a very close question for every program. The halting problem cannot be answered in general either. If either result is new to you, the glossary entry for decidable sets out both, and Unit 18, on the complete route, returns to them. This lab tests a finite range. A particular program can sometimes be proved correct for every input, but that proof is about that program: a certificate speaks for one artefact. Whether minus one half was the right thing to specify is a different question: validation. No check against the contract can answer it, because the contract is what the check compares with. You validate by using the numbers for what they are for. Unit 6 does it with sums of powers. The same formula gives a different sum under each convention, and the sum you need settles which one to specify. That example is on the routes with the mathematics.

Most clauses are bugs that already happened

Read the six clauses on this slide again: each is in the precise prompt. Ask why each one is there. Most of them are there because something can go wrong. The failures are the lab's examples, written for the lab to show each one.

[On screen: Headed 'Most clauses are bugs that already happened', the bare prompt 'Write a Python function that returns Bernoulli numbers.' gains six clauses one at a time: exact arithmetic with fractions.Fraction; the convention B1 = -1/2; modern indexing, every index numbered; zero terms, Fraction(0) for odd n > 1; anchor values at named indices, B0 to B30; and the domain, defined for every n >= 0. Beside them, under 'Failure modes these clauses address', each clause strikes out its failure mode, not-fraction, b1-sign, ada-indexing, odd-terms, off-by-one and raises, which is then marked a spec violation. The fully constrained prompt leads to a testable candidate, which is then passed to independent checks to yield pass or diagnostic failure.]

The exact-arithmetic clause exists because a routine can come back in floating point.

The convention clause exists because one can come back correct under the second convention.

The indexing clause exists because one can come back numbering only the non-zero terms, faithfully reproducing Note G.

The zero-terms clause exists because one can come back with minus one over thirty at index three.

The anchor-values clause exists because a routine can come back shifted by one index: naming the values at fixed indices pins the sequence in place.

And the domain clause was written before its bug: no supplied example raises at index zero, but a routine that did would fail the specification. So, of the six pairs on the screen, five answer a failure the examples show. One answers a failure that was foreseen. Many lasting requirements are lessons written down after earlier failures. That is why this is engineering, not magic. You are not searching for magic words. You are writing down a requirement each time reality teaches you one. Requirements come from elsewhere too: what a user needs, what the law or a standard demands, what a hazard analysis turns up. Those clauses are just as real, even before anything has gone wrong.

And what comes back is still a candidate, to be checked independently. On the slide is the checkpoint, cp10-spec-clauses: it has you match clauses like these to the failure each one prevents.

Overreliance

Now the uncomfortable part — the reason Unit 6 came before Unit 7. Overreliance is not using AI. It is relying on generated output past the point where you can check it. And the same goes for any source or tool you rely on past that point: a calculator, a search result, a colleague's answer, a reference book. In this course the source is generated output; the point holds for all of them. Pause when you cannot predict relevant behaviour, explain the transformation, name an independent check, or interpret a failed check. Confidence is not evidence that the gap has disappeared.

Why Unit 6 came first

On the routes with the mathematics, Unit 6 has you work out B-four by hand: a few fractions, no calculator, and out comes minus one over thirty. That was slow and looked like a detour. It was not. It is the reason that if a routine returns plus one over thirty, you know something is wrong rather than assuming you have misremembered. Without that derivation, or some other reference independent of the output, minus one over thirty and one over thirty are two equally plausible strings. Every tool in this course — the oracle, the checker, the property test — becomes a judgement only if someone can read the result.

A correct answer is not learning

Overreliance has a second form, and it matters most while you are still learning. Checking an answer tells you whether it is right. It does not tell you whether you can now do the task, and the two come apart.

[On screen: Headed 'Practice up, exam down', with the label 'One field experiment, school students [E21]'. Three rows of students: with a standard chat assistant, with a hint-giving version (hints, not answers), and a group that never had it. In the practice column an arrow rises on the standard-assistant row, marked 'up, sharply'; no practice figure is given. The exam column then becomes 'Exam, assistant taken away', and the assistant and the hints are struck out. Exam bars grow: the standard-assistant bar stops short of the comparison group's, and the gap is marked '17% lower'. A dashed line from the end of the comparison bar shows what it is compared with. Last, the hint-giving group's bar grows almost to the line, marked 'largely avoided the drop'. A note says the bar lengths are schematic and the only figure shown is the study's 17%.]

In one field experiment, school students practised mathematics with a standard chat assistant. Practice went up, sharply. Exam down: once the assistant was taken away, they scored 17% lower than students who had never had it. A version set up to give hints, not answers, largely avoided the drop. That was one setting, and research on how these tools affect learning is still early. What it shows is why assisted and unaided performance need separate checks. This course's answer is its own order: attempt first, then use the system to check or challenge your attempt, not in place of it. Nearly all the UK undergraduates surveyed in December 2025 said they use it for assessed work, so for them the question is how, not whether. The test is short. Put the system away and do a comparable case yourself. If you cannot, the answer was right and the learning has not happened yet.

The honest position

Questioner: So should I stop using these systems?

Narrator: This is not an argument against using these systems: the precise prompt plus the oracle is genuinely faster than writing the routine from scratch. Two things do not transfer. First, responsibility: the work is yours to answer for, whatever produced it. Second, judgement: someone has to know what correct looks like, and in your work that someone is you.

And note the luxury

Note the luxury you have had. This problem has an oracle. Most questions you will put to an AI system do not. Without one, the specification work is still what you have: write down what you need, ask for evidence you can check, check against independent sources, and run a second implementation.

[On screen: Headed 'Asking the same system whether it is sure', with the note 'Illustrative exchange; the finding is from a study published in 2024 [L5]'. An answer reads B4 = -1/30, marked correct. The user replies 'Are you sure? I got 1/30.', labelled 'same system, asked again: not a second implementation'. On a line from -1/30 (correct) to 1/30 (the user's), a marker for the answer slides towards the user's end, labelled 'shifts towards what the user appears to believe', and a new answer appears, B4 = 1/30, marked 'correct answer abandoned'. The caption reads 'It tests pressure, not the answer.' Then a card for a second, independent version arrives, reading -1/30 and marked 'matches the first', and a dashed line joins it to the first answer, which dims while abandoned and brightens again: 'Instead: a second, independent version, compared.' No rate or percentage is shown.]

Asking the same system whether it is sure is not a second implementation. In a study published in twenty twenty-four, assistants trained partly on people's ratings shifted towards what the user appeared to believe. They were assistants from twenty twenty-three. That question alone could turn a correct answer into an incorrect one. It tests pressure, not the answer. Where you cannot check a value, check a relation: reword the question or reorder the sources, and claims the sources settle should not change. Be explicit about what remains unverified. And know the point at which you stop and seek qualified human review.

Your turn

Take your Unit 7 prompt, or the vague prompt if your route skipped Unit 7, annotate which clause removes which failure mode, then run the precise prompt and compare the two results. Then apply the same checking habit to three short cases without an exact oracle, and finish at cp10-spec-clauses.

Closing, 20 seconds, no speech. A candidate record on the left, a check against an independent reference in the middle, and a bracketed claim on the right fade in side by side. Below them: Continue with the course material, ai.qyber.dev/edu-ai-literacy-ada-verification/, Course playlist: youtube.com/playlist?list=PLPsBWTfgvakM, All links are also in the description, and Keep your record. Check the next claim.