Unit 7 video transcript: Ask AI to generate the routine | AI Literacy: Ada, AI and Verification

Release candidate. The unit videos are added in the first release.

The words spoken in the Unit 7 video, scene by scene, as its captions show them, with what the animations show marked [On screen: ...] where they play. Each speaker is named where the speaker changes. The Unit 7 reading covers the same material.

The narration is spoken by synthetic voices made for this course, each shown on screen as a drawn robot. No person is speaking.

Unit 7 — Ask AI to generate the routine

Narrator: You now know what a Bernoulli number is. You have derived one yourself. You know there are two conventions and that Lovelace numbered things differently from us. In this unit you ask an AI system to write the routine that computes them, with the kind of one-sentence prompt people actually type, and you look at what comes back before you run it.

The prompt for this unit

Here is the prompt for this unit, in full: "Write a Python function that returns Bernoulli numbers." That is it. One sentence.

Questioner: Would anyone really type a prompt that vague?

Narrator: This is not a trick. The prompt was not invented to make the system fail. It is what people actually type, when they roughly know what they want and the tool is right there. It can produce plausible-looking code, and it leaves every decision unstated. And it needs nothing private. Paste nothing confidential, personal or assessment-restricted, and no one else's unpublished work, into an AI system; this prompt is one sentence about a published sequence.

The precise prompt is not in this unit

There is a precise version of this prompt. It is in Unit 10, and it will not be shown before then. That is deliberate, and worth saying plainly: if you saw the good specification now, you would not experience the gap between what you meant and what you asked for. That gap is the subject of this unit. You have to walk into it once.

Count what the sentence leaves open

[On screen: The whole prompt, 'Write a Python function that returns Bernoulli numbers.', sits at the top. Below it a panel labelled 'what the code has to contain' shows five dashed blanks, each opening beside one decision the sentence leaves open: which convention, what type to return, what range of indices, what happens at the odd indices, and whether the answers need to be exact. A caption says every one has to be settled before the code can exist, and you settled none of them. The panel then becomes the generated output, under the caption: a language model produces a statistically plausible continuation of your text. A cursor writes each line and, without stopping, fills each blank with the words 'a choice, unstated', as the captions continue: where your text is silent, the continuation still has to contain something; what it contains reflects patterns in similar code, not your requirements. The decisions fade and the blanks take the colour of the rest of the text, and the panel is marked 'arrives looking complete'. The closing text reads: silence in a prompt does not produce a gap in the output; it produces a confident default.]

Look at the sentence again and count the things it leaves open.

Checker: It does not say which convention. It does not say what type to return. It does not say what range of indices to cover. It does not say what should happen at the odd indices. It does not say whether the answers need to be exact.

Narrator: Every one of those has to be settled before the code can exist, and if you did not settle it, the generated code carries a default you did not set, unstated.

Why the gap fills itself

That is the mechanism to hold on to. A language model produces a statistically plausible continuation of your text. Where your text is silent, the continuation still has to contain something, and what it contains reflects patterns in similar code rather than your requirements. Silence in a prompt does not produce a gap in the output. It produces a confident default.

Five named failure modes

Five specific ways this can go wrong have names in this course.

[On screen: Headed 'Diagnostic failure modes', a diagnostic UI toggles between Table and Python modes. In Table mode, one output row from each of five candidate routines, A to E, gets an amber tag in turn: A, n = 1, gives +1/2, tagged 'b1-sign'; B, n = 3, gives -1.1102230246251565e-16, tagged 'odd-terms'; C, n = 1, gives 1/6, the modern B2, tagged 'ada-indexing'; D, n = 2, gives 0, the modern B3, tagged 'off-by-one'; E, n = 4, gives -0.03333333333333333, tagged 'not-fraction'. In Python mode, one real line from each of four files in course/lab/examples is shown with the codes check.py reports for that file: wrong_b1_sign.py, whose docstring names the second convention, B1 = +1/2, is tagged 'b1-sign', first divergence at n=1; float_output.py, whose function is declared to return float, is tagged 'not-fraction' and 'odd-terms', first divergence at n=2; off_by_one_range.py, which returns _b(n + 1), and ada_indexing.py, which returns _modern_bernoulli(n + 1), share one tag, 'off-by-one', 'ada-indexing', 'odd-terms' and 'one-observation', with first divergences at n=0 and n=1. Above them, an illustrative comment written for this animation, not taken from the lab, reads 'Ada Lovelace used this same recurrence in Note G (1843).' and is tagged 'unsupported historical claim': a specimen of a claim the code cannot establish, whose truth is not asserted either way. The animation concludes: review may catch some of these by eye; only targeted tests establish exact behaviour.]

b1-sign. B-one comes back as plus one half, the second convention's value. Everything else agrees. If that is what the routine computes, nothing is broken; the wrong question was answered.

odd-terms. The odd values above B-one come back as something other than exactly zero: a tiny non-zero decimal, or a value moved there from another index.

ada-indexing. Only the non-zero terms are numbered, the way Note G does it. A prompt that asks for Lovelace's Bernoulli numbers invites it.

off-by-one. Every value shifted by one index. Real Bernoulli numbers throughout, each filed under the wrong label.

not-fraction. Floating point instead of exact fractions. Minus zero point zero three three looks close to minus one thirtieth, but the two are not equal.

Questioner: Could careful reading catch these without running anything?

Narrator: Notice what those five have in common. Careful code review may catch some of them, especially an inexact return type or an obvious range error. None is settled by appearance alone, and a targeted test exposes each one directly. When you name what to test, aim to break the routine rather than confirm it. Property-based testing does that: it states a property that must hold for every input, such as every odd index above one giving exactly zero, and checks it on many inputs rather than the few you thought of. The checker has three more codes. Raises is for code that fails at some index in the range; import-error and no-function are for code that does not run at all. Those announce themselves.

The sixth one, which no test catches

There is a sixth failure mode that no test will catch. Generated code can arrive with a comment attached. Something like: "Ada Lovelace used this same recurrence in Note G, eighteen forty-three." Confident, specific, dated, and sitting directly above arithmetic that turns out to be correct.

Questioner: If the code is correct, is the comment right too?

[On screen: Headed 'One generated output contains different claims'. Four boxes. 'Code behaviour: does the routine meet its contract?' gets an arrow to 'Targeted tests: evidence about behaviour'. A dashed red arrow from the targeted tests to 'Historical comment: did Note G use this recurrence?' is crossed out and labelled 'evidence does not transfer', and a line below reads: correct arithmetic is not evidence that the comment is accurate. Then the historical comment gets its own arrow, to 'Historical source: evidence about the comment'.]

Narrator: Correct arithmetic is not evidence about the comment. Both were produced the same way, and either can be wrong while the other is right. The code got checked because code can be run. The historical claim sat beside it, and nobody checked it. That mixture is what makes generated output awkward to evaluate. It contains arithmetic you can test and assertions about the world you can only look up, and both can be equally polished. Their reliability must be checked differently. The same holds for tests. Some tools write tests for the code they generate, run them, and report that they pass. Those tests are the tool's own claims, and "all tests pass" is one more. Check them like any other output, starting with what they test. A displayed step-by-step explanation is in the same position: one study in twenty twenty-three showed it need not reflect what drove the answer.

A claim with no code attached

The same triage applies to claims that arrive with no code attached at all. Here is one phrased with total confidence: Gödel's theorem proves that an AI cannot verify its own output. You do not need formal logic to triage it. It names a theorem about one kind of system, then applies it to the broad category "an AI", without showing that the theorem's conditions or its conclusion carry over. Mark the theorem for a source check, and the transfer for a separate argument. Unit 18 states the theorem within its actual scope, and explains why it does not settle what a language model can verify.

Three piles

Take the vague prompt to whichever AI system you have access to. Save the output exactly as it came, without tidying it. Save the prompt too, and the date. Then go through the output and put each substantive part into one of three piles. Trust: things you are willing to rely on now, with a reason. Doubt: things that look plausible but that you cannot confirm. Must test: things where you can name the specific check that would settle it. The third pile is the useful one. If you cannot name the check, the item does not belong under must test. It belongs under doubt.

Do not run it yet

Alongside the prompt and the date, record which tools were on: web search, code execution, memory, a reasoning setting. Some may be on without your choosing. One provider's documentation, as of November twenty twenty-six, says its assistant may search the web without being asked, and that memory can be on by default. Do not run the code yet. Unit 8 is where it gets run, and it is more instructive if your predictions are on paper first: which codes apply, and the first index that diverges. There is a second reason, and it holds outside this course too. Generated code is code from a source you cannot vouch for. Read it before you run it, and run it somewhere disposable.

No AI access?

If you have no AI access, or your system is unavailable, use the sample implementations in the lab's examples folder. They are real instances of these failure modes, written for the lab. One more file has a single wrong value that none of the codes describes. The activity works the same way with them: you predict the fault instead of spotting it.

Your checkpoint

Keep what you saved. Unit 8 puts it through the checker. Unit 17 uses it as evidence of what you generated and what you did about it. One checkpoint for this unit, cp07-trust-triage, and it is about the comment rather than the code.

Closing, 20 seconds, no speech. A candidate record on the left, a check against an independent reference in the middle, and a bracketed claim on the right fade in side by side. Below them: Continue with the course material, ai.qyber.dev/edu-ai-literacy-ada-verification/, Course playlist: youtube.com/playlist?list=PLPsBWTfgvakM, All links are also in the description, and Keep your record. Check the next claim.