Unit 4 activity: Ask AI to generate the routine

For learners after attempting the activity, and for instructors reviewing it.

Most of this activity has no single correct answer, because what comes back from a generation is not fixed. What is fixed is the standard a good triage meets, and that is what this page sets out. Where the activity asks for judgement, the guidance describes what a strong answer does rather than supplying the answer.

Checkpoint answers are not restated here. cp04-trust-triage is defined in course/lab/checkpoints.py, and both self-check routes grade against that file.

Expected time: 12 minutes for the required work, Tasks 1 to 4. The optional extensions at the end of the activity add roughly 4, 2, 8 and 10 minutes, sit outside the core budget, and are not needed for the checkpoint. The guidance below marks which material belongs to them, and does not assume any of it has been done.

Task 1 — Generate

Nothing to mark. Two things to check about your own work.

Did you save it untouched? The commonest failure at this step is quiet tidying — reformatting, deleting a comment that looked wrong, renaming a variable. Each of those edits is a judgement you made, and once folded into the file it becomes indistinguishable from what was generated. Your evidence pack in Unit 8 depends on being able to tell the two apart.

Did you record provenance? Prompt, date, and the kind of system. This module avoids naming commercial products, and you can too: "a general-purpose conversational AI assistant, accessed via its web interface, on 4 March" is adequate provenance. What matters is that a reader could understand how the artefact came to exist.

If your generated routine turned out to be entirely correct, that is a real and unremarkable outcome — this is a well-known sequence and there is a great deal of correct Bernoulli code in the world. It does not mean you have missed the point. It means your triage in Task 2 has to justify the trust rather than merely record it, which is the harder version of the exercise. The optional "triage a second specimen" extension gives you a contrasting case from course/lab/examples/ if you want one.

Task 2 — Triage

What a strong table looks like

A strong triage is recognisable by three features, none of which is about being right.

Reasons in the "trust" column reference something outside the specimen. The strongest reason available to you at this point in the module is your own Unit 3 working: "this returns 1/6 for B2, which matches the value I derived by hand". That is evidence. "The code is clean and well-commented" is not evidence about correctness; it is an observation about style, and generated code is reliably well-styled regardless of whether it works.

Checks in the "must test" column are specific enough that someone else could run them. Compare these two entries:

• "Must test: whether the values are right." — not a check. It names an anxiety.
• "Must test: bernoulli(n) for n = 0 to 30, compared against reference_bernoulli.bernoulli(n) as exact Fraction equality, reporting the first index that differs." — a check. It has a range, a comparison method, an authority, and a definition of failure.

The difference is the difference between doubting something and being able to resolve the doubt. A "must test" entry with no named check belongs in the "doubt" column, and moving it there honestly is better work than leaving it where it flatters you.

Claims about the world are on separate rows from claims about behaviour. If one row covers "the recurrence and the historical comment above it", the triage has already blurred the distinction the unit is about.

The prompts, worked through

Task 2 requires the first three of these — convention, return type, and the behaviour at n = 3, n = 0 and n = 1. The last two belong to the optional "extend the triage table" extension, so a required-work-only table will not cover them; they are worked through here for anyone who goes on to it.

Which convention does it use? Determinable from the code alone, without running it, in most implementations. Look for a special case at n == 1, or a sign flip applied to a single term. course/lab/examples/correct_example.py shows what an explicit answer looks like: the Akiyama–Tanigawa transform naturally yields B1 = +1/2, so the file negates it and says so in a comment. wrong_b1_sign.py runs the same kind of recurrence and simply omits that step. The visible difference between them is one branch.

If you cannot tell from reading, that is a finding in itself, and it belongs in "must test" with the check "evaluate bernoulli(1) and compare against -1/2".

What type does it return? Look at the imports first. from fractions import Fraction at the top is a strong signal; its absence in numerical code is a stronger one. Then look at the literals: 1.0 and / on floats produce floats. This is the not-fraction failure mode, and it is the default outcome rather than an unusual one, because most numerical Python is float-based.

Worth internalising: -0.03333333333333333 is not -1/30. It is close, it prints convincingly, and it will fail an exact equality test. For small indices nothing appears wrong. float_output.py in the examples folder exists to make this concrete — it passes a casual eyeball check across the early indices and diverges once the denominators grow.

What does it do at n = 3, n = 0, n = 1? These three indices are where the specification is least likely to have been guessed correctly. n = 3 should be exactly zero; a routine that omits odd indices, raises, or returns a small non-zero value has hit odd-terms. n = 0 should be 1. n = 1 is where the convention shows. Three single-index checks, chosen because each one isolates a different failure mode — which is what makes them worth more than thirty arbitrary ones.

What is the largest index you would trust it at? The question is designed to be uncomfortable. Most people's honest answer, before testing, is "the ones I have personally checked", which is a much shorter list than the routine's apparent range. That gap between apparent range and verified range is what the differential test in Unit 5 closes. Note also that a float-based routine has no clean answer to this question at all: the failure is gradual rather than sudden, so there is no index at which it starts being wrong.

Does any comment make a claim a test could not settle? See Task 4.

The failure modes, and the missing requirement each one traces back to

This mapping is the substance of the unit, and it is what Unit 6 builds on directly.

Read the right-hand column on its own. It is very nearly the precise prompt from Unit 6, arrived at from the other direction. That is the general lesson: a specification is not written in advance by a sufficiently careful person. It accumulates, one clause per failure that has already happened. Every clause in a good prompt is a bug somebody met.

Note too that none of these five is a coding error. Each is a decision that the prompt left open and the generation filled in. The code does what it does correctly; it was answering a question you did not quite ask.

Task 3 — Predicting the checker's verdict

No correct answer, and no penalty for a wrong prediction. What the task is for is calibration: after Unit 5 runs the checker, you will be able to compare what you expected against what was there, and that comparison tells you something about your own reading that no amount of further reading would.

Two patterns are worth naming in advance.

Predicting no failures because the code looked fine. Very common, and the five named failure modes are precisely the ones that leave code looking fine. off_by_one_range.py in the examples folder is the extreme case: clean code, exact arithmetic, every returned value a genuine Bernoulli number, and every one under the wrong index. There is nothing in it for a careful reader to find, because the error exists only relative to a specification.

Predicting a first divergence too late. People tend to nominate a large index, on the reasoning that small cases are easy. In practice the first divergence is very often at B1, because that is where the convention shows. If you predicted B12 and the answer is B1, that is the most informative outcome the task can produce.

The mirror-image prediction — what convinced you a correct routine was correct, and what that check would have missed — is now an optional extension rather than required work, so do not expect to see it. It is worth raising in feedback even when it has not been attempted: a check that passes tells you only about what it tested. Checking B0 to B12 with a tolerance would pass float_output.py, which is wrong, and it would also pass a correct routine. A check that cannot fail for the right reason has not told you anything.

Task 4 — Handling the prose separately

Taking the checkpoint's example:

# Ada Lovelace used this same recurrence in Note G (1843).

1. What evidence would settle it? A primary source, or a cited secondary source: the published Notes themselves, or scholarship that quotes them. It is a claim about a document that exists, so it is settled by consulting the document. Note that the claim also has parts that could be true and false separately — a date, an attribution, and an assertion that a particular recurrence was used. An answer that treats it as one atomic claim has already lost some resolution.

2. Is that the same evidence that settles the code? No, and the difference is categorical rather than one of degree. The code is settled by execution: run it against known values and observe. The claim cannot be executed at all. Different failure modes, different checks, different kinds of authority — and no amount of one substitutes for the other.

3. If the code passes every test, what have you learnt about the claim? Nothing. This is the whole point of the checkpoint and it is worth stating without softening: correct arithmetic is not weak evidence for an adjacent historical claim, it is no evidence. The two were produced by the same process and fail independently. If anything, the checkable part being checked makes the uncheckable part more dangerous, because the specimen as a whole now feels verified.

The module is not asserting that this particular comment is false. The point is that you do not know, and that nothing about the code puts you in a position to find out. Unit 7 addresses what is genuinely established about Note G; course/historical-notes.md states which parts of that account are settled and which are contested. Both are careful in a way that a generated comment has no mechanism to be.

A note on why this happens. It is not that the historical part is generated less carefully. Generated text is a single fluent continuation throughout; the arithmetic and the history are produced the same way and are equally plausible on the surface. What differs is not the output but your access to the truth: one part you can settle in seconds by running it, and the other requires a library. Both parts can be equally fluent while differing in reliability. That asymmetry may not announce itself, which is why triage has to be a deliberate step rather than an impression.

The student page's short Gödel example applies the same triage outside code: check what the cited theorem actually establishes, then check the separate argument that transfers it to "an AI". Unit 9 supplies the formal definitions and full scope; they are not required for identifying the unsupported transfer here.

Optional extensions

None of this is required work, and none of it is needed for the checkpoint. The notes below are for learners who choose an extension, and for instructors fielding questions about one.

Algorithm identification. The examples folder contains two distinct methods, which is deliberate: reference_bernoulli.py uses the binomial recurrence and correct_example.py uses the Akiyama–Tanigawa transform. Two independent routes agreeing on exact values is much stronger evidence than one routine agreeing with a copy of itself, and that is a general principle of differential testing rather than a detail of this module.

Representation versus method. The distinction the task is after. A routine using a sound algorithm with float arithmetic is fixed by changing the type; the method was never wrong. A routine using Lovelace's indexing is not fixed by any change of type, because the arithmetic was exact all along and the labels were wrong. Diagnosing which of the two you are looking at determines whether the fix is one line or a rewrite, and generated code gives no outward sign of the difference.

Slow rather than wrong. A naive recurrence that recomputes every earlier term from scratch is correct and impractical. Correctness and efficiency are separate properties with separate checks, and the module's checker deliberately tests only the first.

Negative n. Undefined in the contract, and most generated routines say nothing about it — they return something meaningless, or raise an unhelpful error. The checker names this raises when it occurs inside the required range. The lesson is that "the specification does not say" and "the behaviour does not matter" are different statements, and code cannot tell them apart.

The smallest distinguishing test. A compact answer checks four indices:

• bernoulli(1) — separates b1-sign from everything else.
• bernoulli(3) — must be exactly zero; catches odd-terms and ada-indexing.
• bernoulli(0) — must be 1; catches off-by-one.
• isinstance(bernoulli(2), Fraction) and bernoulli(2) == Fraction(1, 6) — catches not-fraction.

Four checks, one per failure mode, each isolating a single decision the vague prompt left open. Notice how much Unit 3 that required: you cannot design this test without knowing the convention, the zero property, and the indexing scheme. That is what "build understanding before using the tool" means in practice — not a slogan about diligence, but the reason you were able to write a four-line test instead of thirty arbitrary ones.

Checkpoint

• cp04-trust-triage — what you can trust before testing

python3 selfcheck.py run cp04-trust-triage    # or use the browser self-check

Both routes ask the same question and give the same feedback. The answer and its diagnostics live in course/lab/checkpoints.py.