Unit 7 activity: Ask AI to generate the routine

Read this after attempting the activity.

Most of this activity has no single correct answer, because what comes back from a generation is not fixed. What is fixed is the standard a good triage meets, and that is what this page sets out. Where the activity asks for judgement, the guidance describes what a strong answer does rather than supplying the answer.

Checkpoint answers are not restated here. cp07-trust-triage is defined once, and the browser self-check and the terminal both read that one definition and grade against it.

Expected time: 12 minutes for the required work, Tasks 1 to 4. The optional extensions at the end of the activity add roughly 4, 2, 8 and 10 minutes, sit outside the core budget, and are not needed for the checkpoint. The guidance below marks which material belongs to them, and does not assume any of it has been done.

Task 1 — Generate

Nothing to mark. Two things to check about your own work.

Did you save it untouched? The failure to watch for at this step is quiet tidying — reformatting, deleting a comment that looked wrong, renaming a variable. Each of those edits is a judgement you made, and once folded into the file it becomes indistinguishable from what was generated. Your evidence pack in Unit 17 depends on being able to tell the two apart.

Did you record provenance? Prompt, date, the kind of system, and which tools and settings were on: search, code execution, memory, a reasoning mode. Each changes what produced the output, and some may be on without your choosing (Unit 5); "not recorded" is a weaker record than "off". This course avoids naming commercial products, and you can too: "a general-purpose conversational AI assistant, accessed via its web interface, on 4 March" is adequate provenance. What matters is that a reader could understand how the artefact came to exist.

If your generated routine turned out to be entirely correct, that is a real and unremarkable outcome — this is a well-known sequence and there is a great deal of correct Bernoulli code in the world. It does not mean you have missed the point. It means your triage in Task 2 has to justify the trust rather than merely record it, which is the harder version of the exercise. The optional "triage a second specimen" extension gives you a contrasting case from The verification lab and self-check if you want one.

Task 2 — Triage

What a strong table looks like

A strong triage is recognisable by three features, none of which is about being right.

Reasons in the "trust" column reference something outside the specimen. The strongest reason available to you at this point in the course is your own Unit 6 working: "this returns 1/6 for B2, which matches the value I derived by hand". That is evidence. "The code is clean and well-commented" is not evidence about correctness; it is an observation about style, and generated code can be well styled whether or not it works.

Checks in the "must test" column are specific enough that someone else could run them. Compare these two entries:

• "Must test: whether the values are right." — not a check. It names an anxiety.
• "Must test: bernoulli(n) for n = 0 to 30, compared against reference_bernoulli.bernoulli(n) as exact Fraction equality, reporting the first index that differs." — a check. It has a range, a comparison method, an authority, and a definition of failure.

The difference is the difference between doubting something and being able to resolve the doubt. A "must test" entry with no named check belongs in the "doubt" column, and moving it there honestly is better work than leaving it where it flatters you.

Claims about the world are on separate rows from claims about behaviour. If one row covers "the recurrence and the historical comment above it", the triage has already blurred the distinction the unit is about.

The prompts, worked through

Task 2's two prompts cover the first three of these -- convention, return type, and the behaviour at n = 3, n = 0 and n = 1. The last two belong to the optional "extend the triage table" extension, so a required-work-only table will not cover them; they are worked through here for anyone who goes on to it.

Which convention does it use? Determinable from the code alone, without running it, in most implementations. Look for a special case at n == 1, or a sign flip applied to a single term. course/lab/examples/correct_example.py shows what an explicit answer looks like: the Akiyama–Tanigawa transform naturally yields B1 = +1/2, so the file negates it and says so in a comment. wrong_b1_sign.py uses a different method, a binomial recurrence that also yields B1 = +1/2, and has no such step. What gives it away is a branch that is not there: nothing special-cases n == 1.

If you cannot tell from reading, that is a finding in itself, and it belongs in "must test" with the check "evaluate bernoulli(1) and compare against -1/2".

What type does it return? Look at the imports first. from fractions import Fraction at the top is a strong signal; its absence in numerical code is a stronger one. Then look at the literals: 1.0 and / on floats produce floats. This is the not-fraction failure mode, and it is a real possibility rather than an exotic one, because in Python dividing one integer by another already gives a float: exact fractions have to be asked for.

Worth internalising: -0.03333333333333333 is not -1/30. It is close, it prints convincingly, and it will fail an exact equality test. To the eye nothing appears wrong. float_output.py in the examples folder exists to make this concrete -- it passes a casual eyeball check at every index, and an exact comparison catches it at B2. The size of its error grows with the values, while its digits stay good to about fourteen significant figures.

What does it do at n = 3, n = 0, n = 1? These three indices are where a default filled in for an unstated requirement is most likely to differ from what you meant. n = 3 should be exactly zero; a small non-zero value is odd-terms, a raised error is raises, and a routine that skips the odd indices and closes up the numbering shows odd-terms at the terms it moves there. n = 0 should be 1. n = 1 is where the convention shows. Three single-index checks, chosen because each one isolates a different failure mode — which is what makes them worth more than thirty arbitrary ones.

What is the largest index you would trust it at? The question is designed to be uncomfortable. An honest answer, before testing, is "the ones I have personally checked", which is a much shorter list than the routine's apparent range. That gap between apparent range and verified range is what the differential test in Unit 8 closes. Note also that a float-based routine has no clean answer to this question at all: its values are inexact from B2 on, so there is no index up to which it can be trusted as exact.

Does any comment make a claim a test could not settle? See Task 4.

The failure modes, and the missing requirement each one traces back to

This mapping is the substance of the unit, and it is what Unit 10 builds on directly.

Read the right-hand column on its own. It is very nearly the precise prompt from Unit 10, arrived at from the other direction. That is the general lesson: a specification is not written in advance by a sufficiently careful person. It accumulates, mostly one clause per failure that has already happened. Most clauses in a good prompt are bugs somebody met.

Each of these five may come from a decision the prompt left open and the generated code filled in, or it may be an ordinary bug. The output alone cannot tell you which. Either way, it counts as a failure only once the requirement is written down.

Task 3 — Predicting the checker's verdict

No correct answer, and no penalty for a wrong prediction. What the task is for is calibration: after Unit 8 runs the checker, you will be able to compare what you expected against what was there, and that comparison tells you something about your own reading that no amount of further reading would.

Two patterns are worth naming in advance.

Predicting no failures because the code looked fine. An easy prediction to make, and the five named failure modes are precisely the ones that leave code looking fine. off_by_one_range.py in the examples folder is the extreme case: clean code, exact arithmetic, every returned value a genuine Bernoulli number, and every one under the wrong index. The shift is visible -- it returns _b(n + 1) -- but nothing about that line looks wrong unless you hold it against a specification, because the error exists only relative to one.

Predicting a first divergence too late. It is tempting to nominate a large index, on the reasoning that small cases are easy. The first divergence can just as easily be at B1, because that is where the convention shows. If you predicted B12 and the answer is B1, that is the most informative outcome the task can produce.

The mirror-image prediction — what convinced you a correct routine was correct, and what that check would have missed — is now an optional extension rather than required work, so do not expect to see it. It is worth raising in feedback even when it has not been attempted: a check that passes tells you only about what it tested. Checking B0 to B12 with an absolute tolerance would pass float_output.py, which is wrong, and it would also pass a correct routine. A check that cannot fail for the right reason has not told you anything.

Task 4 — Handling the prose separately

Taking the checkpoint's example:

# Ada Lovelace used this same recurrence in Note G (1843).

1. What evidence would settle it? A primary source, or a cited secondary source: the published Notes themselves, or scholarship that quotes them. It is a claim about a document that exists, so it is settled by consulting the document. Note that the claim also has parts that could be true and false separately — a date, an attribution, and an assertion that a particular recurrence was used. An answer that treats it as one atomic claim has already lost some resolution.

2. Is that the same evidence that settles the code? No, and the difference is categorical rather than one of degree. The code is settled by execution: run it against known values and observe. The claim cannot be executed at all. Different failure modes, different checks, different kinds of authority — and no amount of one substitutes for the other.

3. If the code passes every test, what have you learnt about the claim? Nothing. This is the whole point of the checkpoint and it is worth stating without softening: correct arithmetic is not weak evidence for an adjacent historical claim, it is no evidence. The two were produced by the same process, and either can be wrong while the other is right. If anything, the checkable part being checked makes the uncheckable part more dangerous, because the specimen as a whole now feels verified.

The course is not asserting that this particular comment is false. The point is that you do not know, and that nothing about the code puts you in a position to find out. Unit 16 addresses what is genuinely established about Note G; Historical Notes states which parts of that account are settled and which are contested. Both are careful in a way that a generated comment has no mechanism to be.

A note on why this happens. It is not that the historical part is generated less carefully. Generated text is a single fluent continuation throughout; the arithmetic and the history are produced the same way and are equally plausible on the surface. What differs is not the output but your access to the truth: one part you can settle in seconds by running it, and the other requires a library. Both parts can be equally fluent while differing in reliability. That asymmetry may not announce itself, which is why triage has to be a deliberate step rather than an impression.

The student page's short Gödel example applies the same triage outside code: check what the cited theorem actually establishes, then check the separate argument that transfers it to "an AI". Unit 18 supplies the formal definitions and full scope; they are not required for identifying the unsupported transfer here.

Optional extensions

None of this is required work, and none of it is needed for the checkpoint. The notes below are for anyone who takes an extension.

Algorithm identification. The examples folder contains two distinct methods, which is deliberate: reference_bernoulli.py uses the binomial recurrence and correct_example.py uses the Akiyama–Tanigawa transform. Two independent routes agreeing on exact values is much stronger evidence than one routine agreeing with a copy of itself, and that is a general principle of differential testing rather than a detail of this course.

Representation versus method. The distinction the task is after. A routine using a sound algorithm with float arithmetic is fixed by changing the type; the method was never wrong. A routine using Lovelace's indexing is not fixed by any change of type, because the arithmetic was exact all along and the labels were wrong. Diagnosing which of the two you are looking at determines whether the fix is one line or a rewrite, and generated code gives no outward sign of the difference.

Slow rather than wrong. A naive recurrence that recomputes every earlier term from scratch is correct and impractical. Correctness and efficiency are separate properties with separate checks, and the course's checker deliberately tests only the first.

Negative n. Undefined in the four-line contract (Unit 10's C4 adds it), and a generated routine may say nothing about it -- returning something meaningless, or raising an unhelpful error. The checker names this raises when it occurs inside the required range. The lesson is that "the specification does not say" and "the behaviour does not matter" are different statements, and code cannot tell them apart.

A compact distinguishing test. Four indices cover the five failure modes:

• bernoulli(1) — separates b1-sign from everything else.
• bernoulli(3) — must be exactly zero; catches odd-terms and ada-indexing.
• bernoulli(0) — must be 1; catches off-by-one.
• isinstance(bernoulli(2), Fraction) and bernoulli(2) == Fraction(1, 6) — catches not-fraction.

Four checks cover the five named failure modes (the zero-term check covers both odd-terms and ada-indexing), each isolating a single decision the vague prompt left open. Notice how much Unit 6 that required: you cannot design this test without knowing the convention, the zero property, and the indexing scheme. That is what "build understanding before using the tool" means in practice — not a slogan about diligence, but the reason you were able to write a four-line test instead of thirty arbitrary ones.

And notice what it misses. wrong_b4_sign.py returns 1/30 for B4 and is right everywhere else, so it passes all four checks: a test aimed at the named modes says nothing about an index it never reads. That is the case for also comparing every index against the oracle (Unit 8), and for checks that try to break the routine rather than confirm the cases you expected.

Checkpoint

• cp07-trust-triage — what you can trust before testing

python3 selfcheck.py run cp07-trust-triage  # or in the browser

Both routes ask the same question and give the same feedback. The answer and its diagnostics live in the checkpoint data below, which both routes read.