← Module hub · Unit 6 · Self-check

Unit 6 activity: Re-prompting, specification, and overreliance

DEVELOPMENT REVIEW DEPLOYMENT - NOT READY FOR RELEASE

Time: about 20 minutes. Parts 1 to 4 take about 10 minutes; the no-oracle transfer in Part 5 takes about another 10. Only Part 3 differs between the Python and table routes, and either version completes the unit.

Keep what you write. The two prompts and the two results are the core of your Unit 8 evidence pack.

Part 1 — Annotate the clauses (4 minutes)

The precise prompt in page.md has nine labelled clauses. Eight of them remove a failure mode that the checker can name. Fill in the middle column with a clause label, using each of C1 to C8 exactly once.

failure mode clause that removes it why that clause makes it impossible
b1-sign
odd-terms
ada-indexing
off-by-one
not-fraction
raises
import-error
no-function

The third column is the part that matters. A clause label on its own is a guess; one sentence saying what the clause makes testable is an argument. For example, a clause that names the return type turns "does this look about right?" into a question with a yes-or-no answer.

Then answer two short questions:

  1. C9 — the clause forbidding unrequested historical claims in comments — maps to none of the eight codes. Why not, and what would you have to do instead to check it?
  2. Which single clause, if you deleted it from the prompt, would be least likely to be noticed by the checker? Say why.

Part 2 — Look at your own prompt (2 minutes)

Put your Unit 4 prompt beside the precise prompt and answer in writing:

That last group is the interesting one. Getting a requirement satisfied without stating it is luck, and luck is not a property you can rely on the next time.

Part 3 — Re-run the generation

Take the precise prompt from page.md — clauses C1 to C9, without the labels — and send it to the same AI system you used in Unit 4. Then check the result. A better prompt is not evidence of a better answer; the new output gets verified exactly like the old one.

Route A — Python (4 minutes)

Read the new code first, as in Unit 5: check the imports, check for anything that runs at import time, and run it somewhere disposable.

cd course/lab
python3 check.py /full/path/to/your_new_file.py

or paste it into the exercise stub and use:

python3 check.py exercises/ex04_bernoulli.py

Record for both versions:

vague prompt (Unit 4) precise prompt (Unit 6)
first divergence at n
diagnosis code(s)
result

Route B — Table (4 minutes)

Ask the same system for the values produced under the precise prompt:

Using the specification above, list B0 to B12 as exact fractions, and give B30.

Compare against the reference table in the Unit 5 student page, in index order, stopping at the first row that differs. Record the same three rows as the table above: first divergence, which named failure mode it matches, and the result.

Check the same three easily-missed things: are the odd rows present and exactly 0; are the values exact fractions rather than decimals; and is B30 exactly 8615841276005/14322.

Both routes

Write two sentences comparing the two runs:

  1. What changed between the vague and precise results?
  2. What did not change — that is, which faults survived a fully specified prompt, or which new ones appeared?

If the precise prompt still produced a fault, that is a useful result and not a failed exercise. Add the clause that would have prevented it, and note that you have just done for yourself what clauses C2 to C6 record: a bug happened, so a requirement got written down.

Part 4 — Overreliance, in one paragraph

Write three or four sentences answering this:

Which specific piece of understanding, built before you used AI, made you able to judge the generated output? What would you have concluded without it?

Be concrete. Name the value you derived, the convention you knew existed, or the property you knew must hold. "I understood the topic better" is not an answer; "I had derived B4 = -1/30 myself, so 1/30 was a red flag rather than a plausible alternative" is.

Keep this paragraph. It is most of your Unit 8 correctness defence already written.

Part 5 — Transfer when there is no oracle (10 minutes)

The Bernoulli task has one exact reference. These three cases do not. For each case, fill one row of the table below.

Case Claim being relied on Feasible independent check Judgement that remains Stop or escalation condition
Citation
Policy summary
Recommendation

Case A — a citation claim

An AI assistant writes:

Nguyen (2021) found that first-year students who used weekly retrieval practice improved their examination scores by 18 per cent.

Do not assume that the paper, author, date, statistic, or causal wording is real. State the smallest useful claim you would check first and name where you would check it. If the paper exists, checking the catalogue record establishes only that existence and metadata; supporting the sentence requires reading the paper.

Case B — a supplied policy passage

This is fictional practice text, not your institution's policy:

Students may use generative AI to brainstorm questions and organise notes. They must not submit generated prose or code as their own work. Any permitted use must be disclosed using the course declaration. Where an assessment brief sets stricter conditions, the assessment brief takes precedence.

The assistant summarises it as:

AI may be used to draft assessed work provided that its use is disclosed.

Compare the summary directly with the supplied passage. Identify the changed permission and the precedence rule that disappeared. State when you would stop interpreting and ask the named course contact for a ruling.

Case C — a recommendation with an unstated value judgement

The assistant recommends an unpaid placement at a famous company over a paid placement at a smaller local employer because it says the famous name will be “better for your career”.

Separate checkable claims from the hidden judgement. Salary, travel, duties, training, working conditions, and published destination evidence can be checked from records and, where appropriate, triangulated across independent sources. How much prestige, income, time, accessibility, or particular experience should matter is a value judgement that remains yours.

Use these four checking routes precisely:

Multiple generated answers agreeing is not triangulation. Consensus can reveal stability and can help locate a question, but agreement alone is not proof and does not make dependent sources independent.

Finish the unit at the checkpoint

cp06-spec-clauses is available by both routes. It is formative and unmarked.

Python route — from course/lab:

python3 selfcheck.py run cp06-spec-clauses

or, equivalently:

python3 selfcheck.py run --unit 6

Browser route — open build/site/selfcheck.html in any browser, with no installation, and answer the same checkpoint there.

To see where you have got to, and to export a record for your evidence pack:

python3 selfcheck.py progress
python3 selfcheck.py progress --evidence

Before you finish

You should be leaving this unit with:

All of these feed directly into the evidence pack you assemble in Unit 8.

Marking guidance — open this once you have done the activity, to check your own work

Answer key for the Unit 6 activity. The self-check checkpoint marks itself and gives its own diagnostic feedback by either route, so its answer is not reproduced here.

Part 1 — The clause mapping

Each clause is tied to the concrete check that becomes possible once the clause exists. That third column is the point: a requirement you cannot check is a preference, not a requirement.

failure mode clause why it makes the failure impossible
b1-sign C2 convention Fixes the one value the two published conventions disagree about. Without it, both -1/2 and +1/2 satisfy the request, and no test can adjudicate. With it, bernoulli(1) has exactly one acceptable result.
odd-terms C5 zero terms Requires odd indices above 1 to return Fraction(0) rather than being skipped, omitted or raised. This is what makes the property check — every odd term above B1 is exactly zero — a test rather than an assumption.
ada-indexing C3 indexing Rules out the nineteenth-century scheme in which only the non-zero terms are labelled. Without it, a routine whose B1, B2, B3 are the modern B2, B4, B6 is a defensible reading of "like the ones in Ada Lovelace's program".
off-by-one C7 anchor values Pins the sequence to named indices — B0 = 1, B4 = -1/30 — so a shift of one place contradicts a stated value at a stated index instead of merely looking unfamiliar.
not-fraction C6 arithmetic Makes the return type part of the contract, so comparison can be exact. A tolerance would accept -1.11e-16 as zero; exact comparison of Fraction values cannot.
raises C4 domain States that every integer n >= 0 is in the domain, so a routine that works from n = 2 upwards is a specification violation rather than a design choice, and testing at n = 0 and n = 1 becomes meaningful.
import-error C8 importability Requires the module to import cleanly with no side effects and no third-party packages, which is precisely the precondition any automated check needs before it can call the function at all.
no-function C1 interface Names the function, its parameter type and its return type, so "does this file meet the contract?" is a decidable question. It is also the line the checker itself states.

Question 1 — why C9 maps to none of the codes. Nothing in the lab reads comments. No code is executed to produce them, no test compares them, and a file full of false history passes every check in the module with a clean PASS. A historical claim in a comment is generated by the same process as the code beside it and fails independently of it. Checking it means leaving the toolchain entirely: find the primary or secondary source, read it, and cite it — or label the claim as unverified. That is the work of Unit 7, and the reason C9 asks for the claims not to be generated in the first place is that an unrequested claim is one nobody has budgeted time to check.

Question 2 — which clause would be least missed by the checker. The strongest answer is C8, importability. Its failure mode is the loudest one there is — the file will not import, and the checker reports import-error immediately — but the clause also covers requirements that fail silently: a module that prints on import, or writes a file, or reaches the network, still passes every value comparison. The checker never notices, and only reading the code does.

C7 is a defensible alternative answer, for a different reason: its work is mostly done by C1 to C6 already, so deleting it often changes nothing. The good version of that answer notes that C7 earns its place by catching the shift faults that the property clauses do not constrain.

Answers naming C9 are outside the exercise, since C9 is not one of the eight and the previous question already covers it.

Part 2 — Your own prompt

There is no fixed key. A complete answer names specific clauses in all three groups, including the third:

My prompt stated C1 in a rough form ("a function called bernoulli") and nothing else. It left C2 to C8 open. Of those, the code happened to satisfy C6 — it returned Fractions without my asking — and C4, because it handled n = 0.

The observation to draw out is that the third group exists at all. A requirement met without being stated is met by coincidence, and a coincidence is not reproducible: the next generation, the next model version, or the same prompt on another day may settle it differently. This is also why "it worked when I tried it" is weak evidence about a system whose output varies between runs.

Part 3 — Comparing the two runs

Expected patterns, and what each means.

Most faults disappear. The convention, indexing, type and domain faults are the ones a precise clause reliably removes, because each is a single decision that the clause now makes. This is the ordinary result and it is worth stating plainly: the improvement came from specification, not from tone, politeness, or phrasing.

A fault survives. Perfectly possible, and a useful result rather than a failed exercise. Common survivors are performance problems at larger n, an unstated error type for n < 0, and mixed exact and inexact arithmetic inside the computation even though the return type is a Fraction. The correct response is the one the unit teaches: write the clause that would have prevented it, and note that you have just extended the specification the same way C2 to C6 were extended.

A new fault appears. Also possible. Longer specifications produce longer code, and more code has more places to be wrong. Watch particularly for a hard-coded lookup table built from the C7 anchor values, which passes at the seven indices you named and fails elsewhere — the checker compares thirty-one indices, so it catches this, and a learner comparing only the small values by hand may not. If a learner reports this, they have found the sharpest point in the unit: examples are not a specification.

It passed both times. Then the comparison is about confidence rather than outcome. Before, a pass meant the defaults happened to align with the specification; now, a pass means the stated requirements were met. Those are different claims about the same green result, and only the second is repeatable.

Part 4 — Overreliance

A good paragraph is concrete and names the specific piece of prior understanding. Strong answers look like these:

I had derived B4 = -1/30 by hand, so when the routine returned 1/30 I knew the sign was wrong rather than assuming I had misremembered. Without that I would have accepted it, because 1/30 is exactly as plausible a string as -1/30.

I knew two Bernoulli conventions existed. Without that, B1 = +1/2 would have looked like a correct answer, and the disagreement with the oracle would have looked like a bug in the oracle.

I knew every odd term above B1 must be zero. That turned B3 = -1.11e-16 from a rounding detail into a failed requirement.

Weak answers are general — "I understood the topic better", "I was more careful" — and are worth pushing back on, because generality is what fails at the moment of judgement. The test of the paragraph is whether it names something that would have changed a specific decision.

The connection to make explicit when discussing this: the derivation in Unit 3 was not preparation for the lab, it was the thing that made the lab mean anything. Overreliance is not a character flaw or a matter of using AI too much; it is a gap between what you rely on and what you can check. The remedy is on the second side of that gap.

Part 5 — No-oracle transfer

Strong responses keep four different questions apart: what the output claims, what independent evidence is feasible, what remains a judgement, and when the learner must stop or escalate.

Case Claim Suitable check Remaining judgement Stop or escalate when
Citation The paper exists and supports the stated population, method, result, figure, and strength of conclusion. Search an authoritative catalogue or publisher record for existence and metadata, then read the paper for support. A second generated citation is not independent. Whether the evidence is relevant and strong enough for the learner's own argument. The source cannot be located, the statistic or causal wording is absent, or the learner cannot assess the method. Ask a librarian, tutor, or subject specialist as appropriate.
Policy summary The supplied passage permits drafting assessed work if disclosed. Direct line-by-line comparison shows the summary is wrong: the passage permits brainstorming and note organisation, prohibits submitting generated prose or code, and gives the assessment brief precedence. How a real rule applies to a borderline activity not settled by its wording. The applicable brief differs, the terms conflict, or the learner would be guessing about permission. Ask the named course contact before acting.
Recommendation The famous unpaid placement produces better career outcomes for this learner. Check concrete conditions directly and triangulate outcome claims using independent employer information, careers data, and accounts whose dependence is understood. The weight given to income, prestige, travel, accessibility, duties, learning, and personal constraints. Material conditions remain unknown, sources all repeat one claim, or the consequence exceeds what the learner can reasonably assess. Seek qualified careers or accessibility advice.

Consensus alone does not move any row to “proved”. Several independent sources can strengthen a claim when their evidence and dependence are understood; several outputs or pages repeating the same unsupported statement cannot.

The honest position, for instructors

Keep both halves. Learners who leave this unit thinking the module is against using these systems have taken the wrong lesson, and so have learners who leave thinking a good prompt is a substitute for a check. The two sentences to preserve are that the specification work is a genuine gain, and that responsibility and judgement do not transfer.

The other half worth protecting is the limitation. Bernoulli numbers were chosen because they have an exact, published, checkable answer. Almost nothing a student will generate in the rest of their degree does. The transferable move is not "run the checker" but the ordered habit underneath it: state the requirement, find an independent reference, compare in a defined order, name what failed, and say plainly what you could not verify.

Checkpoint

cp06-spec-clauses is self-marking, with its own diagnostic feedback, and its answer is deliberately not reproduced here. Learners reach it by either route:

python3 selfcheck.py run cp06-spec-clauses    # or use the browser self-check

Your private activity record

Browser storage is not a permanent copy

Progress is kept only in this browser, profile and device. Private browsing, clearing site data, removing the profile, a browser reset, storage eviction or device loss can erase it. Keep important answers and contributions separately.

These notes stay in this browser unless you download a backup or activity log.