Unit 2: What a language model does differently

Reading time: about 9 minutes. Activity: about 16 minutes.

Start from what changes

Unit 1 gave you three useful properties of an ordered procedure: a specified sequence, inspectable state, and reproducible execution under fixed conditions. A language-model service is also software with algorithms, state, and control flow. The important difference is what its generated answer exposes to you: the answer is not an execution trace that proves each claim. This unit explains why that changes how you check it.

The mechanism, at the level this module will defend

Most current large language models use a neural-network architecture called a Transformer [L1]. Text is divided into tokens, which are words, parts of words, punctuation, or other text fragments. The model turns those tokens into numerical representations. Through many layers, attention lets the representation at each position combine information from relevant positions in the available context. The final layer assigns a score, converted to a probability, to each possible next token.

During generation, one token is selected and appended. The model then processes the longer context and produces a new distribution for the next position. A paragraph is built by repeating this step. Attention is a way of combining context; it is not a database lookup, a citation, or a truth test.

The base model's pre-training adjusts many numerical parameters to improve next-token prediction over a large training corpus. A deployed assistant is usually more than this base model. It may be post-trained on demonstrations and preference judgements to follow instructions [L2], and a product may add search, document retrieval [L3], calculators, code execution, memory, or other tools. Some systems also produce intermediate plans or reasoning text. These layers can improve the result, but none makes a generated claim correct by definition. Visible reasoning is material to inspect, not a guaranteed trace of every internal computation.

This is the level of mechanism the module needs: transformer processing over a finite context, next-token generation, and optional system layers around the model. Detailed claims about internal representations or any particular product need separate evidence.

Three consequences

Selection can involve sampling

When generation samples from the token distribution rather than always choosing the highest-probability token, the same prompt and context can produce different text on different occasions. A service can also change its hidden instructions, model version, tools, or settings between runs.

Compare that with Unit 1. There, a deterministic procedure should repeat its output when its implementation, input, environment, and settings are fixed. Here, variation may reflect sampling or a changed service. Deterministic decoding can also repeat the same answer. Repetition can reveal stability or change, but neither outcome establishes correctness, and re-asking is not an independent check.

Correctness is not computed at the generation step

The base generation objective produces a continuation that fits its context. In a system with no search, database, or tool attached, the generation step does not consult an external record of what is true.

Some deployed systems do attach retrieval or search. That is a genuine difference and it changes where you aim your checking: whether the retrieved material is appropriate, whether it says what the answer claims, and whether the answer preserves its qualifications. It does not remove the need to check.

Fluency does not establish correctness

This is the one to keep.

A true sentence and a false sentence can both be well formed, specific, confident in register, and correctly punctuated. A false answer need not look visibly worse.

Now list the cues you actually use when judging whether a piece of writing is reliable: it reads fluently, it is specific, it is not hedging, it is properly formatted, the details sound plausible. A text-generating system can produce every one whether or not the content is correct. Surface cues may help you notice a poor answer, but they cannot substitute for checking the underlying claim.

The established term for fluent false output is hallucination; it is defined in course/glossary.md and used consistently across this module. Treat the word with some caution. It suggests a malfunction, an unusual event. It is the same process that produced the correct output, on an occasion when the plausible continuation and the true one happened to differ.

Two things this unit is not saying

It is not saying the output is random. It is not: token probabilities constrain which continuations are likely. That still does not make a generated claim self-verifying.

It is not saying avoid these systems. In Unit 4 you will use one deliberately. The claim here is narrower and more awkward than "do not trust AI": inspection alone cannot establish which case you are in, so confidence must be proportionate to an appropriate check.

The two systems, side by side

The last row is why the rest of the module exists.

The triage

If you cannot judge output by looking at it, you need a habit that runs before you look. The question is not "does this look right?" It is:

How would this be wrong, and what check catches that?

That question has three broad answers, and sorting outputs into them takes a couple of seconds once you are used to it.

Needs exact checking. The output has a right answer that you can compare against: a calculation, a conversion, a piece of code, a data transformation. The check is to run it against values you already know, or to execute it against an automated test suite. Examples include unit conversion, budget formulas, recipe scaling, a data-parsing function, or a small mathematical algorithm. This is the category where a check can be decisive about the values it covers — and it is the category Units 3 to 5 live in.

Needs source checking. The output makes a claim about the world: a date, a citation, an attribution, a legal or institutional requirement, an API specification, or a summary of a document you have not read. Nothing can be executed here. The check is to look it up in the official record: the source document, the handbook, the API documentation, the licence, or another authoritative record. Running the output a second time, or asking the system whether it is sure, may produce a useful revision, but it is not independent verification: the second output can preserve the first output's unsupported assumptions. Check against an appropriate source, calculation, test, or independently derived method.

Low-risk drafting. An error would be cheap and visible, and you were going to rework the text anyway: a first draft of a paragraph you will rewrite, candidate titles, a docstring you will edit, a meeting-summary outline, or a rephrasing of your own notes. This category is real, and pretending otherwise produces a policy nobody follows.

Two warnings about using the triage.

Sort by how an error would reach you, not by how important the task feels. Important work can contain low-risk drafting; a throwaway task can contain a number that has to be exactly right.

And notice that a single output usually spans categories. Generated code with an explanatory comment about who invented the method contains both an exactly checkable part (the code) and a source-checkable one (the historical comment), and they fail independently. That combination is precisely what Unit 4 puts in front of you.

A note on the general case

The operational lesson is broader than text generation: an answer's own assurance is not independent evidence for that answer. Re-prompting and self-critique can improve an output [T9], but they can also preserve a shared error; a stronger check uses evidence appropriate to the claim, such as an authoritative source, an exact calculation, or an independently derived test. The closing unit sets this lesson through the older, published idea of second-order reasoning; it does not present that as a theorem about language models. See course/second-order.md; the closing Unit 9 teaches from it, and nothing on this page, in the activity, or in either Unit 2 checkpoint depends on either.

Summary

• A transformer uses attention to combine token context and produce next-token probabilities; generation selects and appends tokens repeatedly.
• Modern assistants may add post-training, retrieval, tools, and intermediate reasoning around that model. These layers change capability, not the need for evidence.
• Selection is often sampled, so repetition can vary an answer without independently verifying it.
• Correctness is not a quantity computed at the generation step.
• Fluent presentation can accompany correct or incorrect content, so surface cues alone are insufficient.
• Triage by failure mode: exact checking, source checking, low-risk drafting.

Now do activity.md, then take checkpoints cp02-risk-triage and cp02-system-layers.