Unit 3: What a language model does differently

Reading time: about 14 minutes. Activity: about 16 minutes.

1. Start from what changes

Unit 2 gave you three properties of an ordered procedure, such as the step-by-step table Ada Lovelace published in 1843 (Lovelace 1843, Note G): determinism, explicit state and control flow. A language-model service is also software with algorithms, state, and control flow. An algorithm is a fixed set of steps that turns valid input into output and stops; the ordered procedure of Unit 2 is one. The important difference is what its generated answer exposes to you: the answer is not an execution trace that proves each claim. This unit explains why that changes how you check it.

2. The mechanism, at the level this course will defend

This unit describes a large language model built on a Transformer, a design of neural network introduced in 2017 [L1, sec. 3]. A neural network is a large set of numerical parameters, arranged in layers and adjusted during training. Which architecture a particular product uses is for its maker to state, and the argument of this unit does not depend on it: it rests on next-token generation over a context (all the text the model is given to work from), and would hold if the architecture changed.

Text is divided into tokens, which are words, parts of words, punctuation, or other text fragments. The model turns those tokens into numerical representations, long lists of numbers. Through many layers, attention gives each position (one token's place in the text) a weight for every token up to that point. Each weight comes from how closely two representations match. The representation at each position then combines information from those tokens by weight, so some tokens count far more than others: in "The capital of Australia is", the words "capital" and "Australia" can count most at "is". The final layer assigns a score, converted to a probability, to each possible next token.

During generation, one token is selected and appended. The model then processes the longer context and produces a new distribution, a probability for each possible token, for the next position. A paragraph is built by repeating this step. Attention is a way of combining context; it is not a database lookup, a citation, or a truth test.

The base model's pre-training adjusts many numerical parameters to improve next-token prediction over a large training corpus. A deployed assistant is usually more than this base model. It may be post-trained to follow instructions: trained further on demonstrations, examples of the responses its makers want, written by people, and on preference judgements, people's rankings of several of the model's own responses [L2, secs. 3.1, 3.5]. Post-training changes the style and format of the output and how closely it follows an instruction; it adds no check of whether a claim is true. A product may also add search, document retrieval [L3, sec. 2], calculators, code execution, memory, or other tools. Some systems also produce intermediate plans or reasoning text, and a process that loops before it answers, often called agentic, adds a generated plan, a tool action and a generated critique. These layers can improve the result, but none makes a generated claim correct by definition. Visible reasoning is material to inspect, not a guaranteed trace of every internal computation.

The training text ends at a date. Pre-training uses text collected up to some point, the training cut-off, and nothing written after that date shaped the base model's parameters. Ask about something later, such as a new result, a changed rule or this year's figures, and a model working without tools may still answer instead of saying it cannot know. It can produce a plausible continuation from older text and present an outdated state as the current one. A model's reply about its own cut-off is generated text like any other; for the date, check the provider's documentation.

Search and other tools change this, but only for what they actually fetch. A system with internet access can bring in a newer page, and the answer then rests on that page, which needs the retrieval checks: does the source say what the answer says it says, and is it the right source to settle this (Unit 4)? One provider's documentation says, as of November 2026, that its assistant may search without being asked [D4, opening note], so the same question can be answered from training text one day and from a fetched page the next. For anything that may have changed, ask two things: when does this system's training text end (its provider's documentation may say), and did it fetch anything this time? The answer cannot settle the second on its own word: a link in it is generated text, not a record that anything was fetched. Where the product shows a record of its searches, that is the evidence; where it does not, write the question down as unknown. Either way, open what the answer links to check that the page says what the answer says. That tells you whether the source is sound, not whether the system searched this time. If the answer names no source, find one yourself.

This is the level of mechanism the course needs: transformer processing over a finite context, next-token generation, and optional system layers around the model. Detailed claims about internal representations or any particular product need separate evidence.

3. Three consequences

Selection can involve sampling

When generation samples from the token distribution rather than always choosing the highest-probability token, the same prompt and context can produce different text on different occasions. A service can also change its hidden instructions, model version, tools, or settings between runs.

Compare that with Unit 2. There, a deterministic procedure should repeat its output when its implementation, input, environment, and settings are fixed. Here, variation may reflect sampling or a changed service. Decoding is the rule that picks the next token from the scores: always the top one, or one drawn at random according to the probabilities the scores are turned into. The random numbers usually come not from a physical random process but from a pseudo-random number generator, a fixed calculation started from a number called a seed: the same seed repeats the same draws, so with everything else unchanged the same text comes out again. Deterministic decoding can also repeat the same answer. Repetition can reveal stability or change, but neither outcome establishes correctness, and re-asking is not an independent check.

Asking whether it is sure tests something else again. A study published in 2024, of assistants from 2023, found that those trained with human feedback shift towards what the user appears to believe, and that a bare "are you sure?" can lead to a reply that retracts a correct answer [L5, secs. 3.2-3.3]. This is called sycophancy. That question tests the response to pressure, not the answer. If you do ask again, ask neutrally, or ask for the evidence against the answer.

Correctness is not computed at the generation step

The system is built to produce a continuation that fits its context. In a system with no search, database, or tool attached, the generation step does not consult an external record of what is true.

Some deployed systems do attach retrieval or search. That is a genuine difference and it changes where you aim your checking: whether the retrieved material is appropriate, whether it says what the answer claims, and whether the answer preserves its qualifications. It does not remove the need to check.

Fluency does not establish correctness

This is the one to keep.

A true sentence and a false sentence can both be well formed, specific, confident in tone, and correctly punctuated. A false answer need not look visibly worse.

Now list the cues you actually use when judging whether a piece of writing is reliable: it reads fluently, it is specific, it is not hedging, it is properly formatted, the details sound plausible. A text-generating system can produce every one whether or not the content is correct. Surface cues may help you notice a poor answer, but they cannot substitute for checking the underlying claim.

This was never a safe test for human writing either: confident textbooks and polished websites can be wrong, and people misremember and misattribute. The difference is that polish used to cost effort, which made it a weak signal of care. Generated text has it for free, in any quantity, so even that weak signal is gone.

It also reads as though a person wrote it. Many assistants answer in the first person and say what they did: "I checked", "I'm confident", "I searched for that". You trust a person who says these things, because a person who says them has usually done them. From a system, they are more generated text: "I checked" is not a record of a check. Read them as claims about the work, and look for the work itself: a source you can open, output you can run again, or the product's record of the searches it ran.

The established term for fluent false output is hallucination; it is defined in Glossary and used consistently across this course. Treat the word with some caution. It suggests a malfunction, an unusual event. It is the same process that produced the correct output, on an occasion when the plausible continuation and the true one happened to differ.

4. Two things this unit is not saying

It is not saying the output is random. It is not: token probabilities constrain which continuations are likely. That still does not make a generated claim self-verifying.

It is not saying avoid these systems. In Unit 7 you will use one deliberately. The claim here is narrower and more awkward than "do not trust AI": inspection alone cannot establish which case you are in, so confidence must be proportionate to an appropriate check.

5. The two systems, side by side

The last row is why the rest of the course exists.

6. The triage

If you cannot judge output by looking at it, you need a habit that runs before you look. The question is not "does this look right?" It is:

How would this be wrong, and what check catches that?

That question has three broad answers, and sorting outputs into them takes a couple of seconds once you are used to it. It sharpens Unit 1's question (where does the reference you would check an output against live?), and the two give the same answers: a reference you can execute against gives exact checking, a reference in a record gives source checking, and judgement, with no single reference, is where neither applies.

Needs exact checking. The output has a right answer that you can compare against: a calculation, a conversion, a piece of code, a data transformation. The check is to run it against values you already know, or to execute it against an automated test suite. Examples include unit conversion, budget formulas, recipe scaling, a data-parsing function, or a small mathematical algorithm. The lab's oracle, its trusted source of correct Bernoulli values, is checked this way: it compares its values with twenty-two published ones and agrees with all of them. Even with no known values, units of measurement give a cheap exact check: a result whose units do not match what was asked for is wrong before you look at a digit. This is the category where a check can be decisive about the values it covers — and it is the category Units 6, 7 and 8 live in.

Needs source checking. The output makes a claim about the world: a date, a citation, an attribution, a legal or institutional requirement, the specification of a software interface (an API), or a summary of a document you have not read. Nothing can be executed here. The check is to look it up in the official record: the source document, the handbook, the API documentation, the licence, or another authoritative record. Running the output a second time, or asking the system whether it is sure, may produce a useful revision, but it is not independent verification: the second output can preserve the first output's unsupported assumptions. Check against an appropriate source, calculation, test, or independently derived method.

The same goes for a system's account of its own work. "I checked this", "I verified the references" or "I ran the tests" is generated text like the rest of the output. The evidence is the log, the quoted passage or the test output you can see or rerun yourself, and tests count only if they are independent of what they test.

Low-risk drafting. An error would be cheap and visible, and you were going to rework the text anyway: a first draft of a paragraph you will rewrite, candidate titles, a short description in code that you will edit, a meeting-summary outline, or a rephrasing of your own notes. This category is real, and pretending otherwise produces a policy nobody follows.

A judgement fits none of the three. Unit 1's fifth class has no single right answer, and that does not make it low-risk drafting. Check its factual premises by source or calculation, keep its conclusion a proposal, and let what being wrong would cost decide what you do next (Unit 12).

Two warnings about using the triage.

Sort by how an error would reach you, not by how important the task feels. Important work can contain low-risk drafting; a throwaway task can contain a number that has to be exactly right.

And notice that a single output usually spans categories. Generated code with an explanatory comment about who invented the method contains both an exactly checkable part (the code) and a source-checkable one (the historical comment), and they fail independently. That combination is precisely what Unit 7 puts in front of you.

A note on the general case

The operational lesson is broader than text generation: an answer's own assurance is not independent evidence for that answer. Re-prompting and self-critique can improve an output [T9, sec. 3], but they can also preserve a shared error [T9, sec. 3.3]; a stronger check uses evidence appropriate to the claim, such as an authoritative source, an exact calculation, or an independently derived test. The closing unit sets this lesson through the older, published idea of second-order reasoning; it does not present that as a theorem about language models. See Second-Order Reasoning: The Lens Behind The Course; the closing Unit 18 teaches from it, and nothing on this page, in the activity, or in any Unit 3 checkpoint depends on either.

Summary

• A transformer uses attention to combine token context and produce next-token probabilities; generation selects and appends tokens repeatedly.
• Modern assistants may add post-training, retrieval, tools, and intermediate reasoning around that model. These layers change capability, not the need for evidence.
• Selection is often sampled, so repetition can vary an answer without independently verifying it.
• Correctness is not a quantity computed at the generation step.
• Fluent presentation can accompany correct or incorrect content, so surface cues alone are insufficient.
• Triage by failure mode: exact checking, source checking, low-risk drafting. A judgement is none of these: check its premises, and keep its conclusion a proposal.

Then answer cp03-risk-triage, cp03-system-layers and cp03-training-cutoff at the foot of this page.