Unit 9: Grounding and citation checking

Reading time: about 11 minutes. Activity: about 30 minutes.

The second worked case

Units 6, 7 and 8 took one thing seriously: you built a reference by hand, asked for a routine, compared value by value, and stopped at the first divergence. That is the method, and it worked because the case had a decidable answer.

Most of what you will actually check is not like that. There is no oracle for "does this source support this claim". So this unit runs the same method on a case with no oracle, and the point is that the method survives intact:

This is not a lighter version of the lab. Building the reference is still the work, the comparison is still item by item, and the scoped statement at the end is still the deliverable. What changes is that the last column's judgements are yours, because no program can make them.

What a citation from a generative system establishes

By itself: nothing.

A citation produced alongside a claim is another piece of generated text. It sits in the same output, produced the same way, with no more evidence attached than the sentence it follows. Unit 3 has the mechanism; nothing in generation consults a record of what is true, and attaching a bibliography does not change that.

But a citation is the most useful sentence in the output, for a reason that has nothing to do with its reliability: it is the one part that is cheap to falsify. An unbounded claim leaves you a research task. A claim with a locator leaves you a bounded one — you know which source to open and where to look. How long that then takes depends on the source: a page you can reach settles it in a minute, a paywalled or dense one does not. Asking for citations is worth doing not because they are evidence but because they are testable.

Three questions, in order

A citation has to survive three questions, and they get very unequal attention.

1. Does the work exist? Author, title, year, venue. This is the question everyone asks, and it is the one that fails least often.
2. Does the locator exist? The page, section, clause or figure. A real work with an invented page number is common and is invisible unless you look.
3. Does the source support this claim, as stated, at this strength? This is where most real failures are, and it is the one that costs actual reading.

Question 3 is the whole exercise. A summary can cite a real paper at a real page that genuinely discusses the topic, and still misstate what it found, and every surface signal, including the correctness of the first two answers, points the wrong way.

Six citation-failure modes used in this exercise

A closed vocabulary for this case, used with no synonyms, exactly as the code failure modes work in the lab. Closed for this exercise, not exhaustive of reality: a nonexistent work, a selective quotation, a compound claim only half supported, a retraction, and several sources that turn out to share one origin are all real failures outside these six.

These are primary diagnoses, not exclusive descriptions

The six sit at different layers. fabricated-locator is about the pointer, stale-version about which edition, misattributed about which work, and the last three about what the passage actually supports. A real claim can fail at several layers at once — a fabricated page in a superseded edition making an unsupported inference is one claim with three faults, and no wording of the definitions will make the categories exclusive in every case.

So you are naming the primary fault, and there is an order for deciding it. Work down; the first one that applies is the diagnosis.

1. Does the locator exist? No → fabricated-locator.
2. Is an explicitly superseded statement being presented as current? Yes → stale-version.
3. Is the claimed finding absent from the cited work, but supported by another identified source? Yes → misattributed.
4. Does the source support the same proposition, population, direction and scope, only less precisely? Yes → over-specified.
5. Does the source support a weaker proposition, with the summary adding causation, generality, certainty or scope? Yes → unsupported-inference.
6. Is the cited passage the right place to check, and does it report a different value, population, direction or result? Yes → right-source-wrong-claim.

The order is not arbitrary. It runs from the cheapest check to the most expensive: whether a page exists, then which edition, then which work, and only then what the passage means. That is also the order in which a fault makes the later questions moot — there is no point asking what a passage supports when the passage does not exist.

Two distinctions are worth pinning down, because they are the ones people run together even with the order in front of them.

over-specified against unsupported-inference. Both are "the source says less than the summary", and the difference is what kind of less. Over- specification keeps the proposition, the population and the scope and sharpens a quantity — an approximate magnitude becomes an exact figure, "in the 1990s" becomes 1994. An unsupported inference changes the logical force — from association to cause, from some to all, from one setting to any setting.

misattributed against right-source-wrong-claim. Both are "wrong source", and the difference is whether the cited work contains the finding at all. In misattribution it does not: the finding is real and belongs to a different work, so correcting the citation repairs the claim. In right-source-wrong-claim the cited passage is exactly where you would look, and it reports something else, so no citation fixes it, because the claim itself is wrong.

Note what that distinction is not. It is not about whether the source is "relevant". Relevance is a matter of degree and a book can be squarely on a subject without containing a particular result. The operational question is narrower and answerable: open the cited work, and see whether the finding is in it.

These are not detected by a program

The lab's check.py names b1-sign by re-running a candidate against an oracle. Nothing equivalent exists here, and nothing will: whether a source supports a claim is a judgement about meaning, and a checker that claimed to settle it would be committing exactly the overreach this module trains you to catch.

So the discipline is not automation. It is the closed list. Having to choose between unsupported-inference and over-specified forces you to say what kind of wrong you found, and "the citation was bad" does not.

Building the reference

This is the step people skip, and it is the same step you did not skip in Unit 6.

The reference is what the sources actually say, in your words, written down before you compare. One row per locator the summary uses, plus a row for any passage that changes how another source may be used:

The third column is the one people leave out, and it is where two of the six failure modes are caught. A passage saying "this supersedes the targets in the 2019 review" supports no claim of its own and decides whether another source may still be cited. So does "we could not separate these two effects", and so does "across the six authorities studied".

The middle column is the reference. It is the counterpart of the value you computed by hand, and it has the same property: you produced it, so comparing against it means something.

Record what you could not retrieve. A source behind a paywall, a broken link, an edition you cannot get — those are findings, not gaps in your work. "Could not retrieve" is a legitimate row and it changes what you can claim at the end.

Comparing, and stopping at the first divergence

Take the claims in order. For each, ask question 3: does the passage support this, as stated, at this strength?

Stop at the first claim that fails, and name its code.

The reason is the reason from Unit 8, and it is worth repeating because it feels wrong: later claims may look fine, and their looking fine tells you very little once you know that the summary is capable of producing an unsupported claim in this register. The first divergence is a precise place to start, not a proof about everything after it.

What it is not is a licence to stop working. You stop the diagnosis there; you still check the rest before you rely on any of it.

What you can say at the end

The deliverable is a scoped statement, exactly as it is in Unit 17.

Not "the summary is accurate", which nothing you did establishes. Something like:

Of eight claims, I retrieved sources for six. Five are supported as stated. One (claim 4) cites a section that does not exist in that document (fabricated-locator); I found the same claim supported at section 2.1 of the same work, so the claim stands and the locator does not. Two claims I could not check because I could not retrieve the source.

Every part of that is defensible: it says what you checked, what you found, what you could not do, and what follows. Compare it with "I verified the references", which says none of those things and cannot be interrogated.

What this unit is not saying

It is not saying generated citations are usually wrong. It makes no claim about how often, because this module has no measurement of that and neither should you. The claim is about what a citation establishes before you check it, which is nothing, and how cheaply you can change that.

It is not saying retrieval-backed systems remove the problem. They move it, as Unit 4 said: a fetched source is a real source, and the question of whether it says what the answer says it says is untouched.

Summary

• A citation from a generative system is generated text, and establishes nothing by itself, but it is the cheapest part of an output to falsify, which is why asking for one is worth doing.
• Three questions in order: does the work exist, does the locator exist, does the source support this claim as stated at this strength. Most attention goes to the first; most failures are at the third.
• Six named failure modes, a closed list, assigned by you rather than by a program, because there is no oracle for what a source supports.
• The method is Unit 8's method: build the reference first, compare item by item, stop at the first divergence, name the fault, and end with a scoped statement rather than a verdict.

Then answer cp09-diagnose-source and cp09-scope-the-claim-source at the foot of this page.