Unit 5: Prompts, context and attachments

Reading time: about 11 minutes. Activity: about 14 minutes.

1. Why the same question gives different answers

You ask something, get a useful answer, ask a colleague to try the same thing, and they get something different. Or you return to a conversation a week later, ask a follow-up, and the answer no longer fits what you remember agreeing.

Neither is mysterious, and neither is, on its own, evidence about the underlying model. Sampling or a changed service can produce it (Unit 3). So can something easier to miss: the two questions may not have been the same question, because a prompt is not what reaches the system.

This unit is about what actually reaches it, and what follows for checking.

2. What actually goes in

Unit 3 described generation as depending on a context. For a text-only model, that context is a single sequence of tokens assembled just before generation, and your typed prompt is only part of it. Multimodal systems may also receive encoded images, audio or video, and how an attachment is processed varies. The rest of the context can include:

• Instructions you did not write. A deployed product may add its own standing instructions about tone, refusals, format and role. Where it does, you do not normally see them, and they can change between releases.
• Earlier turns of this conversation. Possibly all of them, though a long conversation may be truncated or summarised rather than sent whole. This is why a long conversation can cost more per question than a short one, and why something you said twenty messages ago can still be shaping the answer.
• The contents of attachments, converted to text or otherwise encoded.
• Retrieved passages, if the system searched (Unit 4).
• Tool results from earlier in the exchange.
• Saved memory and instructions, in products that carry them from one conversation into the next (see "What follows for checking").

Two consequences follow immediately, and they are the practical content of this unit.

Your prompt can be a minority of the context. The thing you are tempted to perfect is one contribution among several. So good prompting is not a matter of finding magic words: much of what shapes an answer is the other material that goes in with your prompt.

The same words in a different conversation are a different input. So "it gave me a different answer" is not necessarily sampling, or a changed model, or anything about the system at all. It may simply be that the input differed, because the input was never just your sentence.

3. The window has an edge

The context is finite. Every system has a limit on how much text it can take as input at once, and when a conversation or a document exceeds it, something must go: earlier turns may be dropped, or summarised into something shorter.

Overflow is not the only way material goes missing, and with long limits it may not be the likeliest. A long document may be cut into pieces of which only some are retrieved. And a document that fits whole may still be used unevenly: in one study published in 2024, testing models from 2023, material in the middle of a long input was used worst on the tasks it set [L8, p. 157]. None of this is usually shown to you. All of it means that material you supplied may not be in the context when the answer is generated, or may be there and carry little weight.

This produces a failure mode with a distinctive shape, and it is the one worth carrying out of this unit:

The system answers as though your document said less than it does — and the answer reads as confident, because the context it was generated from did say less.

Suppose you supply a forty-page report and ask about a clause on page thirty-one, and page thirty-one did not survive the window, or was not retrieved, or was there and barely used. The answer can still be fluent and wrong, built from the pages that did arrive, and cite one of them: page four. Nothing about its tone distinguishes this from the case where the whole document was present. This is why a locator matters — the part of a citation that says where to look, such as a page, a section or a clause. An answer that names a place lets you check whether the cited passage exists and supports the answer. It does not establish whether that passage reached the model.

The practical response is not to trust a limit you cannot see. For anything that matters, ask about one short part of a document rather than the whole of it, and ask for the passage the answer relies on.

4. Attachments are not references

Attaching a document does two things worth keeping apart.

It moves the task into grounded answering (Unit 1): the reference is now something you hold, so the check has a clear limit.

It does not make the answer an extract. What comes back is still generated text based on the document. It can paraphrase past what the document supports, attach a real page number to a claim from elsewhere, or answer from the part of the document that survived the window while sounding as though the whole of it had been there. Human summaries do this too: a colleague's short summary, a slide quoting a policy. What is new is the speed and the volume. The check is the one you would apply to any second-hand summary.

So the discipline is short and it is the same one every time:

1. Ask for the passage, not just the answer.
2. Read the passage in the document, not in the answer.
3. Ask whether that passage settles the question you asked.

Step 3 is the one people skip. A passage can be genuinely present, genuinely quoted, and about something adjacent to your question.

An attachment has one more property to keep in mind. Its text joins the same context as your instructions, and text inside an attachment or a retrieved page can act as instructions: a line addressed to the system, visible or hidden, which the output then follows. This is called indirect prompt injection, and it was shown working against a real deployed system in 2023 [L6, abstract]; industry security guidance from 2025 counts web pages and files as routes, counts hidden text [L7, "Indirect Prompt Injections", opening paragraph], and lists no fool-proof prevention [L7, "Prevention and Mitigation Strategies"]. So treat attached and retrieved content as data, and check what the output did with it. If an answer does something you did not ask for, look in the source for text that asked for it.

5. Three things that change the answer, and are not magic words

Stating what would count as correct. This is the move from Unit 1: it turns a vague request into generation to a specification, and it is what Unit 10 is about. It changes the answer more reliably than any phrasing trick, because it changes what you can check the answer against.

Giving an example of the form you want. Useful and cheap, with one hazard: an example constrains the shape of the answer, not its truth. A response in exactly the format you demonstrated can be entirely wrong, and the matching format makes it read as compliant. One of the course's lab examples returns every answer as an exact fraction, as its instructions ask, and none of its thirty-one answers is right.

Starting a new conversation. Underrated. If a conversation has gone somewhere you did not intend, the accumulated context is now working against you, and clearing it is more effective than adding turns to correct it. This is also the honest way to ask a second time: a fresh conversation is a different input, and if you want to know whether an answer depends on how you got there, that is how to find out. Check first that nothing carries over into it (see below).

Note what none of these do. None makes the output self-verifying. They change what you get and how easily you can check it; they do not remove the check.

6. One before and after, on a non-code task

You have attached a twenty-page departmental handbook and you want to know what it says about late submission.

Before: the prompt as first written.

What does this say about late submission?

What comes back is likely to be fluent, plausible, and about late submission. It may also be right. The problem is not that it is wrong; it is that you cannot tell, and finding out means reading the handbook, which is the work you were trying to save.

After.

Using only the attached handbook, list every rule it states about late submission. For each one give the section number and quote the sentence it comes from. If the handbook does not state a rule about something, say so rather than filling the gap.

Three things changed, and none of them is a magic word.

• It states what would count as correct — every rule, from this document only. That is Unit 1's move into generation to a specification, and it gives you something to compare the answer against.
• It asks for locators. Each row now points at a place you can open. A fabricated section number is visible immediately; a paraphrase that names no place is not.
• It says what to do about absence. Silence about a rule and a statement that no rule exists are different findings. If a request does not tell them apart, the answer tends to be whichever reads more fluently.

What has not changed is worth being equally clear about. The second answer is not verified. It is checkable, which is different and is the whole point: you can now spend two minutes confirming three section numbers rather than an afternoon reading a handbook to find out whether each returned rule is right. These checks establish support for the returned rules. Checking completeness still requires searching the handbook for missing rules, for exceptions, and for clauses that say which rule wins. A quoted sentence with its section number is what Unit 0 called a certificate: anyone can check it without trusting the system, and checking costs a fraction of producing it.

And if a quoted sentence turns out not to be at the section given, you have learned something specific, not "the answer was wrong" but how it was wrong, which is what Unit 9 turns into a named vocabulary.

7. What follows for checking

These points lead to three habits.

Record the whole input, not just your prompt. When you keep a prompt for your evidence pack — as later units ask you to — a prompt without its conversation is not a record of what produced the output. Note whether the conversation was fresh, what was attached, and whether anything was retrieved. If you keep the record as a shared link to the conversation, remember who else can read it: one provider's documentation (as of November 2026) says such a link is readable by anyone who has it, and copies made from it stay readable after you withdraw the link [D3, "Frequently asked questions"].

Treat a fresh conversation as the reproducible case. If you want to know whether a result holds, ask in a clean context. Repetition inside the same conversation is the least informative kind, because the earlier answer is now part of the input. A new conversation is not automatically clean, though. Where a product carries saved memory, account-wide instructions or project instructions into new conversations, they are part of the input as well; one provider's documentation (as of November 2026) describes all three, with memory able to be on by default [D1, "Instructions for Claude", "Project instructions"], [D2, "What is Claude's memory?"]. Note whether they were on.

Asking again can also mean asking differently: reword the question, or reorder the options it offers. Sometimes the right answer should not depend on the wording. If the answer then changes, one of the answers was wrong, and you have found that without knowing the right answer. Software testers call a relation of this kind metamorphic [E20, sec. 2.2]. An answer that stays the same when reworded is consistent. That does not make it correct.

Ask for locators, always. "The policy permits this" points nowhere. "Clause 4.2, second sentence" points to one place. The second costs the system very little extra and turns an open-ended search into a check you can finish, and when the locator turns out not to exist, you have learned something the fluent version would have hidden.

Two claims are easy to hear as opposites, and both hold. Prompting matters: Unit 10 is about writing a specification precise enough to be worth prompting with. And wording cannot recover what is not in the context: no better phrase reaches material the system never received. This unit's subject is the second.

Long documents are usable. The limit is one you cannot see from the outside, so the move is to supply the part that matters rather than to hope the whole arrived.

Summary

• What reaches the system is an assembled context: possibly standing instructions you did not write, earlier turns of the conversation (perhaps shortened), attachments, retrieved passages and tool results. Your prompt can be a minority of it.
• Different answers to "the same question" may mean the inputs differed, because the input was never just your sentence.
• The context has an edge, and what falls off it is usually not shown to you; a document that fits may still be used unevenly. The characteristic failure is a confident answer generated from a document that was only partly present or partly used.
• An attachment moves the task into grounded answering; it does not turn the answer into an extract. Its text is data, and can carry instructions.
• A fresh conversation is clean only if no memory or saved instructions carry into it.
• Ask for the passage, read it in the document, and ask whether it settles your question. Record the whole input, not just the prompt.

Then answer cp05-what-reaches-the-system and cp05-locators at the foot of this page.