Unit 4: Choosing the tool

Reading time: about 10 minutes. Activity: about 14 minutes.

What this unit adds

Unit 3 described what a language model does: attention over a context, a distribution over next tokens, and optional layers built around that base — post- training, retrieval, tools. Unit 1 sorted tasks by where the reference lives.

This unit joins them. Deployed systems differ in which of those layers they have, and each layer you add changes what you have to check — not by removing checks, but by moving them somewhere else and adding one of its own.

The practical claim is narrow: choosing a system is choosing a checking job. If you know which one you have taken on, you can pick the system whose checking job you can actually do.

Four kinds of system

Not products. Kinds, named by what they add to the base model. A single product may be several of these at once, and may change which it is between one release and the next, so identify what a system is doing on this occasion rather than what its name is.

Plain generation

A base model with post-training and nothing else. Everything in the answer is generated, so a task in Unit 1's recall class stays there: the reference is external and you must go and find it.

There is nothing wrong with this kind. It is the right choice when the class is transformation — you supplied the content, so you already hold the reference — and it is usually the fastest and cheapest option. What it is not is a way of finding things out.

Retrieval-backed

The system searches, or is given documents, and generates an answer conditioned on what came back [L3]. This is a genuine difference and it is the most useful single upgrade for the recall class, because it moves the task into Unit 1's grounded answering.

It does not remove the check. It splits it into two questions that are now answerable:

1. Does the source say what the answer says it says? The answer is still generated text about the source, not an extract from it. A summary can stretch a source's qualification, or attach a real citation to a claim the source does not make.
2. Is this the right source? Retrieval returns what matches the query, which is not the same as what settles the question. An outdated page, a draft, a forum post and the authoritative specification can all match well.

The second question is the one people skip, because a citation looks like the end of an argument rather than the start of one.

Tool-calling

The system calls something deterministic — a calculator, a database query, code it wrote and ran, and uses the result.

The check genuinely splits, and the halves are unequal:

• The computation is deterministic and, within its own terms, either right or wrong in the ordinary way. This half is a real gain: arithmetic done by a calculator is arithmetic done by a calculator.
• The decision to call it, with those inputs, for this question is generated. Nothing deterministic chose which numbers went in or what the result would be taken to mean.

So the failure to look for is a correct computation of the wrong question: the right formula on the wrong column, a currency conversion at an unstated date, a query that answers a slightly different question than the one you asked. The result carries the authority of arithmetic and the fallibility of the choice that produced it.

Unit 8's lab is this idea taken seriously in one direction: run the code, compare against a reference you built yourself.

Multimodal

The system accepts images, audio or video, or produces them.

What changes is that the reference stops being searchable. When the source is a document you can find the sentence the answer relies on. When it is a photograph, a chart or a recording, checking means going and looking, and there is no equivalent of searching for a phrase.

Two specific hazards, both worth naming:

• A description of an image is generated text about the image. It can be detailed, confident and about something that is not in the frame, and unlike a fabricated citation, there is no catalogue in which its absence shows up.
• A chart read by a system is a chart read from pixels. Values between gridlines, a truncated axis, a log scale: all are places where a confident reading can be wrong in a way that reads as precise.

Systems that act rather than answer — tool calls with side effects, multi-step loops that carry out tasks — change the question again, because a wrong answer you can discard and a wrong action you cannot. That is Unit 11's subject and this unit does not anticipate it.

A decision aid you can apply in a minute

For a task you actually have, in order:

1. Name the class (Unit 1). Where would the reference come from?
2. If the class is recall, try to move it. Can you supply the source? If yes, do that and use whichever system lets you attach it. This single step removes more checking work than any choice of product.
3. Ask what the answer will contain. Fetched sources? A computed number? A reading of an image? Each of those is a specific check you are taking on.
4. Ask whether you can do that check. Not "is the system good" — can you, with the time and access you have, carry out the check its output will need? If not, you have chosen the wrong kind, or the wrong task.
5. Take the cheapest kind that leaves you a check you can do. Extra layers are not free, and the next section is about that.

Step 4 is the one that does work. A system that produces claims you cannot check is not more useful than one that produces fewer claims you can.

What it costs to ask

Three costs, and they move together because they have the same driver.

Money. Priced by the amount of text going in and coming out, and by the size of the model. Longer conversations cost more than short ones for the same question, because the conversation's history generally goes with it.

Time. Roughly the same drivers, plus the extra steps: retrieval adds a search, tool-calling adds an execution, and a system that generates intermediate text before answering generates more text.

Energy. Real, and driven by the same quantities: more computation for longer inputs, larger models and more steps.

A figure this module will not print

You will have seen numbers for what a single query costs in energy or carbon. This module does not give one, and the reason is the module's own subject.

A defensible figure would have to name the model, the hardware, the length of the query, whether the cost of training the model is amortised into it, and the electricity supply, and it would be out of date within months. Printing a number without those conditions would be exactly the confident-and-unsupported claim this module trains you to catch.

So instead: the shape is reliable even when the numbers are not. A longer prompt, a longer answer, a bigger model and more steps all cost more of all three. If you need an actual figure, that is a Unit 1 recall task, and it needs a source that states its conditions and its date.

When the cheap option is the right one

Often. The instinct to reach for the most capable available system treats capability as free, and it is not — in money, in time, and in checking work, because a more elaborate system produces more kinds of claim to check.

The cheap option is right when the class is transformation or grounded answering and the check is a comparison you will do anyway; when the task is a draft you will rewrite; and whenever the expensive option's extra output is something you would not check.

The expensive option earns its cost when the extra layer removes a check you could not otherwise do — retrieval on a question you cannot research yourself, a tool call on a computation you cannot perform, and when you will actually use the check it makes possible.

What this unit is not saying

It is not saying retrieval-backed and tool-calling systems are more trustworthy. They are differently checkable. Each removes one kind of uncertainty and adds a different one.

It is not a recommendation of any product. Named products date, change what they are between releases, and are not the level at which this reasoning works.

Summary

• Systems differ by which layers they add to a base model, and each layer moves the check rather than removing it.
• Plain generation leaves the whole check with you; retrieval aims it at a source; tool-calling splits it into a computation and the choice to make it; multimodal puts the reference somewhere you cannot search.
• Choose by asking what check the output will need and whether you can do it, not by which system is most capable.
• Money, time and energy all rise with input length, output length, model size and number of steps. The shape is stable; the numbers are not, and this module declines to print one it cannot defend.
• The cheapest kind that leaves you a check you can carry out is usually the right one.

Then answer cp04-what-changes-the-check and cp04-cheapest-adequate at the foot of this page.