Where the tokens go

The invoice also redrew my three pots. It reported input and generated output, with thinking and the visible response inside generated output. In a measured sample from the claim-comparison task, 55.7% of generated tokens were reasoning tokens. The returned reasoning block held a signature and no readable text. I paid for deliberation that I could not inspect.

The number of reasoning tokens depended on the model and on the problem I asked it to solve. Even the same model behaved differently across tasks. In Antilibrary, deciding whether two claims meant the same thing required open-ended judgment, while extracting a specified field followed a defined instruction. Claim comparison used far more reasoning. Reasoning and the visible answer drew from the same output allowance. In the 8,192-token response, the model spent the entire allowance reasoning and left nothing for the answer.

My working framework divided token use into three parts: input, thinking and output. The provider’s accounting used two billing categories: input and generated output. Thinking sat inside generated output, alongside the visible answer.

02 · Two maps of the same inputPrompt, context, cached, fresh

I initially tried to divide the input into a tidy, simplistic hierarchy. The categories overlapped.

We often talk about a prompt and its context as two parcels. The system assembles instructions, tools, examples, retrieved material, history and the current request into one token sequence.

Prompt versus context describes why each token was included. Cached versus fresh describes whether the provider had already processed it. A cached prefix might include the whole prompt and part of the context. These are overlapping classifications of the same input.

This distinction mattered when I audited sixteen prompts for caching. A character-based estimate said five had enough stable material to clear Anthropic’s 1,024-token minimum. OpenAI uses the same minimum for eligible prompt caching. Exact token counts found one.

The estimate had counted the whole template. A cacheable prefix ends at the first variable. One long template inserted a book identifier in its opening line, leaving no useful shared prefix despite thousands of characters below it. The opening variable reduced its cacheable prefix to zero.

The one eligible prompt supplied the proof. Its first request wrote a 1,182-token prefix to the cache. The next three read the same 1,182 tokens back at one-tenth of the normal input price. I had seen the saving work.

03 · The cache floorA longer prompt can be cheaper

Another prompt stopped at 913 cacheable tokens, 111 short of the floor. My first thought was to add dummy characters until it crossed the line. That would lower the price, but it would also put nonsense into every call.

Opus, the model helping me inspect the prompt, suggested a better move. Part of the output definition sat after the first changing value, where it could never be cached. We moved those instructions to the front and kept the wording intact. The stable prefix grew from 913 to 981 tokens without adding a word. I tested the rearranged prompt on 22 books. Every response parsed cleanly, and the measured output was essentially unchanged.

The threshold mattered because tokens did not share one price. In this Anthropic Sonnet run, an ordinary input token was the baseline. Writing it to the five-minute cache cost 1.25 times as much. Reading it back cost one-tenth as much. Generating an output token cost five times as much. The bill was four token counts multiplied by four different rates.

That was when the calculation surprised me. If I could extend the prefix with useful content beyond the threshold, to 1,111 tokens, the estimated cost across 1,679 repeated calls fell from $6.42 to $2.41. I had started by trying to cut the prompt. Now I could add tokens and lower the bill. I remember the jolt of it.

The 1,111-token scenario was a calculation. The implemented prefix remained at 981. I could not find 43 more tokens that genuinely improved the instruction, so I stopped. Crossing the threshold did not justify filling the model’s context with rubbish.

The lesson was precise: once a threshold governs the price, token count alone no longer predicts cost. Frequency, order, exact reuse and time matter too.

Cached tokens still occupy context and receive attention. A thousand tokens of filler may be cheap to bill and expensive to reason over. If a design needs to cross the floor, those tokens should earn their place through examples, definitions or decision rules the task actually needs.

04 · Divide the workWe split the workflow in two and saved tokens.

Napkin-Solo has two parts. The Idea Engine reads complex material, breaks it down, explores several visual approaches and selects three ideas. The Render Engine turns those ideas into illustrations.

While processing a hundred important psychology and behavioral science papers (Canon100), I separated the two engines in time. The Idea Engine completed its work across the collection first, leaving a compact brief for each paper. Later, an independent Render Engine agent opened each brief and drew the illustrations. It did not need to revisit the paper or repeat the idea work.

With that separation in place, I could simplify the Render Engine further. Each agent received a settled brief, generated less material and carried less history from one turn to the next. Fresh token use eventually fell from a median of 105,000 to 10,600 per paper.

The evidence supports the broad explanation: the Render Engine had become a smaller job. It cannot assign an exact share of the saving to each change. Separating the engines, shortening the output, reducing the examples, reusing a shared frame, targeting edits, batching checks and moving to a cheaper model all happened in the same redesign. The earlier conversations had not been retained.

This limit matters. Production telemetry showed the scale of the improvement and helped explain its direction. Measuring the contribution of each change required a different instrument.

05 · Two lives of a tokenThe provider counts tokens. The model reads their sequence.

I had started by treating tokens as units of cost. A token is a unit into which the system encodes text. Once an input becomes tokens, those tokens have two lives. The provider counts and classifies them to calculate the bill. The model receives their identities, order and positions to determine what the input means.

This means the same sentences can cost roughly the same and produce different results when their order or location changes. The count tells us how much text the model received. The sequence, position and surrounding context shape how the model interprets it.

That question led me to set up a prompting lab for iterative experiments on prompts already running in production. I began with a small question, changed one rendered part of the prompt and ran each version against the same real books. I preserved every request and response. Each result shaped the next experiment.

One study examined a prompt that contained its length instruction in two places. Near the top, a rule list told the model what to do: 2–3 sentences maximum in the body. Near the end, a JSON output template showed the shape of the answer: “body”: “2–3 sentences stating the exact mechanism.” The rule was a written instruction. The template looked like the answer the model was expected to produce.

Each call extracted several ideas from a book. The field called body was the short explanation attached to each idea. I measured its sentence length.

The production data had contained exactly two sentences in 895 of 895 generated explanations. I first removed the word maximum from the rule. Almost nothing changed. I then made the rule unambiguous: exactly 1 sentence. The template still said 2–3 sentences. None of the 180 generated explanations had one sentence.

I then changed the template to 1 sentence and restored the original rule. The model produced one sentence in 441 of 540 generated explanations. When the rule and template both specified one sentence, it complied in 540 of 540.

The billing view could count the tokens in each version. The model view explained the change in behavior. An instruction inside the expected output shape exerted far more control than the same instruction in the rule list.

This result covered one model, one production prompt and one output feature. It exposed a weakness in the diagnosis “the prompt contains the instruction.” A prompt is an ordered sequence. Examples establish precedent. Field order creates priority. Nearby output labels define local shape. Context changes how each part is interpreted.

Other studies found adjacent effects. Six register labels inside otherwise fixed examples changed the modal classification for 34 of 60 books. Moving the same output field from sixth position to first changed book classifications. Reordering items within an unchanged group barely moved a merge task, while changing which items appeared together transformed it.

Token count helps explain the bill. Token sequence helps explain the result.

06 · Instrument the harnessEvery model call should leave a trace.

The prompting lab depended on an earlier decision to instrument the production system. I had rebuilt the Antilibrary backend as a harness: the machinery around the model that assembles context, sequences calls, checks outputs, controls what gets written and records what happened. The model was one replaceable part. The system’s guarantees lived in the harness around it.

Telemetry began as a thin sensor. A wrapper around each model call wrote token counts, cost, latency and outcome to one append-only ledger. Once the pattern worked, I extended it across all sixteen production call sites. Calls sharing a run identifier could be joined into one book’s journey through the system.

The first version recorded 6,045 receipts but preserved no complete prompt or response. It could show that a step had become expensive, but the evidence needed to explain the number had disappeared.

The gap came from the tracing interface. It received a function that could make the model call, while the request remained sealed inside that function. The response passed back through the tracer and was easy to record. Capturing the request required changing all sixteen call sites so each handed the tracer the exact object sent to the provider.

The ledger grew one blind spot at a time. The empty response led me to record thinking tokens and stop reasons. Model drift led me to record the model that actually served each call. Hidden SDK retries became another field. The missing conversations led to full request and response archives.

I called the resulting telemetry ledger Hisab, Hindi for account or reckoning. Each call now records one wide event and links to the exact request and complete response in content-addressed archives. Identical content is stored once, and an archive failure cannot interrupt production.

A pricing error added the next lesson. One model had been priced at three times its actual rate. The ledger now keeps raw token counts as the durable measurement and calculates cost from a price table when the data is read. The original counts allowed every historical call to be repriced.

This ledger also closes a control loop. The scheduler reads the traces alongside the current files, detects which work has become stale and reruns only the affected part of the library. Measurement guides the next action instead of sitting in a dashboard.

The first 6,045 calls remain a permanent absence. Their token counts survive; their conversations do not. No amount of later reconstruction can show exactly why those calls behaved as they did. That gap is part of the result.

Instrumentation also needs skepticism. I once measured latency by stopping the clock when HTTP headers arrived, before the response body had been read. The result understated latency by roughly 200×. Another metric reported an 18.4% “subject echo” failure rate because it matched gene inside generate and self inside itself. Fixing the substring bug did not fix the concept. A later semantic instrument found zero of 64 alleged failures.

A bad metric closes a question with the wrong answer.

07 · Six operating practicesWhat I would now require from an AI system

I still look at token counts. I now treat them as traces left by a larger system. They expose repetition, hidden defaults, misplaced work and accidental loops. Following them changed what I would now require:

1. 01
   Require every model call to leave a trace.

   Record the task, model, token classes, latency, retries, stop reason and outcome.

2. 02
   Preserve the exact request and response.

   Token counts reconstruct the bill. The archived exchange helps explain the behavior.

3. 03
   Keep raw measurements.

   Store token counts as the durable record and calculate cost from dated provider prices.

4. 04
   Examine the whole workflow.

   Cost may come from repeated context, hidden reasoning, generated output or work assigned to the wrong stage.

5. 05
   Treat token sequence as part of the design.

   Position, proximity, examples and output structure can change the result without materially changing the count.

6. 06
   Test one change at a time.

   Use the same real cases, preserve every exchange and keep causal claims within what the experiment can establish.