Four AI agents examined a proposed group of six books in my library. They concluded that only two of the books existed. I knew the conclusion was wrong. I owned all six.
The agents were checking the work of Antilibrary, a production AI system I had built around my collection of roughly 1,250 books. The system maintains an inventory, extracts the central ideas from each book, classifies them and looks for patterns across the collection. It can find books that share a recurring idea, reveal unexpected connections and show me where the collection is thin. Some of these findings are published at anti-library.ai.
Before a finding can be published, a quality-control layer checks whether the books and ideas supporting it can be traced back to the library records. In this case, the checker decided that four of the six books could not be found. Three additional AI agents reviewed that verdict and agreed.
The problem was in the evidence tool they all used. It could search book titles and extracted ideas, but it could not search authors. When an agent looked for a book through its author, the search returned nothing. The agents converted “I cannot see it” into “it does not exist.”
In that 20-group rerun, adding authors changed five verdicts and left no group classified as unsupported. All four agents had relied on the same incomplete search tool. The human check found the error because I could inspect the actual books.
One’s first working framework of AI is simple. We give the model a prompt, it thinks and returns an answer. That framework is useful for understanding a single exchange. A production system must perform many exchanges as parts of one continuing process. The work must be divided, repeated and checked. Evidence must be retrieved, state preserved and failures repaired.
The author error showed how many parts of the system participated in a single verdict. The result depended on the model, the evidence it received, the tools it could use, the checks applied to its work and the decision reserved for a person. The structure around each model call began to matter as much as the call itself.
The machinery around the model is often called a harness. A harness assembles context, sequences work, calls models, checks outputs, preserves state, records what happened and controls what the system may change or publish.
I built Antilibrary through a deliberate progression. I began with the smallest complete system that could do useful work. Once it was running, I observed it, studied the problems it exposed and added the next layer when its purpose was clear. This allowed the system to grow without requiring every part to be designed in advance.
When the system exposed a specific problem, I studied the relevant theory and ran an experiment. The result determined what I added to the harness. This article describes six changes that followed that method.
Break a large job into small, recoverable jobs
One early Antilibrary task was to break each book into a small set of transferable ideas, claims, stories and frameworks. A prompt could do this for one book. The production job contained 703 books.
I began with a ten-book vertical slice. The system gathered the material available for each book, sent it to the model, read the response and saved the resulting ideas. Those ten books proved that the complete path worked before I committed to the full run.
I divided the 703 books into six ranges of roughly 117 books. Six workers processed those ranges at the same time, with each worker handling up to four books at once. Each worker wrote to its own partial file, so no worker could overwrite another. When they finished, the six files were combined.
The model had been instructed to return each answer in a fixed data format (JSON). This allowed the program to read the ideas and save them as structured records. In three cases, a misplaced quotation mark or line break broke that format. A person could still recognize the answer, but the program could not read it.
Three unreadable responses among 703 calls were enough to require a recovery method. I allowed the program to try again, up to three times, and taught it to recover the usable list from an imperfect response. A failed book was recorded and the worker continued. One unreadable answer could not stop the other books or erase work already completed.
This was the first useful shape of the harness: test the complete path on a small sample, divide the full job into bounded pieces, run those pieces in parallel, keep their writes separate, recover what can be recovered and make the remaining failures visible. The prompt extracted the ideas. The harness made the extraction repeatable across hundreds of books.
Make the model replaceable
Claude had classified all 1,232 books. The result was complete, orderly and plausible. Without a comparison, I had no way to know whether that apparent order reflected good classification or one model’s consistent choices.
I separated the classification task from the model that performed it. The harness owned the books, instructions, taxonomy and required output. A thin adapter translated that common task into the interface required by Anthropic, an OpenAI-compatible service or a model running locally on my computer.
Before receiving the full library, each model took a small qualification test. The test included deliberate traps: a genetics book by an Indian author had to remain Genetics rather than become India; a book about the Toyota production system had to remain Operations rather than become Japan. The model also had to use only categories that actually existed.
I then ran the same 1,232-book task through seven engines while holding the prompt, taxonomy and output format constant. In the dated July 11 comparison, a full run cost between 13 cents and $17.47. Price did not determine quality. The model with the broadest coverage differed from the one that produced the cleanest classifications.
Comparing models made disagreements visible. A second model did not establish which answer was correct, but it identified results that needed investigation. A difference could come from the model, the instructions, the available data, the classification structure or the way I measured quality.
The production system therefore owned the task and its output contract. The model occupied a replaceable slot. I could test capability, quality, time and price while leaving the rest of the workflow unchanged.
One system, seven models
× 7 engines
Give the checker independent evidence
I built a quality-control layer (Parakh, the Hindi word for discernment) to test whether Antilibrary’s findings could be traced to evidence. A different model acted as the judge. It received evidence retrieved separately from the material used to create the finding and returned one of three verdicts: grounded, inferred or unsupported.
Before trusting it on live work, I planted two obvious errors in the notes for 1984: a claim about photosynthesis and a fabricated chapter about yacht races. The checker rejected both and retained the real material. This test established that the checker could detect clear fabrication. I still needed to test whether its evidence and tools were complete.
An early audit examined 82 findings and flagged 23. On inspection, 20 of those flags came from the checker applying the wrong test. It had judged claims about the collection, such as its eras or absences, against the contents of individual books. I had to correct the evaluation method before I could use its verdicts.
The author-blind verdict from the opening exposed a second weakness. One grader and three skeptical agents all used the same search tool. The four calls were separate, but their method was the same. None could search by author, so all four reached the same false conclusion. Adding author data changed 5 of the 20 group verdicts when I ran the check again.
A checker is another AI system, with its own model, evidence, tools, rules and program logic. I now test it with known-good and known-bad cases. I inspect surprising verdicts against records it may not possess. I use another method when a shared tool could create a shared blind spot.
The harness also stops the automated process at a proposal. Agents may search, compare and recommend a correction. A human checkpoint is required before any change to the library or a published finding. The system can investigate freely because the consequential write remains gated.
Freeze the boundary before rebuilding the engine
Antilibrary’s website does not communicate directly with the models. The engine writes a small, fixed set of data files. The website reads those files to display books, ideas and findings. Together, the files form an output contract: the engine may change, but the website must continue to receive the same names, fields and values.
Before rebuilding the engine, I saved the exact live outputs and marked that version in the code history. I made an exact match against those files the definition of completion. This fixed the scope of the rewrite: the internal machinery could change while the website continued to receive the same data.
I then traced which part of the engine produced each file and which parts of the website actually used it. The task had been described as a rewrite of the whole output contract. The audit showed that most of it did not need rebuilding. Some files were already owned by the new engine, some were deliberately preserved history and some fields were no longer read. Only one file lacked a current producer.
That file revealed another weakness: it had been assembled by hand during an earlier curation session. No program could reproduce it. The new engine combined machine-generated facts with recorded human choices, checked that every book named in a finding existed in the evidence and then wrote the file in a fixed order.
The new file matched the live file byte for byte: all 34,209 bytes were identical on the first run. The website therefore received exactly what it had received before. After checking every remaining reference, I retired 19 old files and two obsolete commands.
The output contract gave the rewrite a testable completion condition. Mapping the files that the website actually used reduced the work. Exact parity showed that the rebuilt engine could replace the old one without changing the public output.
Make every model call auditable
I began with a telemetry ledger (Hisab, the Hindi word for account or reckoning). Its first version recorded the cost and duration of each model call. This showed where money went and which parts ran slowly.
The next layer added token-level telemetry. It recorded input tokens, reused input served from a cache, output tokens and thinking-token use where the provider reported it. It was like receiving a restaurant bill in stages. The first slip showed only the total. The second said how many starters, mains and drinks I had ordered. Only the final receipt named the dishes. The ledger could now reconstruct the charge, but it did not preserve the request or response. After 6,045 calls, I spent hours trying to infer where one stage’s output tokens had gone. An archived response answered the question in a minute.
The limitation came from the program’s structure. The telemetry wrapper had been handed a function that executed the model call. The request was sealed inside that function, where the wrapper could not see it. I changed every call site to pass the request as data. The response was archived as it came back.
Each new receipt now records the run, production step, caller, trigger, timestamp, model, tokens, cost, latency, outcome and stopping reason. It also points to the complete request and response. The large request and response bodies are stored separately and identified by a hash, a digital fingerprint derived from their content. Repeated material need not be saved twice. The older receipts remain readable and their missing fields remain visible.
The additional fields later exposed two problems. A stale price table had charged 95 Opus calls at three times the correct rate. Because the raw token counts were preserved, I could recalculate the history. The trigger field exposed a model stage with code, documentation and passing tests, but all five receipts came from test runs or experiments. Production had never called it. I removed the unused prompt and replaced its remaining work with fixed program logic.
By July 31, the ledger held 6,657 receipts. It could answer what ran, why it ran, what the model received, what it returned and how much the call cost.
Let records decide what runs
Once the telemetry ledger could record what had happened, I used those records to determine when each part of the system should run again. Adding one book should require work on that book and the findings it affected. It should not require the entire 1,250-book library to pass through every model again.
I built a reconciliation controller (Taal, the Hindi word for rhythm or beat). The telemetry ledger supplies its signals. The controller reads the system’s files and ledger, calculates what is due and routes the work to the appropriate engine or human checkpoint.
For example, one Antilibrary process assigns new books to the categories and recurring ideas that the full library has already established. The controller takes the complete inventory, subtracts the books covered by the last full run and subtracts the books covered by later update reports. The remaining set is the work that needs attention.
The controller reads several signals. File dates reveal stale outputs. Hashes, or digital fingerprints of files, reveal differences between the engine and website. Queue ages reveal waiting human decisions. Ledger timestamps reveal an outdated status board. Assumption records reveal engineering decisions that still need testing. The controller calculates each condition anew rather than maintaining a “last checked” date that could itself become stale.
The result is a status report in which each process is marked due or current. Free programmed work may run automatically. A paid model call is named and priced, then waits. Work requiring judgment goes to a human checkpoint. An informational watch remains observational until its decision rule exists. The controller makes no model calls and costs no tokens.
I first designed the controller as a scheduler. A clock would launch each process at a fixed interval. It gradually became a state-based reconciliation controller. The weekly clock now wakes it, but does not decide what to run. The controller compares current records with the state those records imply and determines what work is due.
Compare records · mark DUE or current
run
wait
checkpoint
observe
I tested the controller by changing an input file. Its dependent findings changed to due. The controller rebuilt them and the status report returned to current. These records now formed a control loop: sense, decide and act. The program could calculate which records had changed and limit reprocessing to the affected work.