Frontier language models—the most capable general-purpose models available—can write software, interpret difficult texts and reason across fields. Their range can obscure their limits. I found a much plainer task that Claude Sonnet 5 could not complete reliably: sorting statements that might mean the same thing.
The language model in this experiment was Claude Sonnet 5, a frontier LLM from Anthropic. It was the model we first asked to group the 160-stance test in one call and the model we later used on smaller candidate groups.
I was building Antilibrary from roughly 1,250 books. The source file contained about 5,700 extracted ideas. Another step turned those ideas into 3,989 short statements I called stances. A stance kept the claim while removing the book’s subject. “Growth beyond capacity destroys an empire” might become “growth beyond capacity causes collapse.” That made it possible to discover the same claim in a book about a company, a city or a forest.
The next job was to place stances that made the same claim into one group. Shared words were not enough. Two stances could use different words and mean the same thing. Two others could use almost identical words and make opposite claims. The model had to judge meaning.
Same practical claim
“Measure twice, cut once.”
“Careful preparation prevents costly rework.”The wording is distant. The meaning is close.
Opposite claims
“More choice improves decisions.”
“More choice impairs decisions.”The wording is close. The meaning points in opposite directions.
There were 3,989 stances, but there were far more than 3,989 decisions. Any stance might match any other stance. The complete set contains 7,954,066 possible pairs. The model would not necessarily check those pairs in a neat loop. Still, a correct answer had to remain consistent across that whole field of possible relationships.
We did not begin with all 3,989. We selected 46 books from five existing thematic groups whose contents we knew well. These groups gave us a familiar reference set: we could see whether the system rediscovered relationships already visible in the library. They were not an exact answer key because one broad thematic group could contain several distinct claims. The earlier language-model step that generated the stances also varied slightly between runs, producing between 155 and 170—roughly 160 stances.
That slice held about 4% of the full set. It still contained 12,720 possible pairs. The full problem had about 25 times as many stances and 625 times as many possible pairs. If the model could handle the full task, it should first have been able to handle this much smaller test.
Consider what the answer required. For each stance, the model had to decide whether it repeated another claim, merely discussed the same subject, contradicted it, or stood alone. It then had to build groups from those decisions. Every stance had to appear exactly once. No stance could disappear. No stance could belong to two groups.
The groups also affected one another. A new stance might fit an existing group. It might show that two groups should be joined. It might reveal that a group had combined two claims that should remain separate. A decision made later could therefore disturb a decision made earlier.
This is the flicker. The text itself did not change. What changed was the model’s view of how one text related to the others. The model was inferring those relationships inside a long, evolving generation. It had no separate table containing a settled comparison for every pair. Because it generated the groups sequentially, a later judgment could depend on—and conflict with—the grouping it had already written.
That instability showed up in a later repeat test, after we had divided the collection into small candidate groups. We ran the same final grouping prompt twice on eight of those groups. Six produced the same set of matches. Two did not. In one group, one run joined five stances; another found no matches at all. The original sentences were identical. The judgment changed.
At roughly 160 stances, this reduced test failed three times. With a 2,048-token output allowance—a limit on how much the model could return—the answer hit the ceiling. With 8,192 tokens, the model used the entire allowance for internal reasoning and produced no answer. With 32,000 tokens, the projected call became so long that the software client refused the non-streaming request because it risked running beyond ten minutes.
Together, these failures showed that the one-call approach was impractical. One model call had to read 160 stances, judge 12,720 possible relationships, revise the groups as those judgments interacted, remember every assignment and then write one exact answer. The model could interpret individual stances and compare small sets. It could not complete the entire grouping reliably in one call.
That was the wall. It appeared at about 4% of the real collection, before the model faced the remaining 3,829 stances and more than 7.9 million possible pairs.
A stable basis for comparison
To create a stable basis for comparison, we decided to use an embedding model.
An embedding model takes one piece of text and returns a vector: a fixed-length list of numbers. No single number carries a useful meaning by itself. Together, the numbers act like coordinates. They place the text at one location in a large mathematical space.
Each stance could now be processed on its own. In our setup, the same stance passed through the same model received the same coordinates. Adding another stance added another point; it did not require the earlier points to be interpreted again. Before any grouping began, every statement had a stable place from which comparisons could be made.
This changed the shape of the work. The frontier model had been asked to form thousands of relationships while holding the whole collection in context. The embedding model created an independent representation first. Once the coordinates existed, software could compare them with arithmetic. The reference frame stayed still even when the collection grew.
The training of an embedding model makes this geometry useful. Statements with related meanings tend to receive vectors that point in similar directions. We compared those directions with cosine similarity: the smaller the angle between two vectors, the higher the score. It was a measure of resemblance, not a probability that the statements were equivalent.
We used qwen3-embedding:8b, running locally through Ollama. Its size and local availability made repeated experiments inexpensive. We also chose it because it was designed to represent meaning in a form suitable for comparison at scale.
We tested that representation before relying on it. The first probe contained ten statements: five known paraphrases in two groups and five other statements, including a near-miss about the same subject that made a different claim. For each known paraphrase, we found its closest neighbour among the other nine statements. All five nearest-neighbour checks matched the expected group. Among the three statements about the pursuit of power, their mean similarity to one another was roughly 0.20 higher than their mean similarity to the near-miss. The individual known matches scored between about 0.70 and 0.93.
EMBEDDING MODEL TEST
Testing Qwen on ten labelled statements
We labelled ten statements, generated embeddings, and checked the nearest neighbours.
01 · CREATE THE TEST SET
Label the expected groups
Ten statements labelled before the run.
GROUP A · SAME CLAIM
GROUP B · SAME CLAIM
Four additional statements were unrelated distractors.
02 · GENERATE EMBEDDINGS
Embed each statement
Qwen converted each statement into a vector.
Only the statement text was sent to Qwen.
03 · CHECK THE RESULTS
Compare nearest neighbours
We checked each result against the expected groups.
known paraphrases had a nearest neighbour in the expected group
mean similarity gap: power paraphrases versus the same-topic near-miss
These results apply to this ten-statement test.
To turn the similarity scores into candidate groups, software processed the stances in order. It compared each new stance with one representative from every existing group. The stance joined the closest group if its score reached 0.60 and the group contained fewer than 30 statements; otherwise, it started a new group. Each stance entered one group.
We then tried it on the real slice. In one run, 161 stances became 89 candidate groups. On inspection, one ten-stance candidate group coherently expressed the idea of spontaneous order. Its statements came from different books and subjects but had cleared the same similarity threshold.
This was a small test. It did not prove that every valid match would be found, and it did not establish a universal score at which two statements become the same idea. It answered the practical question in front of us: did this model create useful neighbourhoods in which likely matches could find one another? The results justified the next step. A separate visual comparison with Google’s Gemini embedding model reinforced the same point: cosine similarity was useful for gathering candidates, but its scores could not tell us whether two related statements agreed or contradicted each other.
We gave the embedding stage a narrow responsibility: propose the neighbourhoods. The final judgment about whether two stances truly made the same claim would still belong to the frontier model. Meaning remained open to judgment; the search for candidates now had a fixed frame.
Using the frontier model on smaller groups
Before the embedding stage, the frontier model had two jobs inside one call. It had to search 160 stances for possible matches and then decide which matches expressed the same claim. It was trying to discover the candidates while also building one consistent set of groups across 12,720 possible relationships.
The embedding stage removed the search job from the frontier model. It processed each stance separately and returned a vector; software then assembled the candidate groups. The threshold was deliberately generous. A group could contain statements that did not belong together because the next step could reject them. A true match placed in a different group would be harder to recover.
The frontier model—still Claude Sonnet 5—then received the original sentences from one candidate group, rather than the vectors. Its task was limited to deciding which statements made the same claim. A group of 30 contains at most 435 possible pairs, compared with 12,720 among 160 stances. The model returned only the matches it judged genuine. Every statement absent from that merge list remained a one-statement group, so completeness and bookkeeping no longer depended on the model.
We ran this complete two-stage method on a 158-stance test. It finished without hitting the one-call wall. The largest candidate group sent to the frontier model contained nine stances, and the system produced seven themes supported by more than one book.
The frontier model could still make an unstable judgment. The difference was the reach of that instability. Candidate discovery was completed before each frontier-model call, each call covered one small group, and one uncertain decision could not reorganize the entire collection. We continued to use the frontier model for the part that required judgment, after giving it a smaller and bounded problem.
The solution came from separating two kinds of work. The embedding model handled the broad search for possible matches. The frontier model handled the narrower judgment about meaning.
Each model was used for what it could do reliably. The embedding model gave every stance a stable representation that software could compare at scale. The frontier model read language closely, but only after the field had been reduced to a small set of plausible candidates.
A more powerful model is not always the answer to a difficult AI problem. Sometimes the task has to be divided differently. In this case, the system began to work when one model found what was similar and another decided what that similarity meant.