Grade a claim about AI by how far its evidence has travelled from the person making it, not by how confident it sounds or who repeats it. The Next Token uses six rungs: rumoured (no checkable source), announced (the company says so), documented (it is written into the docs, price list or model card that customers rely on), available (you can use it, and say where and on what terms), measured (someone ran it, and you can say who, how and when) and independently reproduced (a second party got the same result). Each rung adds a check that someone other than the claimant can make. A claim sits at the highest rung its evidence has reached, and it keeps the name of whoever put it there.
The ladder is useful because launch-week claims arrive with all six mixed together. Take five from October 2026, each checked against its primary source on 20 October:
- A model called "Claude Next" being tested inside Claude Code is rumoured: one social post, as reported second-hand, no model ID, nothing from Anthropic.
- Claude Haiku 5.5 costing "around 75% less to run" than Haiku 4.5 is announced, resting on documented prices; on three illustrative workloads our script gives 35% to 90%, depending on prompt length.
- Gemini 4 Argon's million-token output is documented, with conditions attached: past 65,536 tokens the response comes in hops, run by Google's servers or by your own code, and input and thinking share the same million. We found no published measurement of a response that long.
- DeepSeek V4.1-Flash-1017 is available as open weights under the MIT licence. On Terminal-Bench 4.0, DeepSeek announces 42.3% with only the harness named; Vals AI, running every model in one harness, measured 31.82%.
- Vals AI's finding that two-thirds of the coding tasks Xiaomi released for MiMo v2.6 leak the answer is measured by Vals. Xiaomi re-pushed the task images nine days later, on 16 October; we found no published reproduction of the count.
The six rungs of the evidence ladder
Each rung answers a different question, and each leaves something unproved.
| Rung | What exists | Who can check it | What it still does not tell you |
|---|---|---|---|
| Rumoured | A post, a leak, a screenshot, a "sources say" | Nobody, beyond whether the source exists | Whether the thing exists |
| Announced | The company's own statement: a blog post, a launch table, a keynote | Anyone can check that it was said | Whether it is true as stated, under which conditions |
| Documented | Docs, API reference, price list, model card, licence | Anyone can read the terms the company commits to | Whether it works as written for you |
| Available | A model ID that answers, weights you can download, a plan you can buy | Anyone with access can try it | How well it works |
| Measured (by whom) | A result with method, settings, version and date | Anyone can read the method; few can rerun it | Whether the result survives a different measurer or setup |
| Independently reproduced | A second party, with no stake, got the same result | Anyone can compare the two | Whether it holds under conditions neither tested |
Three points keep the ladder honest.
First, the rungs describe kinds of evidence, not an order of events: open weights can be available before anyone documents them, and a model can be documented weeks before it is available to you.
Second, one announcement often holds several claims at different rungs. Anthropic's Haiku 5.5 launch calls it "the cheapest, fastest, and most capable small model we've ever released" and says "On average, it now costs around 75% less to run." Together those contain a documented price, an announced comparison with Anthropic's own earlier models, an announced average saving and a speed claim whose condition sits in a footnote: fastest "at each model's standard speed, although it runs less quickly than our Opus models in Fast Mode". Split the sentence before you grade it.
Third, a company is the authority on what it announced, what it documented and what it charges. It is not the authority on whether its product beats a competitor's, however many decimal places the launch table carries. The rule we apply: a vendor's number about its own model, published with its method, settings, version and date, is "measured, by the vendor"; with part of that missing, it stays "announced", with a partial method; and its ranking against a rival stays "announced" either way.
Rumoured: a claim with nothing to check
We name two rumours here only as checked examples; normally we would not publish them. According to OrcaRouter, which traced the claim to one post on X on 11 October 2026 that we could not open, the post said a model called "Claude Next" was being tested inside Claude Code and made Claude Opus 5.5 look primitive. OrcaRouter found "no model ID, no rate card, no documentation row, no model card, no arena listing, and no statement from Anthropic". A day later a YouTube video whose title begins "Claude Opus 6 LEAKED" made a similar claim; AI Tools Review, working from the video's description, found no reliable reporting behind it.
What can be checked is absence. On 20 October 2026, Anthropic's models overview listed four current models (Claude Fable 5.1, Opus 5.5, Sonnet 5.5 and Haiku 5.5), its pricing page had no other new entry, and none of the 468 models in OpenRouter's model list was a Claude Next or an Opus 6. Absence does not disprove a rumour, since companies test models they have not released; it means there is nothing to grade yet. Only a statement from the company moves a rumour up.
Checking can move a claim up as well as down. When we planned this essay, OpenAI's "Dots" was on our list as an example of a rumour that appeared only on aggregator sites. It is not one. OpenAI documents dots as always-on agents running on GPT-6 Astra, with eligibility set by plan, age and region (Pro plans for users over 18 outside the European Economic Area, the United Kingdom and Switzerland; Business Premium and Enterprise worldwide). The same page shows the gap between documented and available: "Dots are rolling out gradually. You may not see dots immediately, even if your plan is eligible." Had we graded from the aggregators, we would have published a mistake about our own subject.
Announced: Claude Haiku 5.5 and "around 75% less to run"
Anthropic announced Claude Haiku 5.5 on 7 October 2026 with the superlatives quoted above and an average saving over Claude Haiku 4.5. Neither of those is the price.
The price is documented. Anthropic's pricing page, read on 20 October 2026, lists claude-haiku-5-5 at $0.10 per million input tokens and $0.50 per million output tokens for prompts up to 100,000 tokens, and $0.50 and $2.50 for longer prompts. Claude Haiku 4.5 is listed at $1 and $5. OpenRouter lists the same two tiers for anthropic/claude-haiku-5.5. Per token, that is 90% cheaper for short prompts and 50% cheaper for long ones.
Anthropic's footnote says what went into the 75%: the 90% and 50% cuts, the fact that 90% of requests to Haiku 4.5 were under 100,000 tokens, and Haiku 5.5's updated tokeniser, which "uses slightly more tokens per task". That is an outline of the inputs: a split of requests by count but not the token-weighted mix that sets the bill, no tokens-per-task figure and no date. Without method, settings, version and date it does not qualify as "measured, by the vendor", so it stays at "announced", and it describes the average customer rather than you.
Your own figure depends on your prompt lengths. Anthropic's pricing page says its newer tokeniser "produces approximately 30% more tokens for the same text", and our explainer on why token counts differ shows how much that varies by content. Tokens per task also depend on how many turns the model takes and how much it writes, which is why Anthropic can say "slightly more", so treat the 30% column below as a stress case, not Anthropic's figure. Our script applies the documented prices to three invented workloads, counted in Haiku 4.5 tokens:
| Illustrative workload | Cheaper on Haiku 5.5, same token count | Cheaper with 30% more tokens |
|---|---|---|
| All prompts 5,000 tokens in, 500 out | 90.0% | 87.0% |
| 90% of requests at 10,000 in and 1,000 out, 10% at 150,000 in and 2,000 out | 68.3% | 58.8% |
| All prompts 85,000 tokens in, 1,000 out | 90.0% | 35.0% |
In the third row, an 85,000-token prompt becomes about 110,000 tokens under the newer tokeniser and crosses into the higher tier. In the second, a few long requests carry most of the tokens, so a 90% cut per token becomes a much smaller cut per request. Neither is Anthropic's traffic; they show that the 75% is a property of a mix.
The capability claim is about Anthropic's own models, but the benchmark table beside it lists GPT-6 Luna too, and that column has mixed provenance like the one dissected below. The Haiku 5.5 system card says Luna's 16.4% on Terminal-Bench 4.0 comes from "the public leaderboard", scored "with the Codex CLI", and that "OpenAI has not reported its own Terminal-Bench 4.0 scores" for its latest models. Anthropic's own rows were run in its Claude Code harness; the Luna row was not.
To move the 75% up for yourself, replay a day of your requests through the token counting endpoint for both models and price the result: a measurement, by you, on your traffic.
Documented: Gemini 4 Argon's million-token output, and the conditions attached
Google's Gemini API changelog records Gemini 4 Argon (gemini-4-argon) as generally available on 13 October 2026. Its model page, read on 20 October, gives a maximum output of "65,536 (default synchronous) / up to 1,048,576 (with long decode)". The headline is documented; so are the conditions, which matter more.
Google's long decode guide says a single model call generates up to 64K tokens and documents two ways past that. On the Interactions API, a request with background: true and max_output_tokens above 65,536 has Google's servers generate the response "in 64K hops" for up to 24 hours, which you poll for or stream. On the GenerateContent API, "client-side continuation" has your code start each hop, at "64K per stream, 32K per non-streaming request", within 20 minutes. An interaction with background: false that asks for more than 65,536 tokens returns a 400 error. The million is "shared across input, thinking, and output", so the room for the answer is 1,048,576 tokens minus your input and thinking. Other routes set other limits: on 21 October 2026, OpenRouter's endpoints for google/gemini-4-argon listed 943,718 tokens on most providers and 65,536 on one, and Vals AI evaluated with a 262,144-token cap. We did not test which route reaches what.
The price is documented with an end date. Google's pricing page lists $2 per million input tokens and $10 per million output tokens (thinking tokens included) "through November 30, 2026", then $4 and $20 from 1 December. A sentence that says "Argon costs $2 and $10" is true until 30 November 2026; without the date it becomes false on its own. On 20 October, Vals AI's model page already listed $4 and $20. The output price is not the whole bill for a long response either: the guide says each hop "sends your prompt plus everything generated so far back to the model as input" and "You pay for every token in every hop", with implicit caching taking 95% off the repeated input. A maximum-length response from an empty prompt takes 16 hops and bills about 7.86 million input tokens along the way. Our script puts it at $11.27 to $13.14 at introductory prices ($10.49 of it output), depending on whether the hop just generated is billed at the cached rate on its first resend, and $22.54 to $26.28 from 1 December.
The million-token claim sits at documented, on a model Google lists as generally available on the paid tier (its pricing page says Argon is not on the free tier). We found no published measurement of a response near that length: nobody reporting how long it took, what it cost, or whether the end of the text still followed from the start. That measurement, published with the request, the settings and the output, is what would move the claim up a rung.
Available: DeepSeek V4.1-Flash-1017 and a model name that moved
DeepSeek-V4.1-Flash-1017 is the strongest kind of available. Its weights, 48 files totalling about 510 GB, are on Hugging Face under the MIT licence, in a repository created on 17 October 2026, and DeepSeek's release note of the same date put it on the website, app and API. Anyone with the hardware can run it and test a claim without asking anyone's permission.
Available also changed what a familiar name means. DeepSeek's Your First API Call page says the model name deepseek-flash "is now served by DeepSeek-V4.1-Flash-1017", and its models and pricing page says the retired names deepseek-v4-flash and deepseek-v4-flash-vision-exp are still accepted but their "requests are served by the DeepSeek-V4.1-Flash-1017 model". A test run through deepseek-flash on 16 October would have measured a different model from the same test on 18 October, with nothing in the request to show it. An alias is a pointer the vendor can move, so a claim needs the exact identifier and the date.
Available measures nothing. DeepSeek's release note calls the update "a Pareto improvement in performance and cost", and the model card offers a table: Terminal-Bench 4.0 at 42.3 against 31.2 for the previous release, DeepSWE v1.1 at 77.0 against 74.2. A footnote says DeepSeek used "the Claude Code harness for Terminal-Bench 4.0 and the Mini-SWE-V2 harness for DeepSWE v1.1". That footnote is the whole method: no effort setting, number of runs or date. By the rule above, these are announced numbers with a partial method, made in harnesses DeepSeek chose, on its own model.
Measured, and by whom: reading a launch benchmark table
A launch benchmark table looks like one measurement and can be many, made by different people. The table on the Gemini 4 Argon model card has 19 rows comparing Argon with GPT-6 Astra, Claude Fable 5.1 and Claude Opus 5.5. Google's evaluation methodology says, row by row, where each number came from, and says that "All the results for non-Gemini models are sourced from providers' self reported numbers unless otherwise mentioned".
Sorting the rows by who produced Argon's own score:
- In nine rows, Argon's number comes from someone else: Vals AI, Zapier's AutomationBench leaderboard, Proximal's FrontierSWE leaderboard, Surge's RiemannBench and Chartography leaderboards, and the CWE-bench leaderboard. The rival models in those rows come from the same source.
- In five rows, Google ran Argon and every rival itself: PostTrainBench, LABBench 2, both GraphWalks rows and LVBench. For LVBench the methodology notes different inputs per model: "1FPS for Gemini, 800 frames for GPT-6 Astra and 300 frames for fable 5.1 and 600 for opus 5.5 due to API limitations."
- In five rows, Google ran Argon and took the rivals' numbers from elsewhere: DeepSWE v1.1 (Argon "self computed, using a mini-swe agent harness"; Fable 5.1 and Opus 5.5 "taken from their respective system cards"), Terminal-Bench 4.0, Terminal-Bench Science 0.1 (Argon run "with 6x verifier timeout"), Agent's Last Exam and OSWorld-2.0 (Argon's score "maxed over 3 runs", GPT-6 Astra's "taken from the official blog post").
Google published all of this, but the last five rows set a number Google measured in its own setup beside numbers others measured in theirs, and one column of bold figures does not show it. A leaderboard source is not automatically independent either: some leaderboards run every model themselves and some accept submissions, so check which before treating a row as a third party's measurement.
A measured number also has a date. Google's table gives Argon 68.9% on the Vals Index, sourced from Vals AI. That matches Vals' own run on 30 September 2026, two weeks before general availability, when Argon placed first. By 19 October the Vals Index, with some components on newer versions, listed Argon third at 69.24%, behind Claude Sonnet 5.5 (70.85%) and Opus 5.5 (70.18%), whose scores include some tasks answered by a fallback model after a provider refusal; counting those as failures gives 69.03% and 64.16%, and Argon first. "Argon leads the Vals Index" was true on 30 September, and whether it was true on 19 October depends on a scoring rule. Google's table also chose its comparison columns: Claude Sonnet 5.5, second on Vals' 30 September run, is not in it.
Why benchmark numbers from different harnesses are not comparable
The same benchmark name does not mean the same measurement. A harness is the agent scaffolding around the model: the tools it gets, the prompt, the step and time limits, the retries. On agentic benchmarks it is part of what is being measured, so two scores for the same model in different harnesses are two different experiments.
Terminal-Bench 4.0 shows the size of the effect. Vals AI runs it on 66 tasks with every model in one harness, mini-swe-agent, which gives the model a single bash tool and "no step or cost limit", with an 8-hour time limit per task. Its scores are avg@3: three full runs, each scored pass@1, averaged, with error bars of one standard error across the three runs. Its model pages list "max" effort for the Claude and DeepSeek models (though the DeepSeek V4.1-Flash page's prose says "high") and "high", the top setting, for Argon. The developers' figures sit at different rungs. Anthropic's are measured, by the vendor: the system card says they "use Claude Code in --bare mode at max thinking effort", averaged over 10 trials per task for Haiku 5.5 and five for Sonnet 5.5, with standard errors of ±1.9 and ±2.5 points. Google's 57.4% is self-computed at the highest thinking setting with no harness named, and DeepSeek's name the harness and nothing else, so both are announced, with a partial method. Here are Vals' results as of 19 October 2026 beside each developer's figure:
Developer figures from the Claude Haiku 5.5 system card (measured, by the vendor) and the Gemini 4 Argon and DeepSeek V4.1-Flash-1017 model cards (announced, with a partial method). Vals figures updated 19 October 2026; shaded bands are one standard error across its three runs.
Error bars decide which gaps mean anything. Google's 57.4% for Argon and Vals' 57.58% differ by 0.18 points against Vals' standard error of 2.31. Anthropic's 39.2% for Haiku 5.5 sits 2.84 points above Vals', less than one combined standard error of the two parties (2.99 points), so those two agree too. Sonnet 5.5's gap of 6.46 points is 2.4 combined standard errors, which is a real difference; Anthropic also notes that 1.5% of Sonnet's trials touched a fallback model, the issue Vals flags on its index. DeepSeek's figures, made with the Claude Code harness, sit 10.48 points above Vals' for V4.1-Flash-1017 and 11.50 points above for V4.1-Flash, so on Vals' run the older model loses more than a third of its reported score.
A gap is not evidence that anyone misreported. A developer that picks the setup its model does best in reports a legitimate number, but a different one from what the model does in a shared, plain setup, and only the shared setup supports a ranking. The rule follows: compare numbers only when one party measured all of them the same way, or when the harness, settings and version match and are written down.
The same rule stops a tempting comparison on DeepSWE v1.1. Google reports 77.9% for Argon in "a mini-swe agent harness", DeepSeek reports 77.0% for V4.1-Flash-1017 in "the Mini-SWE-V2 harness", and Google's table quotes 74.2% for Claude Opus 5.5 from Anthropic's system card. Three measurers in three setups do not rank three models.
Independently reproduced: the rung most claims never reach
Reproduction needs someone with no stake to repeat the work and get the same answer. The nearest October case is Haiku 5.5 on Terminal-Bench 4.0: Anthropic published its full setup, and Vals, in a different harness, landed within both parties' stated errors. That is a vendor measurement surviving a second measurer and a second setup. Argon's figure agrees as closely, but with Google's harness unnamed it is agreement with an announced number, not a repeat of a known procedure. The MiMo audit shows why the rung is hard to reach even in good faith.
Xiaomi's MiMo-V2.6 technical report says its red-team screening of the coding environments continued "until the hack agent could no longer find a successful exploit in any of the environments", and that "the logged confirmed-hack share remains below 2%". That is Xiaomi measuring its own defences with its own detector.
Xiaomi also released the environments, so outsiders could check. On 7 October 2026, Vals AI published an audit of the released coding tasks: "We audited all 2,698 coding tasks and found that in 1,795 of them (67%), the fix commit survives as unreachable objects", where a model that knows Git can read it. Vals also reported MiMo v2.6 Flash finding such a fix in one task and passing the hidden tests. A named third party with its method described puts the claim on the "measured" rung. For the behavioural findings Vals gives the counts, such as six runs per prompt on one task, and they are small.
Since then, what Vals measured has been replaced. All 2,602 format-code-task images on Docker Hub were re-pushed under the same tag names between 14:43 and 15:05 UTC on 16 October, nine days after Vals published (checked 20 October), and the tag for one dropped task, format-code-task-000014, no longer exists. On 18 October a mimoagent commit titled "code dataset: purge task git history at setup; reset test files without hooks; trim comments" added a step that deletes later refs and prunes unreachable objects, and the released dataset dropped 96 tasks: we counted 2,698 coding tasks at the 26 September revision and 2,602 on 20 October. On 20 October its README carried an "Update (2026-10-19)" saying the code images "now ship with anti-hack preprocessing already applied". Neither mentions Vals or counts affected tasks.
So the claim stands at "measured by Vals AI", and Xiaomi's changes are consistent with it without confirming the number. A reproduction, which we have not found, would be a third party rerunning the audit on the images as they were before 16 October and reporting its own count. The fix makes that harder: the tags were overwritten in place, so pulling a tag after 16 October fetches the image pushed that day, which Xiaomi says has the preprocessing applied, and a reproducer needs the original image digests, not the tag names. A result can only be reproduced against the same version, so the measurement has to name it and the reproducer has to use it. On 20 October 2026 we read 24 of the current images ourselves, which checks Xiaomi's fix rather than Vals' count: in our audit of coding benchmarks that leak their answers, their Git stores were clean, but in 10 of the 24 the newest file times covered every file the hidden tests edit.
Partial reproduction counts if you name the part. Our explainer on temperature 0 reproduced Thinking Machines Lab's kernel-level mechanism, batch size changing the arithmetic, on an Apple M2 Pro, and said plainly that it ran no language model, so it did not reproduce their 80 different completions from 1,000 identical requests.
How The Next Token labels evidence and handles corrections
Every post here follows the rules this essay applies. For a reader, they mean:
- Each claim carries its rung in the wording, from "Google says" for an announcement to "measured by Vals AI" or "we measured". Rumours are normally not published.
- Each post carries one evidence label, the lowest one that is honest. Hands-on means we ran or measured something ourselves. Desk-researched means the post rests on primary documentation and attributed external testing, and is never written as if we had used the product. Editorial means stable concepts or a view, with no testing claim. Our posts on why token counts differ and on temperature 0 are hands-on; our guide to whether max_tokens counts against your rate limit is desk-researched, and dates a documented rule change from archived copies of Anthropic's page. This essay is desk-researched: our only first-hand measurement is a count of rows and image tags in Xiaomi's public release, and we ran no model.
- Each post ends with "How this was made": the sources, and what we ran, measured or calculated, with dates. It never claims a run or first-hand use that did not happen.
- Claims are specific. Model claims name the exact identifier, the date checked and the conditions (plan, region, API or app, settings); prices carry currency, unit and date; benchmark numbers name the version, who ran it, the model, the setup, the date and whether the vendor reported it. We do not compare numbers from different harnesses as if they were like for like.
- Anything a reader could recompute comes from a script kept in our test library, with the inputs and outputs of any run, failures included.
- Each post has a freshness class that sets when it is due for review: 90 days for model capabilities, prices, API behaviour and news, a year for workflows, comparisons and tooling, two years for mechanisms. This essay is in the 90-day class, because its examples are prices and model behaviour.
- Corrections are dated. A wrong published fact gets a dated correction entry on the post saying what was wrong and what it says now. Typo fixes do not.
A checklist for the next AI announcement
Before you act on a claim about AI, decide which rung your decision needs, then check whether the claim has reached it. Building against an API needs documented and available. Choosing between two models needs measured, by one party, in one setup. Staking an argument or a budget on a surprising result needs reproduced. For the claim in front of you:
- Find the primary source. If there is none, it is a rumour; wait.
- Split the sentence. Price, availability, capability and comparison are separate claims and usually sit on different rungs.
- Pin the identifier and the date. Note the exact model ID, not an alias, and when you checked.
- Read the conditions. Which interface, plan, region and settings does a limit or feature apply to? A million tokens through one route can be 65,536 through another.
- Date the price. Introductory prices end, and tiers change what you pay per request.
- Ask whose traffic a percentage describes. "75% cheaper on average" is someone's mix; work out yours.
- Ask who ran each benchmark number, and in what harness. In a launch table, ask it row by row, and ask which benchmarks were left out.
- Treat "beats X" as the vendor's view until one party has measured both models the same way.
- Look for the second measurement, and use the error bars. A gap smaller than the two parties' combined standard error, as with Haiku 5.5 on Terminal-Bench 4.0, is agreement; a gap several times larger tells you how much of the number belonged to the setup.
The decision rule is short: state a claim at the highest rung its evidence has reached, name who put it there, and do not let a number from one rung borrow the authority of another.