The Next Token
RSS

An independent publication about AI

Artificial intelligence, one token at a time.

The news that matters, how the technology actually works, the research behind it and the products worth your time. Reported from primary sources and written to be understood.

Start reading Subscribe by email
next_token()step 1/5
Illustration of next-token sampling. Probabilities are invented for display, not taken from a real model.
Lead storyNo. 003

How it works 32 min read

How prompt caching works, from the KV cache to your bill

Why one edit near the top of a conversation can re-bill the whole request, what September 2026 cache prices do to the cost of a miss, and why Anthropic and OpenAI now want settings changed by appending, not editing.

Read the story

The index

12 pieces
  1. No. 012Research23 min readCoding benchmarks that leak their answers, and which scores to trustVals.ai found the fix still readable in two-thirds of the coding tasks Xiaomi released for MiMo-V2.6, and MiMo found it. We pulled 92 task images from SWE-bench Verified, SWE-bench Pro and Xiaomi's release to see which leaks are closed as of 20 October 2026, and collected what happens to scores when they are.
  2. No. 011News16 min readEU AI Act Article 50: who must mark output from apps built on Claude, GPT and Gemini, and by whenMarking has applied since 2 August 2026 to generative AI systems launched from that date; 2 December only ends a grace period for systems already on the EU market. We read the regulation, the amending act, the Commission guidelines and the Code of Practice, and checked what Anthropic, OpenAI and Google mark by default.
  3. No. 010Products20 min readSix cheap models tested on extracting data into a JSON schemaWe sent invoices, support emails, job ads and addresses to six low-cost models through OpenRouter, three runs each, with strict JSON-schema output. All 3,120 responses matched the schema; what separated the models was planted instructions, reasoning settings and, for the open model, which host served the request.
  4. No. 009Notes21 min readHow to grade a claim about AI, from rumour to reproductionFour of the five launch-week claims checked here stop below independent measurement, and a launch benchmark table mixes numbers measured by different people in different harnesses. This is the ladder The Next Token grades claims on, worked through five claims from October 2026, with what would move each one up.
  5. No. 008How it works17 min readDoes max_tokens count against your rate limit? On Anthropic no, on most other platforms yesA request that writes 400 tokens can use up 32,000 tokens of your per-minute limit before it starts, depending on where you send it. Here is what Anthropic, OpenAI, Azure, Amazon Bedrock, Google and Databricks each charge, and how to set the cap.
  6. No. 007How it works17 min readWhy temperature 0 does not give the same answer twiceGreedy decoding is deterministic; the scores it chooses from are not, because they shift in their last bits with the size of the batch your request lands in. We measured the kernel-level effect on a laptop GPU, and checked what seeds, fingerprints and deterministic modes actually promise.
  7. No. 006How it works22 min readWhy the same text is more tokens on one model than anotherWe counted the same 26 texts with six tokenisers. English prose moves by 5%; numbers, JSON and non-Latin scripts move by up to four times on current tokenisers. Anthropic says Claude Opus 4.7 and later add up to 35%; independent counts found up to 47%.
  8. No. 005How it works18 min readHow much GPU memory a local model needs, and why a 4-bit model fills the cardWeights are the part everyone counts, and a 4-bit file is closer to five bits per weight. The KV cache grows in a straight line with context, but runtimes reserve it for the whole window at load, which is how a 2.3 GiB model took 91% of the memory our GPU could use before it read a word.
  9. No. 004How it works24 min readHow AI text watermarks work, and what detecting one can proveThe mark lives in which words a keyed sampler picks, so finding it is a statistical test. We ran Google's open SynthID Text on a small open model with our own keys and measured how length, edits and paraphrase move the score.
  10. No. 002How it works27 min readReasoning effort: what the dial actually controls, and what it costsThe effort setting is an instruction about how much work to do, not a token allowance, and among request parameters only the output cap is a hard limit on tokens. Higher effort costs more per turn, so it saves money only if it cuts turns, retries or context growth by a margin you can compute, and the measurements so far mostly show turns and input rising.
  11. No. 001How it works62 min readWhy new model APIs reject temperature, forced tool choice, prefill and edited historyClaude Opus 5.5, Sonnet 5.5, Haiku 5.5 and Fable 5.1 answer with 400s, OpenAI tells you which fields to remove for GPT-6 and GPT-6.1 Sol, and Google documents some settings as ignored on Gemini 3.8 Flash. Each old pattern did a job: here is what does that job now, and which reasons the vendors actually give.

How we work

Read closely. Explain first.

  1. 01

    Primary sources first

    Papers, model cards, changelogs and documentation, linked in the text so you can check our reading against the original.

  2. 02

    Known, claimed, guessed

    We keep what has been shown apart from what has been announced, and say plainly when we are speculating.

  3. 03

    Mechanism before verdict

    Before we judge a model or a product, we explain how it works. Opinions are easier to weigh once the machinery is visible.