The Next Token
RSS

Section

How it works

The machinery behind modern AI, explained from first principles with worked examples, diagrams and code.

Writing8 pieces
  1. No. 00817 min readDoes max_tokens count against your rate limit? On Anthropic no, on most other platforms yesA request that writes 400 tokens can use up 32,000 tokens of your per-minute limit before it starts, depending on where you send it. Here is what Anthropic, OpenAI, Azure, Amazon Bedrock, Google and Databricks each charge, and how to set the cap.
  2. No. 00717 min readWhy temperature 0 does not give the same answer twiceGreedy decoding is deterministic; the scores it chooses from are not, because they shift in their last bits with the size of the batch your request lands in. We measured the kernel-level effect on a laptop GPU, and checked what seeds, fingerprints and deterministic modes actually promise.
  3. No. 00622 min readWhy the same text is more tokens on one model than anotherWe counted the same 26 texts with six tokenisers. English prose moves by 5%; numbers, JSON and non-Latin scripts move by up to four times on current tokenisers. Anthropic says Claude Opus 4.7 and later add up to 35%; independent counts found up to 47%.
  4. No. 00518 min readHow much GPU memory a local model needs, and why a 4-bit model fills the cardWeights are the part everyone counts, and a 4-bit file is closer to five bits per weight. The KV cache grows in a straight line with context, but runtimes reserve it for the whole window at load, which is how a 2.3 GiB model took 91% of the memory our GPU could use before it read a word.
  5. No. 00424 min readHow AI text watermarks work, and what detecting one can proveThe mark lives in which words a keyed sampler picks, so finding it is a statistical test. We ran Google's open SynthID Text on a small open model with our own keys and measured how length, edits and paraphrase move the score.
  6. No. 00332 min readHow prompt caching works, from the KV cache to your billWhy one edit near the top of a conversation can re-bill the whole request, what September 2026 cache prices do to the cost of a miss, and why Anthropic and OpenAI now want settings changed by appending, not editing.
  7. No. 00227 min readReasoning effort: what the dial actually controls, and what it costsThe effort setting is an instruction about how much work to do, not a token allowance, and among request parameters only the output cap is a hard limit on tokens. Higher effort costs more per turn, so it saves money only if it cuts turns, retries or context growth by a margin you can compute, and the measurements so far mostly show turns and input rising.
  8. No. 00162 min readWhy new model APIs reject temperature, forced tool choice, prefill and edited historyClaude Opus 5.5, Sonnet 5.5, Haiku 5.5 and Fable 5.1 answer with 400s, OpenAI tells you which fields to remove for GPT-6 and GPT-6.1 Sol, and Google documents some settings as ignored on Gemini 3.8 Flash. Each old pattern did a job: here is what does that job now, and which reasons the vendors actually give.