The same text becomes a different number of tokens on different models because each model family splits text with its own tokeniser, and tokenisers are built from different vocabularies with different rules. On English prose the difference is small: across six tokenisers we measured on 13 October 2026, the same 1,012 news sentences came to between 26,878 and 28,126 tokens, a spread of 5%. On other text the gap is large. A CSV of numbers took 1.67 times as many tokens on Gemini's tokeniser as on OpenAI's, the same sentences in Amharic took 2.92 times as many on OpenAI's current tokeniser as on Gemini's, and the same sentences in Bengali and Tamil took 4.13 times as many on Qwen3's as on Gemini's. (By "Gemini's tokeniser" we mean the file Google's own SDK uses to count tokens for Gemini models from 2.0 to 3.5 Flash.) Because APIs bill per token, those ratios go straight onto the bill: two models at the same list price can charge very different amounts for the same prompt.
The best-known case is inside one vendor. Anthropic says Claude Opus 4.7 and later models use a tokeniser that turns the same input into "roughly 1.0–1.35×" as many tokens as Claude Opus 4.6, and Opus 4.7 launched at the same $5 per million input tokens. Independent counts of that change have found up to 1.47 times on some content, and one put Claude's current tokeniser at 1.40 times OpenAI's on English prose. This piece explains why counts differ, gives our measurements by content type and by language, works out what the differences do to the price per character, and ends with a rule for predicting the bill before you switch models.
One line through six tokenisers
A tokeniser turns text into a sequence of integer IDs from a fixed vocabulary, and the model is billed for the length of that sequence. Here is how six tokenisers split three short strings. The counts are from our test on 13 October 2026; "Gemini" is the Gemma tokeniser file that Google's own SDK uses to count tokens locally for Gemini models from 2.0 to 3.5 Flash, explained below; we did not check it against Google's API, and it does not cover Gemini 3.8 Flash. cl100k_base is GPT-4's tokeniser, kept as a legacy reference.
| Text | cl100k (legacy) | o200k | Gemini | Qwen3 | DeepSeek V3.1 | Mistral Nemo |
|---|---|---|---|---|---|---|
The tokeniser splits text into pieces. |
8 | 8 | 8 | 8 | 8 | 8 |
total_usd: 2750.26 on 2026-01-01 |
17 | 17 | 24 | 24 | 17 | 24 |
मानव अधिकार (Hindi, "human rights") |
13 | 3 | 3 | 11 | 7 | 3 |
The English sentence is eight tokens everywhere, and every tokeniser splits it the same way: The, token, iser, splits, text, into, pieces, .. Common English words are whole tokens in every vocabulary.
The line of numbers splits two ways. OpenAI's tokenisers and DeepSeek's cut digit runs into groups of up to three (275, 0, 202, 6); Gemini, Qwen3 and Mistral give every digit its own token (2, 7, 5, 0). Same text, 17 tokens or 24.
The Hindi phrase splits three ways. o200k, Gemini and Mistral Nemo have whole Devanagari pieces (मान, व, अधिकार). DeepSeek has smaller ones. cl100k and Qwen3 have few Devanagari entries beyond single letters, so many characters split into byte pieces: the phrase is 11 characters, including the space, and 31 bytes, and it comes to 13 tokens on cl100k and 11 on Qwen3.
Those three behaviours, which words are in the vocabulary, how digits and spaces are pre-split, and what happens to characters the vocabulary lacks, explain nearly all of the differences measured below.
Why tokenisers give different counts for the same text
Almost every current model uses a subword tokeniser trained with byte-pair encoding (BPE) or a close relative. Training starts from single bytes or characters and repeatedly merges the most frequent adjacent pair in a training corpus into a new vocabulary entry, until the vocabulary reaches a chosen size: 100,277 entries for cl100k_base, 200,019 for o200k_base, 262,144 for the Gemma 3 and Gemma 4 files, 151,669 for Qwen3, 128,815 for DeepSeek V3.1 and 131,072 for Mistral Nemo (from the tokeniser files, counting special tokens). At encoding time, text is first cut into chunks by a pre-tokenisation rule, then each chunk is covered with the longest vocabulary entries the merges allow. Three design choices then drive the count.
The training mix decides which text gets long tokens
A tokeniser gives long, single-token entries to whatever was frequent in its training corpus. Text that resembles that corpus compresses well; text that was rare is spelled out in short pieces. That is why every tokeniser we measured handles English prose within 5% of the others, and why they diverge on Hindi, Tamil or Amharic.
Vocabulary size helps only when the extra entries go to the text you send. o200k_base has about twice cl100k_base's vocabulary, and OpenAI spent much of the increase on other scripts: on our 1,012 sentences, Tamil fell from 207,778 tokens on cl100k to 53,207 on o200k, 3.91 times fewer, and Hindi fell 3.07 times. OpenAI's GPT-4o announcement gave 3.3 times for Tamil and 2.9 times for Hindi on a single sample sentence each; our larger sample points the same way, with bigger ratios. Qwen3, with 151,669 entries, still needs 4.43 times as many tokens for Hindi as for the same sentences in English, while Mistral Nemo, with fewer entries, needs 1.80 times. Size is a weak predictor; allocation is what counts.
Splitting rules for digits and whitespace
Before any merging, most tokenisers cut text with a pre-tokenisation rule, and the rule for digits is a deliberate design choice. OpenAI's cl100k_base and o200k_base patterns, in tiktoken's source, match \p{N}{1,3}: digit runs are cut into groups of at most three. Google's PaLM paper and Meta's LLaMA paper instead split every number into individual digits, and the Gemini, Qwen3 and Mistral Nemo tokenisers we tested do the same. Single digits make arithmetic more regular for the model, and Singh and Strouse (2024) found that the direction digits are grouped in changes arithmetic accuracy, but single digits cost tokens: our 12,506-character CSV of prices and dates is 7,321 tokens on o200k and 12,215 on Gemini's tokeniser.
Whitespace and punctuation rules matter for code and JSON. Gemini's tokeniser keeps brackets and following identifiers apart ([, key, ] where the others have [key and ]), which is one reason it needs 19% more tokens than o200k for our TypeScript file and 28% more for pretty-printed JSON.
Byte fallback for characters outside the vocabulary
Byte-level tokenisers never fail on unknown text: a character with no vocabulary entry is encoded as its UTF-8 bytes, one token per byte or a few bytes per token. A Latin letter is one byte, but Devanagari, Tamil and Ethiopic characters are three bytes each, so a tokeniser that has not learnt a script pays up to three tokens per character. Amharic, written in Ethiopic script, shows the extreme: the same 1,012 sentences are 53,564 tokens on Gemini's tokeniser and 207,673 on Mistral Nemo's, 3.88 times as many. Petrov et al. (2023), measuring FLORES-200 across many more languages, found differences of up to 15 times between languages on the same tokeniser.
A tokeniser that uses fewer tokens is not automatically better. Schmidt et al. (2024) found that fewer tokens does not by itself make for a better model, and Dagan et al. (2024) found that compression gains shrink as vocabularies grow. Vendors trade token count against model quality, and sometimes, as Anthropic did with Claude Opus 4.7, they choose more tokens.
How much token counts differ by content type
Across the five current tokenisers we measured, the count for the same English text differs by 5% for prose, 9% for Markdown, 12% to 19% for code, 25% to 28% for JSON and 67% for a CSV of numbers. The legacy cl100k is slightly lower than all of them on code and JSON, so including it widens those ranges by a point at most. The table gives tokens per 1,000 characters, which is the number to multiply by a per-token price.
| Sample (characters) | cl100k (legacy) | o200k | Gemini | Qwen3 | DeepSeek V3.1 | Mistral Nemo | Highest ÷ lowest, current |
|---|---|---|---|---|---|---|---|
| English prose (132,978) | 204 | 202 | 212 | 208 | 203 | 209 | 1.05 |
| Markdown article (46,897) | 243 | 242 | 264 | 254 | 245 | 256 | 1.09 |
| Python module (11,039) | 266 | 268 | 299 | 267 | 280 | 279 | 1.12 |
| TypeScript module (11,298) | 257 | 258 | 307 | 259 | 270 | 271 | 1.19 |
| JSON, minified (6,062) | 268 | 270 | 338 | 281 | 293 | 298 | 1.25 |
| JSON, pretty-printed (6,683) | 295 | 296 | 378 | 306 | 321 | 321 | 1.28 |
| CSV of numbers (12,506) | 585 | 585 | 977 | 956 | 606 | 977 | 1.67 |
The prose is the English side of the FLORES-200 devtest set (news and encyclopaedia sentences); the Markdown is our own article on how prompt caching works; the code is two real modules, one Python and one TypeScript, of about 11,000 characters each; the JSON is a GitHub API response for a repository, pretty-printed and minified; and the CSV is 250 rows of synthetic order data with dates, IDs, quantities and prices. Each is one sample, so treat the figures as indicative of a content type, not as its average.
Two patterns hold across the table. Within any one tokeniser, structured text costs more per character than prose: on o200k, pretty-printed JSON is 47% more tokens per character than prose, and numbers are 2.9 times as many. And no tokeniser is cheapest everywhere: among current tokenisers o200k is lowest on every content type except Python, where Qwen3 edges it, while the Gemini tokeniser, which is highest on every content type, is lowest on 10 of the 20 languages below.
How much token counts differ by language
The same 1,012 sentences cost between 0.94 and 7.46 times as many tokens as their English originals on current tokenisers, depending on the language and the tokeniser, and up to 7.64 times on the legacy cl100k. FLORES-200 is a set of the same sentences professionally translated into 200 languages, so each row below is the same content, and the ratio isolates what the language does to the count.
| Language | cl100k (legacy) | o200k | Gemini | Qwen3 | DeepSeek V3.1 | Mistral Nemo |
|---|---|---|---|---|---|---|
| English | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| Spanish | 1.54 | 1.32 | 1.29 | 1.52 | 1.50 | 1.32 |
| French | 1.61 | 1.37 | 1.40 | 1.58 | 1.55 | 1.32 |
| German | 1.59 | 1.31 | 1.32 | 1.56 | 1.52 | 1.31 |
| Portuguese | 1.48 | 1.23 | 1.25 | 1.45 | 1.42 | 1.28 |
| Indonesian | 1.55 | 1.25 | 1.15 | 1.54 | 1.35 | 1.32 |
| Turkish | 1.92 | 1.43 | 1.36 | 1.62 | 2.04 | 1.49 |
| Vietnamese | 2.44 | 1.49 | 1.36 | 1.44 | 2.12 | 1.43 |
| Swahili | 1.95 | 1.49 | 1.62 | 1.94 | 1.94 | 1.82 |
| Yoruba | 2.95 | 2.17 | 2.36 | 2.69 | 2.83 | 2.81 |
| Russian | 2.46 | 1.42 | 1.35 | 1.75 | 1.58 | 1.50 |
| Arabic | 3.03 | 1.38 | 1.45 | 1.63 | 1.62 | 1.24 |
| Hindi | 4.77 | 1.57 | 1.30 | 4.43 | 2.80 | 1.80 |
| Bengali | 5.83 | 1.70 | 1.20 | 5.06 | 2.02 | 2.05 |
| Tamil | 7.64 | 1.98 | 1.45 | 6.11 | 3.07 | 2.18 |
| Thai | 4.38 | 1.99 | 1.59 | 2.58 | 1.81 | 2.41 |
| Amharic | 7.63 | 5.81 | 1.90 | 4.08 | 6.01 | 7.46 |
| Chinese (Simplified) | 1.87 | 1.25 | 1.08 | 1.01 | 0.94 | 1.40 |
| Japanese | 2.27 | 1.66 | 1.21 | 1.40 | 1.42 | 1.57 |
| Korean | 2.37 | 1.47 | 1.36 | 1.63 | 1.64 | 1.28 |
Read down a column to see a tokeniser's language bias, and across a row to see how much choosing a model matters for that language. For Western European languages the choice matters little: every tokeniser charges 1.2 to 1.6 times English. For Indic scripts it matters most: Hindi is 1.30 times English on Gemini's tokeniser and 4.43 times on Qwen3's. Chinese is the only language that any tokeniser encodes in fewer tokens than its English translation, and only just: 0.94 times English on DeepSeek, with Qwen3 level at 1.01. Both were trained with large amounts of Chinese and have whole words like 人权 ("human rights") as single tokens, and Chinese writes the same content in about a third as many characters.
The per-character view hides this. Because Chinese has so few characters for the same content (44,263 against 132,978), a characters-per-token rule calibrated on English would badly underestimate a Chinese prompt even on a tokeniser that handles Chinese well.
Why Claude Opus 4.7 and later count more tokens for the same text
Anthropic changed tokeniser with Claude Opus 4.7, launched on 16 April 2026, and says the newer one produces more tokens for the same text: "roughly 1.0–1.35× depending on the content type", in the Opus 4.7 announcement, which describes it as "an updated tokenizer that improves how the model processes text". The Opus 5.5 migration guide repeats the range. Opus 4.7 kept Opus 4.6's price of $5 per million input tokens and $25 per million output tokens, so for the same prompt the bill could rise by up to 35% on Anthropic's figures, and by more on some content in the independent counts below, with no change on the price list. Later models changed price as well as tokeniser; the price section works through them.
Anthropic's pricing page says "Claude 4.7 and later models and Claude Mythos Preview use a newer tokenizer", and that Sonnet 4.6 and earlier use the previous one. By version number that covers Opus 4.7, 4.8, 5 and 5.5, Sonnet 5 and 5.5 and Haiku 5.5, and the token counting page adds that Fable 5.1, Mythos 5.1, Fable 5 and Mythos 5 "share the tokenizer introduced with Claude Opus 4.7". Only for those four does Anthropic say "A prompt counts the same on all four"; for any other pair of current models, count with the model you plan to use.
Anthropic's pages give a typical figure and a range that fit together. The pricing and token counting pages say the new tokeniser produces "approximately 30 percent more tokens", and the announcement and migration guide give the range up to 35%. The models overview says "1M tokens is roughly 555k words" on the current tokeniser and that earlier models "fit about 750k words in 1M tokens", which works out to 1.35 times as many tokens per word.
Independent measurements put some content above Anthropic's range:
- Claude Code Camp counted 19 samples with Anthropic's counting endpoint on 16 April 2026: 1.20 times as many tokens for English prose, 1.29 for Python, 1.35 for Spanish, 1.47 for technical documentation, 1.07 for CSV and 1.01 for Japanese and Chinese.
- Playcode counted 16 real files with Anthropic's counting endpoint and OpenAI's o200k_base, in a post last updated on 2 September 2026. Against o200k, Claude's current tokeniser needed 1.40 times as many tokens for English prose and 1.50 to 1.73 times for code, against 1.05 and 1.22 to 1.32 times for Claude's previous tokeniser. Moving from the old Claude tokeniser to the new one added 34% on English prose and about nothing on Chinese.
- Simon Willison measured Opus 4.7's system prompt at 1.46 times its Opus 4.6 count (7,335 against 5,039 tokens) and a 30-page PDF at 1.08 times.
- OpenRouter's analysis of more than a million requests from users switching between the two models found effective cost up 12% to 27% for prompts over 2,000 tokens and slightly down for shorter ones. That compares traffic, not identical text, so it mixes the tokeniser with changes in how people used the model.
Two things in those numbers matter for prediction. The increase is smallest on CSV numbers and on Chinese and Japanese, which on our reading suggests the new tokeniser changed mainly how English, Spanish and code are cut. And the published ratios between Claude's two tokenisers range from 1.01 to 1.47 by content, so a single "30%" will be wrong for most workloads in one direction or the other. We did not measure Claude ourselves for this piece; the counting script below runs both tokenisers on your own text when given an Anthropic API key. On our ladder for grading AI claims, Anthropic's 1.0 to 1.35 range is documented, and each count above is measured by the party named.
What a different token count does to the price
The price that matters is the price per unit of your text, which is the list price per token multiplied by the tokens your text needs. Two worked examples show how far that can move from the list price. Prices are from the vendors' pricing pages on 13 October 2026, for input tokens.
The first is one model across content types. Gemini 3.1 Pro Preview (gemini-3.1-pro-preview) costs $2 per million input tokens for prompts up to 200,000 tokens, per Google's pricing page, and Google's SDK counts its tokens with the Gemma 4 tokeniser file we measured. A million characters of English prose is about 212,000 tokens there, or $0.42; a million characters of pretty-printed JSON is 378,000 tokens, or $0.76; and a million characters of CSV numbers is 977,000 tokens, or $1.95. One list price, a 4.6-fold range in price per character.
The second is one vendor across tokenisers. Taking Anthropic's 1.0 to 1.35 range at face value, here is what current Claude models cost per million tokens of text as the previous tokeniser would have counted it, against the last model of the same tier on the previous tokeniser:
| Last model on the previous tokeniser, price per million input tokens | Model on the current tokeniser and list price | Its price per million previous-tokeniser tokens |
|---|---|---|
| Claude Opus 4.6, $5 | Claude Opus 4.7, $5 | $5.00 to $6.75 |
| Claude Opus 4.6, $5 | Claude Opus 5.5, $4 | $4.00 to $5.40 |
| Claude Sonnet 4.6, $3 | Claude Sonnet 5.5, $2 | $2.00 to $2.70 |
| Claude Haiku 4.5, $1 | Claude Haiku 5.5, $0.10 for prompts up to 100,000 tokens | $0.10 to $0.135 |
Opus 5.5's list price is 20% lower than Opus 4.6's, but at the top of Anthropic's range the same text still costs 8% more, and at the 1.47 that independent counts found for some content it costs $5.88, 18% more. Sonnet 5.5 stays cheaper than Sonnet 4.6 across the whole range, at $2.94 even at 1.47. Output tokens are affected the same way, because the model writes its answer in the same tokens, though output length also depends on how much the model chooses to say.
The same arithmetic applies across vendors. Take two hypothetical models priced at $2 per million input tokens, one using o200k_base and one using Gemini's tokeniser. On English prose the Gemini-tokenised model costs 5% more per character; on a CSV of numbers, 67% more; on Hindi, 14% less. List prices compare models only for text that both tokenise alike.
Why characters-per-token rules of thumb mislead
A characters-per-token rule is a single average applied to text that varies by a factor of five. Google's token documentation says that "For Gemini models, a token is equivalent to about 4 characters" and that "100 tokens is equal to about 60-80 English words". On our English prose Gemini's tokeniser averages 4.73 characters per token, so the rule overestimates the cost of prose by about 18%. On pretty-printed JSON, at 2.64 characters per token, it underestimates by about 34%, and on numbers, at 1.02 characters per token, it is nearly four times too low.
Anthropic's rules have changed with its tokeniser. Its glossary gives about 3.5 English characters per token for older models, and the models overview gives 555,000 words per million tokens for the current one, about 1.8 tokens per word. For comparison, our English prose came to 1.23 tokens per word on o200k_base and 1.28 on Gemini's tokeniser. Rules quoted per word drift further, because word length differs by language and by domain.
Rules of thumb are fine for sizing a context window for English prose and code, where they land within about a third. They are not fine for numbers or non-Latin scripts, where Google's rule is nearly four times too low on our CSV. For a cost forecast, count.
Which tokeniser each vendor uses
As of 13 October 2026, OpenAI publishes the tokenisers for its models up to the GPT-5 family but not for GPT-6 Sol, Google publishes files its SDK uses for some Gemini models, and Anthropic publishes none for current models.
- OpenAI. tiktoken 0.14.0 maps GPT-4o, GPT-4.1, the o-series and model names starting
gpt-5to o200k_base, and GPT-4 and GPT-3.5 to cl100k_base. It has no entry for GPT-6 Sol, and OpenAI does not document that model's tokeniser. OpenAI's token counting guide describes aPOST /v1/responses/input_tokensendpoint for exact counts. - Google. The Gemma papers say Gemma uses Gemini's tokeniser: Gemma 3 uses "the same tokenizer as Gemini 2.0". Google's
google-genaiPython SDK (version 2.30.0) has an experimental local counter that uses the Gemma 3 file for Gemini 2.0 to Gemini 3 preview models and the Gemma 4 file for Gemini 3.1 Pro Preview, Gemini 3.1 Flash-Lite and Gemini 3.5 Flash. On all 26 of our texts the two files gave identical counts. The SDK maps no file to Gemini 3.8 Flash, so for that model use thecountTokensendpoint. - Anthropic. The old
@anthropic-ai/tokenizerpackage says it is not accurate for Claude 3 and later models, and its repository is archived. To count before sending, use Anthropic's free/v1/messages/count_tokensendpoint, which takes a model name, so it counts with that model's tokeniser; Anthropic calls the result an "estimate" that can "differ by a small amount" from a real request. After sending, theusagefield of each response gives the tokens the request was billed for. - Open-weight models (Qwen, DeepSeek, Mistral, Gemma, Llama) ship their tokeniser with the weights, usually as a
tokenizer.jsonfile. A host serving them bills by that tokeniser's count.
Vendor counting endpoints count a whole request, which includes a few framing tokens for the message structure, plus tool definitions, system prompts and images. Our measurements count raw text with no special tokens, so they compare tokenisers, not request totals.
How to count tokens for a specific model
Counting a sample of your real prompts with each candidate model's own counter is the only reliable forecast. This script prints the count for a text file with OpenAI's two tokenisers, and with any Claude or Gemini models you name, using the vendors' counting endpoints and your API keys.
import json, os, sys, urllib.request
import tiktoken # pip install tiktoken
def post(url, headers, body):
req = urllib.request.Request(url, json.dumps(body).encode(), {"content-type": "application/json", **headers})
with urllib.request.urlopen(req) as r:
return json.load(r)
def claude(model, text):
h = {"x-api-key": os.environ["ANTHROPIC_API_KEY"], "anthropic-version": "2023-06-01"}
body = lambda t: {"model": model, "messages": [{"role": "user", "content": t}]}
url = "https://api.anthropic.com/v1/messages/count_tokens"
# Subtract the request's own framing: a one-token message tells us how much it is.
return post(url, h, body(text))["input_tokens"] - post(url, h, body("a"))["input_tokens"] + 1
def gemini(model, text):
h = {"x-goog-api-key": os.environ["GEMINI_API_KEY"]}
url = f"https://generativelanguage.googleapis.com/v1beta/models/{model}:countTokens"
return post(url, h, {"contents": [{"parts": [{"text": text}]}]})["totalTokens"]
# Usage: python3 count_tokens.py prompt.txt [model ...]
# Model names starting "claude-" are counted by Anthropic, "gemini-" by Google.
text = open(sys.argv[1], encoding="utf-8").read()
counts = {enc: len(tiktoken.get_encoding(enc).encode(text, disallowed_special=())) for enc in ("cl100k_base", "o200k_base")}
for m in sys.argv[2:]:
counts[m] = claude(m, text) if m.startswith("claude-") else gemini(m, text)
for name, n in counts.items():
print(f"{name:20} {n:>8,} tokens {len(text) / n:5.2f} chars/token")
Run it as python3 count_tokens.py prompt.txt claude-opus-4-6 claude-opus-5-5 gemini-3.8-flash, naming the model you use now and the ones you are considering. We ran the tiktoken part on 13 October 2026; the API calls were checked only as far as the endpoints' authentication errors, because we had no keys for this test, so the Claude and Gemini lines are untested output. Counts from Anthropic's endpoint are free and rate-limited; check the other vendors' pages for theirs.
How this test was run
We counted text, not requests, on 13 October 2026, with tiktoken 0.14.0 for cl100k_base and o200k_base, the SentencePiece file that Google's SDK pins for Gemma 3 (gemma3_cleaned_262144_v2.spiece.model, SHA-256 1299c11d…), and the tokenizer.json files of google/gemma-4-E4B-it, Qwen/Qwen3-8B, deepseek-ai/DeepSeek-V3.1 and mistralai/Mistral-Nemo-Instruct-2407 at pinned Hugging Face revisions, with special tokens off. Each count is one deterministic run; a tokeniser gives the same count every time.
The language samples are the 1,012-sentence devtest split of FLORES-200 (NLLB Team et al., 2022, CC BY-SA 4.0), concatenated one sentence per line. The content samples are the seven described above. Things we did not test: Claude, GPT-6 Sol and Gemini 3.8 Flash, whose tokenisers are not published; images, tool definitions and chat framing; and more than one sample per content type.
How to predict the bill before switching models
Price a switch in cost per unit of your own work, not per token. In practice:
- Take a few hundred real requests from the workload you plan to move, whole: system prompt, tool definitions, conversation history, and the outputs you got. A vendor bills the whole request, not just the user's message.
- Count the inputs and the outputs with the current model's counter and the candidate's. The candidate's tokeniser applies to its answers too.
- Price input and output separately: each model's input price times its input count, plus its output price times its output count. The ratio of the two totals is the price change for the same text.
- Run a sample on the candidate and read the
usagefield of each response. How much a model writes, and how many reasoning or thinking tokens it spends where it uses them, depends on the model rather than its tokeniser, and can move the bill more than the tokeniser does.
How much the counting step can change the answer depends on the switch:
- Same tokeniser on both sides, such as GPT-4o to GPT-5 (both o200k_base): input counts match, so the list-price ratio is the input price change. Between current Claude models, Anthropic promises identical counts only for Fable and Mythos 5 and 5.1, so count the others.
- Between the tokenisers we measured: about 5% on English prose, up to 28% on code and JSON, 67% on numbers, and up to four times on non-Latin scripts.
- To or from Claude Opus 4.7 or later, from an older Claude model or another vendor: always count. The published ratios between Claude's two tokenisers run from 1.01 to 1.47 by content, and Playcode put Claude's current tokeniser at 1.40 to 1.73 times OpenAI's. Either can be enough to cancel the price difference you are switching for. In our test of six cheap models on JSON extraction, Claude Haiku 5.5 and GPT-6 Luna had the same list price on 20 October 2026, but Haiku 5.5 counted nearly twice as many prompt tokens for the same messages and cost 66% more per document.