Why the Same Prompt Has Different Token Counts on GPT-4o, Claude, and Gemini

Paste the same 2,000-word document into GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro, and each provider will bill you for a different number of input tokens. Not because anyone is rounding differently — because a "token" is not a standard unit. Every model family ships its own tokenizer, and the tokenizer decides how your text gets split before the model ever sees it.

If you estimate costs with one provider's numbers and deploy on another, your budget math is wrong from day one. Here is why the counts diverge, how big the differences get, and how to measure your own prompts instead of guessing.

A token is a tokenizer's opinion

Modern LLMs use subword tokenization — usually a variant of byte-pair encoding (BPE). The tokenizer is trained separately from the model: it learns a fixed vocabulary of character sequences from a training corpus, then greedily splits any input into the longest matching pieces.

Two consequences follow:

1. Vocabulary size matters. A tokenizer with 200k entries has longer, more specific pieces than one with 32k entries, so it covers the same text in fewer tokens.
2. Training corpus matters. A tokenizer trained on lots of code splits useEffect(() => { efficiently; one trained mostly on English prose shreds it into fragments. The same asymmetry applies to non-English languages, which can cost 2–3× the tokens of equivalent English text on tokenizers with English-heavy vocabularies.

Who uses what

Model	Tokenizer	Notes
GPT-4o, GPT-4o mini	o200k_base (tiktoken)	~200k vocabulary; noticeably more efficient than its predecessor on code and non-English text
GPT-4, GPT-3.5 Turbo	cl100k_base (tiktoken)	~100k vocabulary; the counts most older blog posts quote
Claude 3.5 Sonnet, Claude 3.5 Haiku	Anthropic proprietary	Not published as an open library; counts come from the API's usage fields or Anthropic's count-tokens endpoint
Gemini 1.5 Pro, Gemini 1.5 Flash	SentencePiece-based	Google exposes counts via a countTokens API call
Llama 3.1	BPE, ~128k vocabulary	Open weights, open tokenizer — countable locally
DeepSeek V3	BPE, ~128k vocabulary	Open tokenizer, similar ballpark to Llama

The practical upshot: OpenAI's counts are reproducible offline with tiktoken, Llama's and DeepSeek's with their open tokenizers — but Claude and Gemini counts require asking the provider (or using a tool that bundles the right encodings).

How big are the differences?

For ordinary English prose, the divergence between modern tokenizers is real but moderate — typically in the range of a few percent to around 10–20% for the same text. Three kinds of content widen the gap:

• Code. Indentation, brackets, and identifiers tokenize very differently across vocabularies. A tokenizer trained with more code in its corpus produces meaningfully fewer tokens for the same source file.
• Non-English text. Vocabularies are dominated by English. Chinese, Hindi, or Arabic text can double or triple the token count relative to an English translation of the same content — and the penalty varies by tokenizer.
• Structured output. JSON keys, quotes, and braces add up. The same schema serialized identically costs different amounts on different models.

A 15% tokenizer difference compounds directly into your bill: at production volume, "Model B is cheaper per million tokens" can be reversed in practice if Model B's tokenizer needs 20% more tokens for your specific payload.

Why this breaks naive cost comparisons

Provider pricing pages quote dollars per million tokens — in their own tokens. Comparing GPT-4o at one price against Claude 3.5 Sonnet at another implicitly assumes a token is a token. It isn't. The honest comparison is:

cost = (your_text → model's tokenizer → token count) × model's rate

for each candidate model, using your actual prompts — the system prompt you'll ship, your few-shot examples, your typical user inputs. Averages from someone else's blog post do not transfer, because their content mix (prose vs. code vs. JSON vs. language) is not yours.

The same reasoning applies to context-window planning. "128k context" is 128k of that model's tokens. A conversation history that fits comfortably in one model's window can overflow another's, even at the same advertised size.

Measuring instead of guessing

You can assemble this yourself — tiktoken for OpenAI models, the Hugging Face tokenizers for Llama and DeepSeek, API calls for Claude and Gemini — or paste your prompt once into the Token Inspector, which runs the encodings in your browser and shows token counts and per-request costs across 20+ models side by side, with no sign-up. Either way, the rules of thumb are:

• Benchmark with your real system prompt and representative user inputs, not lorem ipsum.
• Re-check when you switch model families — a migration from GPT-4 to GPT-4o alone changes counts, because cl100k_base and o200k_base differ.
• If your workload is code-heavy or multilingual, expect the spread between providers to be wider than any published average.
• Watch output tokens too: they are usually priced 3–5× input tokens, and verbose models inflate them regardless of tokenizer.

Tokens are the billing unit of the entire LLM economy, and no two providers agree on what one is. Measure before you commit.