Tokens & Tokenization

A language model never sees your text as letters or words. Before anything else happens, the text is cut into tokens: the atomic units the model reads, predicts one at a time during next-token prediction, and charges you for. Every budget estimate, every context limit, every "why is my prompt too long" error traces back to this one preprocessing step, so it pays to understand it properly.

What a token actually is#

A token is usually a subword: a frequent fragment of text, learned statistically from a training corpus. Most modern tokenizers descend from Byte-Pair Encoding (BPE), an algorithm Sennrich and colleagues adapted for machine translation in 2016. Training a BPE tokenizer is simple to state: start with raw bytes as the alphabet, count which adjacent pairs appear together most often, merge the winner into a new vocabulary entry, and repeat until the vocabulary hits a target size, typically somewhere between 50,000 and 200,000 entries.

The result is a vocabulary where frequency decides granularity. The word "the" is so common it becomes a single token. "Tokenization" is rarer, so it splits into pieces like "token" plus "ization". A typo like "tokenizaiton" shatters into many small fragments. Because the base alphabet is bytes, nothing is ever out of vocabulary: any string, in any script, including emoji, can be encoded, just sometimes expensively.

Two details surprise people. First, whitespace belongs to tokens: " world" with a leading space and "world" without one are different vocabulary entries, which is why a stray trailing space at the end of a prompt can nudge a model into a slightly worse continuation. Second, punctuation usually splits off on its own, so "Hello, world!" is not two units but roughly four: "Hello", ",", " world", "!".

Type a sentence into the widget below and watch where the cuts land.

The four characters rule, and where it breaks#

For ordinary English prose, one token averages about four characters, or roughly three quarters of a word. The reason is frequency: English dominates most training corpora, so its common words earned single-token slots during BPE training. Quick mental math follows: a 1,000 word blog post is about 1,300 to 1,400 tokens; a 50 page contract at 300 words per page is around 20,000 tokens.

The rule breaks the moment you leave clean English prose. Code carries heavy punctuation and indentation. Numbers fragment, since most tokenizers split digit strings into chunks of one to three digits. And non-Latin scripts can fall to two characters per token or worse. The four characters figure is a planning heuristic for English, not a law: when the numbers matter, count with the target model's actual tokenizer (OpenAI ships tiktoken for exactly this, and most open models publish theirs on Hugging Face).

Why tokens are the meter that bills you#

Providers price per million tokens, with input and output metered separately and output typically costing several times more. The context window is likewise a token budget, not a word or page budget: a model advertising 200,000 tokens of context holds roughly 150,000 English words, but far fewer words of dense code or Hindi.

Work one example end to end. A support chatbot has a 2,000 token system prompt, retrieves about 3,000 tokens of documentation per question, and produces 300 token answers. Every single request therefore bills around 5,300 input tokens plus 300 output tokens. At one million questions per month, that is 5.3 billion input tokens, and the static system prompt alone accounts for 2 billion of them, which is exactly the kind of repeated prefix that prompt caching exists to discount. Teams that think in words miss this arithmetic; teams that think in tokens design around it.

Latency follows the same meter: output tokens are generated one at a time, so a response that is twice as long in tokens takes roughly twice as long to stream.

Same text, different tokenizers#

Token counts are not portable between model families, because every family ships its own tokenizer with its own vocabulary. The trend has been steadily upward: GPT-2 used a vocabulary of about 50,000 tokens, GPT-4's cl100k_base roughly doubled that to 100,000, Llama 3 jumped to 128,000, and GPT-4o's o200k_base reaches 200,000. A larger vocabulary stores more whole words and common phrases as single entries, so the same document compresses into fewer tokens.

The practical consequence: a prompt that measures 8,000 tokens on one provider might measure 9,500 on another. Chunk sizes tuned for a RAG pipeline, max-token caps, truncation logic, and cost models all silently assume one specific tokenizer. When you swap models, recount; never copy token-denominated numbers across vendors.

Quirks that bite in practice#

A few tokenization artifacts cause real production bugs, not just trivia-night surprises.

Numbers fragment unpredictably. "3.14159" becomes several tokens split at arbitrary digit boundaries, which is part of why models are shaky at arithmetic and at comparing values like 9.11 versus 9.9: they are manipulating digit chunks, not numbers. For anything quantitative, have the model call a tool rather than compute in its head.

Counting and spelling tasks confuse models. Ask how many letters "strawberry" contains and the model is reasoning over two or three opaque token ids, not eleven characters. Character-level tasks are fighting the representation itself.

Code costs more than it looks. Indentation, braces, and rare identifiers inflate counts, though modern tokenizers added dedicated multi-space tokens that help a lot with Python.

Non-English text pays a tokenizer tax. Petrov and colleagues showed the same sentence translated across languages can differ in token count by up to 15 times. A Hindi or Burmese user of the same app pays more per request, waits longer, and fits less history into the context window, purely because the tokenizer's merge table was trained mostly on English. If you serve multilingual traffic, benchmark token counts per language before promising margins.

Practical takeaways#

Think in tokens, not words. Estimate with four characters per token for English, then verify with the target model's real tokenizer before committing to budgets, chunk sizes, or context math. Meter input and output separately, since they are priced differently and output also sets latency. Recount everything when you switch model families, because vocabularies differ by a factor of four across generations. And when prompts involve numbers, character manipulation, or non-English text, expect the tokenizer, not the model, to be the thing that bites you.