Tokens & Tokenization

Tokens in LLMs are the pieces of text a model actually reads and writes. A token is usually a common word, a fragment of a rarer word, a punctuation mark or a run of spaces. Tokenization is the step that cuts your text into those pieces before the model sees it, and the model then predicts one token at a time during next-token prediction. Providers bill by the token, context windows are measured in tokens, and every "prompt too long" error is a token count.

How text becomes tokens#

Most current tokenizers descend from byte-pair encoding (BPE), which Sennrich and colleagues adapted for machine translation in 2016. Training one is simple to describe. Start with raw bytes as the alphabet, find the pair of neighbours that appears together most often in a large corpus, merge it into a new entry, and repeat until the vocabulary reaches a target size, typically between 50,000 and 200,000 entries.

Frequency then decides how text is cut. "The" is common enough to be one token. "Tokenization" splits into a few pieces, and a typo such as "tokenizaiton" breaks into many small ones. Because the base alphabet is bytes, any string in any script can be encoded, sometimes at a high token count.

Two details surprise people. Whitespace belongs to the token, so " world" with a leading space and "world" without one are different entries. And punctuation usually splits off, so "Hello, world!" is about four tokens, not two words.

Type a sentence into the widget below and watch where the cuts fall.

Four characters per token, and where that breaks#

For ordinary English prose, a token averages about four characters, or roughly three quarters of a word. A 1,000-word article is about 1,300 to 1,400 tokens. A 50-page contract at 300 words a page is around 20,000.

The rule fails as soon as the text stops being English prose. Code carries punctuation and indentation. Digit strings split into chunks of one to three digits. Other scripts can fall to two characters per token or fewer. Petrov and colleagues found the same sentence translated into different languages can differ in token count by up to 15 times, so a user writing in one of those languages pays more per request, waits longer, and fits less history into the context window.

Measure before you blame the tokenizer for a slow language, though. On the bilingual voice agents we built, per-stage timing on AVOX Realty showed that the extra time on Hindi and Hinglish calls was spent in transcription rather than in the language model. A token count per language tells you the cost. It does not tell you where the latency went.

Token counts do not carry over#

Every model family ships its own tokenizer, and vocabularies have grown: GPT-2 used about 50,000 entries, GPT-4's cl100k_base about 100,000, Llama 3 128,000, and GPT-4o's o200k_base about 200,000. A larger vocabulary stores more whole words and phrases as single entries, so the same document takes fewer tokens.

A count can change inside one vendor's models too. Anthropic's launch post for Claude Opus 4.7, in April 2026, says its updated tokenizer can map the same input to roughly 1.0 to 1.35 times as many tokens, depending on the content. The price per token did not change, so the same prompts could cost up to 35% more. Chunk sizes, maximum-token caps, truncation rules and cost models all assume one tokenizer.

Bigcircle recommends recounting with the target model's own tokenizer or its token-counting endpoint as one of the checks before any model change, because a number measured on the old model is wrong on the new one without any error to tell you. Our guide on replacing the model under a production agent sets out the order of those checks.

Tokens set the bill and the wait#

Providers price per million tokens, with input and output metered separately and output usually several times more expensive. Work one example through. A support assistant with a 2,000-token system prompt retrieves about 3,000 tokens of documentation per question and writes a 300-token answer. Each request bills about 5,300 input tokens and 300 output tokens. At one million questions a month that is 5.3 billion input tokens, of which the unchanging system prompt is 2 billion, the repeated prefix that prompt caching exists to discount.

Output tokens are generated one at a time, so an answer twice as long in tokens takes roughly twice as long to stream. Capping answer length is therefore a direct way to cut response time.

Where tokenization causes bugs#

Numbers split at arbitrary digit boundaries, which is part of why models are unreliable at arithmetic and at comparing 9.11 with 9.9. Put any sum, comparison or threshold in code and hand the model the result, as our guide on why temperature 0 is not deterministic argues for every step with one right answer.

Character-level questions fight the representation. Asked how many letters "strawberry" has, the model sees a few token ids, not ten letters. If the product needs spelling, counting or character positions, do that in code as well.

Code costs more than it looks, because braces, indentation and rare identifiers each take tokens, although newer tokenizers include entries for runs of spaces that help with Python. Count a representative file before you size a code-heavy context.