Transformer Architecture

The transformer is the neural network architecture that nearly every large language model is built on. In an LLM, the transformer architecture is a stack of identical blocks: each block lets every token gather information from the tokens before it, through attention, and then processes each token on its own. The design was introduced in the 2017 paper "Attention Is All You Need", by eight authors working mostly at Google, for machine translation. Within a few years it had replaced the recurrent networks that came before it in language, code, vision and audio.

You rarely choose whether to use a transformer. You do make decisions every week whose cost its shape sets: how long the prompt is, how long the answer is, how much history a conversation keeps, and which model fits on which hardware. This page is the anatomy behind those costs.

Why it replaced recurrent networks#

Older sequence models read text one token at a time, carrying a compressed running summary forward. The computation was serial, so a GPU with thousands of cores sat mostly idle, and information faded with distance: whatever the model knew about the first sentence had to survive every step after it.

The transformer removed the recurrence. During training every position in a sequence is processed in one parallel pass, which suits GPUs, and attention connects any token directly to any earlier one, so a pronoun in paragraph nine reaches its noun in paragraph one in a single step. The original paper reported better translation scores than the best recurrent systems at a small fraction of their training cost.

Inside one transformer block#

Each block has two parts. Attention is the only place where tokens exchange information: each token scores every earlier token for relevance and rebuilds its own representation as a weighted mix of them. A layer runs many of these lookups side by side, called attention heads. The second part, a feed-forward network, then processes each position on its own. In a standard block the feed-forward part holds about two thirds of the parameters, which is why it is the part mixture-of-experts models make sparse.

A production model stacks this block many times. Llama 3 70B has 80 of them. Tokens from the tokenizer enter as vectors, pass through every block, and the final vector at the last position is scored against the whole vocabulary to produce the scores for the next token.

Generating a 500-token answer runs that stack 500 times, one new token per pass. To avoid recomputing the earlier tokens each time, the server stores their attention keys and values in the KV cache, and that cache grows with every token a conversation holds.

What the shape costs you#

The two phases of serving cost different things. Reading the prompt, called prefill, happens in one parallel pass, and its time grows with prompt length, which mostly sets the wait before the first token. Writing the answer, called decode, is serial, one token per pass, which is why output tokens are priced several times higher than input tokens and why a longer answer takes proportionally longer to stream.

Memory follows the cache, not only the weights. Our guide to LLM inference works it through for Llama 3 70B in 16-bit precision: 327,680 bytes of cache per token, so one 32,768-token conversation holds about 10.7 GB. The weights stay the same size however many people use the product, and the cache grows with every user and turn. Bigcircle recommends checking that figure for your own model before you set a context limit, because it decides how many conversations fit on one GPU.

So change length before hardware. On AVOX, the insurance voice agent we built, the policy and claim details the agent needed were loaded when the call opened rather than fetched during a turn, which took the largest variable cost off every turn. Trim conversation history before raising a context limit, put stable content at the start of the prompt so prompt caching can reuse it, and cap answer length where response time matters.

Parameter counts mislead on mixture-of-experts models. The feed-forward part is split into many experts and each token uses only a few of them, so DeepSeek-V3 has 671 billion parameters in total and activates 37 billion per token. Memory has to hold the total, and the per-token compute follows the active count.

Decoder-only and encoder-decoder#

The original transformer was an encoder-decoder: an encoder read the whole source sentence with every token seeing every other, and a decoder wrote the translation while consulting it. Encoder-only models such as BERT (2018) kept the reading half, and models like it still produce most of the embeddings behind vector search. Decoder-only models, the GPT line and nearly every chat model since, kept the writing half: each token sees only the tokens before it, and the training objective is predicting the next token. Encoder-decoder survives where the input and output are different kinds of thing, as in Whisper, which maps audio to text.

Why it keeps scaling#

Kaplan and colleagues showed in 2020 that a transformer's loss, its error at predicting held-out text, falls smoothly and predictably as model size, data and compute grow, across more than seven orders of magnitude of compute. That predictability justified spending on ever larger runs: you could estimate what ten times the compute would buy before spending it. GPT-1 in 2018 had 117 million parameters. The block above still describes the open-weight models with hundreds of billions, with changes inside it rather than to its shape: sparse experts, attention variants that shrink the cache, and new ways of encoding token position that stretched context from BERT's 512 tokens to a million or more on several models as of October 2026. If you are choosing a model rather than training one, size each prompt to what the task needs, because the cache arithmetic above applies to every token you send, however large the window.