Transformer

The Transformer is the architecture that nearly every modern language model is built on. It was introduced in the 2017 paper "Attention Is All You Need" by eight Google researchers, originally for machine translation, and within five years it had displaced the recurrent networks (RNNs) that came before it across language, code, vision, and audio. GPT, Claude, Gemini, Llama, DeepSeek: different labs, different training recipes, same skeleton. If you want to understand why today's models behave and cost what they do, this is the right place to start.

Why it replaced recurrent networks#

Older sequence models read text one token at a time, left to right, carrying a compressed running summary forward. That design had two structural problems. First, the computation was inherently serial: the network could not start on token 50 until it had finished token 49. A GPU with thousands of parallel cores sat mostly idle, and training on web-scale corpora was simply impractical. Second, information decayed with distance. Whatever the model knew about the first sentence of a document had to survive being squeezed through every intermediate step, so by token 500 it was usually gone. Tricks like LSTM gates helped, but long-range dependencies stayed unreliable.

The Transformer fixes both at once by deleting recurrence entirely. Every token in the sequence is processed simultaneously in one parallel pass, which is exactly the shape of work GPUs are built for; training throughput stopped being limited by sequence length and started being limited by how many chips you could buy. And because attention connects any token directly to any other token, distance stops mattering: a pronoun in paragraph nine reaches the noun in paragraph one in a single hop rather than through 2,000 lossy steps. The 2017 paper's headline result already showed the trade: better translation quality than the best recurrent systems at a fraction of the training cost.

Attention as the core block#

Attention is the only place in the network where tokens exchange information. For each token, the model asks: which other tokens matter for what I am trying to represent right now, and how much? It scores every token against every other, then rebuilds each token's representation as a weighted blend of the ones it cared about. Each layer runs dozens of these lookups in parallel (attention heads), and each head can specialize: syntax, coreference, copying literal strings.

The other half of every layer is a feed-forward network that processes each token independently, with no cross-token communication. A useful mental model: attention moves information between positions, the feed-forward layer thinks about what each position now holds. The feed-forward side is also where most of the parameters live, roughly two thirds in a standard layer, which is why it is the part modern models make sparse with mixture-of-experts.

Inside a decoder block#

A production model is just this block stacked many times. Llama 3 70B stacks it 80 times; small 7B-class models use around 30. Tokens from the tokenizer enter as vectors, flow through attention and feed-forward layers repeatedly, and the final vector for the last position is scored against the whole vocabulary to produce next-token logits.

That output is a score for every entry in the vocabulary, typically 100,000-plus options, and sampling from those scores is the whole game of next-token prediction. Generate a 500-token answer and this loop runs 500 times, which is why serving systems cache attention keys and values rather than recompute the stack from scratch each step.

Decoder-only vs encoder-decoder#

The original Transformer was an encoder-decoder built for translation: an encoder reads the full source sentence with every token seeing every other token, then a decoder writes the output while peeking at the encoder's result. That split spawned two simplifications. Encoder-only models (BERT, 2018, and the embedding models behind most vector search today) keep just the reading half and excel at understanding and retrieval. Decoder-only models (the GPT line, and essentially every chat model since) keep just the writing half: each token attends only to tokens before it, and the single training objective is predicting the next token.

Decoder-only won the LLM era for pragmatic reasons. One stack instead of two means simpler scaling. One objective means every token of training text provides a learning signal, no labeled pairs required. And generation is the native operation rather than a bolted-on mode. Encoder-decoder survives where the input and output are genuinely different things, such as Whisper mapping audio to text.

Why it scales so well#

The Transformer's most consequential property was discovered after the architecture itself: it improves predictably with scale. Kaplan and colleagues showed in 2020 that loss falls as a smooth power law across more than seven orders of magnitude of compute, with model size, data, and compute as the only levers. That predictability is what justified the bets that produced modern frontier models; you could forecast what ten times the compute would buy before spending it.

The architecture also keeps absorbing upgrades without changing shape: mixture-of-experts makes the feed-forward layers sparse so a model can have hundreds of billions of parameters but activate a fraction per token, grouped-query attention shrinks the KV cache, and better positional encodings stretched contexts from 512 tokens in 2017 to a million-plus in 2026. GPT-1 in 2018 had 117 million parameters; today's frontier models are four orders of magnitude larger, and the block diagram above still describes them.

Practical takeaways#

You rarely choose whether to use a Transformer, but its anatomy explains your operating costs. Parallel training is why capability tracks compute budgets and why scaling curves keep being bet on. Attention's all-pairs design is why long prompts cost disproportionately more and why the KV cache dominates serving memory. The decoder loop is why output tokens are priced several times higher than input tokens and why latency scales with answer length. When you reason about model pricing, context limits, or throughput, you are reasoning about this one architecture's bottlenecks.