Few-Shot & In-Context Learning

In-context learning is the ability of a large model to pick up a new task from examples placed directly in the prompt, with no training at all. You show it a few input-output pairs, it infers the pattern, and it applies that pattern to your real input. The weights never change; the "learning" happens inside a single forward pass and evaporates when the call ends. The GPT-3 paper, "Language Models are Few-Shot Learners" (Brown et al., 2020), made this the headline result of the scaling era: a single frozen model could perform thousands of tasks competitively just by being shown what each task looked like.

The zero, one, few-shot spectrum#

The number of demonstrations defines a spectrum. Zero-shot is instruction only: "Classify this support ticket as billing, technical, or account." One-shot adds a single worked demonstration. Few-shot adds several, typically 3 to 10 in practice. The GPT-3 paper used this exact framing and showed accuracy climbing with each step: on many benchmarks the jump from zero-shot to a handful of examples was worth more than an order of magnitude of model size.

Two things have stayed true since. Larger models extract more from each example, so in-context learning got dramatically more useful as models scaled. And returns diminish: the first two or three examples usually buy most of the gain, with each addition after that worth less while costing the same.

Why examples beat instructions#

Instructions describe the task in the abstract; examples pin it down operationally. For anything involving a precise output format, an unusual label set, or judgment calls at the boundaries, two demonstrations communicate more than two paragraphs of prose, because prose always leaves edge cases ambiguous and examples cannot.

Worked example: ask a model zero-shot to extract {"name", "amount"} from invoice text and you will get JSON most of the time, but with drifting key names, currency symbols sometimes included in the number, and the occasional helpful sentence wrapped around the output. Add two examples, one ordinary and one tricky ("no amount present" mapping to "amount": null), and the output locks onto your exact schema, including the abstention behavior you never managed to specify in words. Pairing few-shot with structured output enforcement closes the last gap.

What the examples cost#

Examples are not free: every demonstration sits in the context window on every single call. Run the numbers on a realistic setup. Ten examples at 150 tokens each is 1,500 tokens of overhead per request. At a frontier input price of $5 per million tokens, that is $0.0075 per call, or $7,500 per million calls, just for the examples.

Prompt caching changes this math substantially. If the examples form a stable prefix at the top of the prompt, cache reads cost about 10% of the base input price on Claude models (with comparable discounts elsewhere), cutting that $7,500 to roughly $750 plus occasional cache-write premiums. Caching also removes most of the latency penalty, since cached prefixes skip recomputation. The practical rules: put examples before the variable user input so the prefix stays cacheable, never shuffle or rotate them per request (that breaks the cache), and remember that caching discounts the cost but does not eliminate it.

Choosing and ordering examples#

Quality and arrangement matter as much as quantity. Three guidelines hold up well in practice.

Make the set diverse and edge-case-bearing. Five near-identical happy-path examples teach less than three that span the space: one typical case, one boundary case, one demonstrating what to do when the input is malformed or the answer is "none." Your examples are the specification; whatever behavior they fail to demonstrate remains undefined.

Balance the labels. For classification, keep label counts roughly even. Zhao et al. (2021) showed GPT-3's few-shot accuracy swinging wildly with the label mix and ordering of the same examples, because models drift toward the majority label in the prompt.

Mind recency effects. Models weight late examples more heavily; they are disproportionately likely to repeat the final example's label or style. If your last three examples are all "approve," borderline inputs lean approve. Either interleave labels or, when inputs are scored for similarity, order examples from least to most relevant so the strongest signal lands last. And whenever you change the example set, re-run your evals: ordering changes alone have produced double-digit accuracy swings on classification tasks.

Many-shot ICL, and when to fine-tune instead#

Long context windows turned few-shot into many-shot. Agarwal et al. (2024, Google DeepMind) showed that scaling from a handful of demonstrations to hundreds or thousands keeps improving performance on many tasks, can override biases the model absorbed in pretraining, and in some cases matches supervised fine-tuning with no weight updates at all. With million-token context windows and prompt caching, "put 500 labeled examples in the prompt" is now a legitimate production architecture rather than a stunt, and it stays instantly editable in a way trained weights never are.

That still leaves a real graduation point to fine-tuning. Move when the per-call token cost of your example set exceeds the amortized cost of a training run at your traffic volume, when latency budgets cannot absorb a long prompt even cached, when you need thousands of examples and a smaller fine-tuned model would be cheaper to serve, or when the task is stable enough that baking it into weights pays off. Until then, prefer in-context learning: it deploys in minutes, rolls back instantly, and requires no ML infrastructure.

Practical takeaways#

Reach for examples before instructions when format or edge cases are the problem; two good demonstrations beat a paragraph of rules. Start with 3 to 5 diverse, edge-case-bearing examples, balance and interleave labels, and watch recency effects at the end of the list. Keep the example block as a stable cached prefix so it costs roughly a tenth of its sticker price. Scale to many-shot before assuming you need training. And treat fine-tuning as the graduation, not the starting point: it earns its complexity only when example sets get too large, too slow, or too expensive to keep in the prompt.