Prompt Engineering & System Prompts

Prompt engineering is the discipline of shaping the text you send a model so it reliably does what you want, without touching its weights. It is the cheapest and fastest lever you have: no training run, no retrieval pipeline, just a better-written request. Because the model is frozen at inference time, every bit of steering happens through the tokens you put in front of it, which makes prompt structure the highest-leverage thing you control, and the first thing to exhaust before reaching for anything more expensive.

Roles: system, user, assistant#

Chat models read a conversation split into roles, and the roles carry different weight. The system prompt (OpenAI now calls it the developer message) holds persistent, durable instructions: who the model is, what rules it follows, what tone and format it uses, how it handles edge cases. It applies to every turn, and providers explicitly train models to prioritize it above user messages. That instruction hierarchy is also your first, imperfect line of defense against prompt injection.

The user role carries the actual request and data for this turn. The assistant role is the model's reply; prior assistant turns become context the model builds on, and some APIs let you prefill the start of an assistant turn to lock in a format, such as the opening brace of a JSON object.

A good mental model: the system prompt is the job description, the user message is today's task. Identity, rules, output contracts, and tool guidance belong in the system prompt. The question, the document, and this request's variables belong in the user message. Mixing them up is the most common beginner mistake: stable behavior buried in a user turn gets forgotten as history grows, and per-request data jammed into the system prompt breaks caching.

Structure beats cleverness#

Models follow clearly delimited, explicitly structured prompts far more reliably than clever phrasing. A dependable anatomy, in order: role, task, context, constraints, output format, examples. Use delimiters (XML-style tags, markdown headers, triple quotes) to separate sections so the model never confuses your instructions with the data it is processing. Anthropic's and OpenAI's official guides converge on the same short list: be direct, structure with tags or headers, show examples.

Specificity wins. "Summarize in three bullet points, each under 15 words, no jargon" outperforms "give me a nice short summary" every time, because the vague version leaves the model guessing at your standards. When output must be machine-readable, name the exact schema or use a structured output mode rather than hoping. And two or three well-chosen input and output pairs are usually worth more than a paragraph of description: models are pattern matchers, and showing the pattern (few-shot prompting, a form of in-context learning) beats describing it.

One practical detail for long inputs: place the document near the top of the prompt and the question or instructions at the end. On long-context tasks this ordering measurably improves answer quality.

A tiny before and after#

Weak: Summarize this support ticket.

Strong: You are a support triage assistant. Read the ticket below (delimited by <ticket>). Output JSON with keys: category (one of billing, bug, feature), urgency (low, medium, high), and one_line_summary (under 12 words). <ticket>...</ticket>

The weak version will produce a different shape of answer every time someone runs it. The strong version assigns a role, isolates untrusted input behind delimiters, names the exact output schema, and constrains length, so it produces parseable, consistent results across thousands of tickets. Nothing about it is clever. That is the point: a prompt is an interface contract, and contracts are boring on purpose.

The reframing: prompt engineering inside context engineering#

For the first few years of the LLM era, prompt wording got most of the attention. Through 2025 and 2026 the framing shifted. Anthropic formalized context engineering in September 2025 as the set of strategies for curating and maintaining the optimal set of tokens during inference, and the industry broadly adopted prompt engineering as a subset of it.

The reason is what production systems actually look like. In a real agent, the hand-written system prompt might be 2,000 tokens while conversation history, retrieved documents, and tool outputs contribute tens of thousands more. The question stops being "what are the right words" and becomes "what configuration of context most likely produces the behavior we want." The context window is a finite resource with diminishing returns, so deciding what gets in, and what gets summarized, truncated, or dropped, matters as much as how the instructions are phrased. Good prompt engineering remains the foundation; context engineering is the budgeting discipline wrapped around it.

When to graduate beyond prompting#

A useful ladder, ordered by cost. First, exhaust zero-shot prompting: structure, constraints, explicit format. Second, add few-shot examples when the model understands the task but misses your conventions; two to five examples in the prompt remain the cheapest accuracy win available. Third, reach for RAG when failures come from missing, stale, or private facts, because no amount of rewording fixes knowledge the model never had. Fourth, consider fine-tuning when failures are about format, tone, or behavior that survives good prompting, or when you need to compress a long prompt into the weights for latency and cost.

The discipline that makes the ladder work is measurement. Without evals you cannot tell whether a failure is a wording problem, a knowledge problem, or a behavior problem, and you will spend money fixing the wrong one.

Practical takeaways#

Treat the prompt like an interface contract, not a conversation. Put stable behavior in the system prompt and per-request data in the user message. Delimit anything the model should treat as data rather than instructions. State the output format explicitly and show one example of it. Prefer specific constraints over clever wording, and prefer examples over adjectives. When a well-structured prompt with good examples still fails, stop polishing words: diagnose whether the gap is knowledge (retrieval), behavior (fine-tuning), or context assembly, and climb the ladder deliberately.