Prompt Engineering & System Prompts

A system prompt is the instruction a developer places ahead of every conversation: who the model is, what rules it follows, and what shape its answers take. Prompt engineering is the wider job of writing that text, and the rest of what goes into the request, so the model does what you need without any change to its weights. The model is fixed when you call it, so every bit of steering happens through the tokens in front of it. That makes the prompt the first thing to fix, and it runs out at a predictable point: when the failure is missing knowledge, or a rule that has to hold every time.

System prompt, user and assistant roles#

Chat models read a conversation split into roles, and the roles carry different weight. The system prompt holds the standing instructions. OpenAI's current API calls it the developer message and gives it priority over user messages, following the chain of command in its model spec. The user message carries this turn's question and data. The assistant role holds the model's own replies, which become context for the next turn.

Think of the system prompt as the job description and the user message as today's task. Identity, rules, output contracts and tool guidance go in the system prompt. The question, the document and this request's variables go in the user message. Mixing them up fails in two ways: standing rules buried in a user turn fade as the history grows, and per-request data put into the system prompt changes the prefix on every call and breaks prompt caching.

One older technique no longer works everywhere: prefilling the start of the assistant's reply, such as an opening brace to force JSON, returns an error on Claude 4.6 models and later as of October 2026. Ask for the format in the prompt, or use a structured output mode, which constrains the model to a schema instead of hoping it follows one.

Structure beats clever wording#

Models follow a clearly delimited prompt far more reliably than a clever one. A dependable order is role, task, context, constraints, output format, examples. Separate the sections with XML-style tags or Markdown headers so the model never confuses your instructions with the data it is reading. Anthropic's and OpenAI's guides both recommend the same handful of habits: be direct, mark the sections, show examples.

Be specific about the standard. "Summarise in three bullet points, each under 15 words" beats "give me a nice short summary", which leaves the model to guess what you mean by short. Two or three chosen input and output pairs are usually worth more than another paragraph of rules, because the model matches the pattern it is shown. That is few-shot prompting, a form of in-context learning.

On long inputs, put the document near the top of the prompt and the question at the end. Anthropic's prompting guide reports that queries at the end improved response quality by up to 30 percent in its tests, most on complex inputs with several documents.

A ticket-triage prompt, before and after#

Weak: Summarize this support ticket.

Strong: You are a support triage assistant. Read the ticket below (delimited by <ticket>). Output JSON with keys: category (one of billing, bug, feature), urgency (low, medium, high), and one_line_summary (under 12 words). <ticket>...</ticket>

The weak version returns a differently shaped answer every time someone runs it. The strong one assigns a role, puts the untrusted ticket behind delimiters, names the output schema and caps the length, so the results parse and stay consistent across thousands of tickets. Nothing in it is clever. It reads like the spec for an API response, which is the standard to hold a production prompt to.

Prompt engineering inside context engineering#

In September 2025 Anthropic described context engineering as the set of strategies for curating and maintaining the right set of tokens during inference, including everything that lands in the window outside the prompt, and called it the natural progression of prompt engineering. A real agent's window shows why. The handwritten system prompt may be a couple of thousand tokens, while conversation history, retrieved documents and tool output add tens of thousands more. The context window is finite, so choosing what gets in, and what gets summarised, cut or dropped, matters as much as how the instructions are phrased. Budget the window the way you budget latency: decide what each turn must carry, and summarise or drop the rest.

What a prompt cannot hold#

A prompt is a request to the model, and the model can decline it, misread it or be talked out of it by text it reads later. Anything that must hold every time belongs outside the prompt. On our insurance voice agent, compliance lives in the tool contract. "Will the model always make the disclosure?" has no answer, while "can this operation run without the disclosure?" does, and the answer is no. On our trademark drafting system, the five-section structure is enforced by schema-constrained decoding and validators rather than by asking nicely. For the same reason, a prompt cannot defend against prompt injection by itself.

Climb past prompting in order of cost, and let the failure pick the step:

1. Exhaust a structured zero-shot prompt: role, constraints, explicit format.
2. Add a few examples when the model understands the task but misses your conventions.
3. Add retrieval when the failures are missing, stale or private facts, because rewording cannot supply knowledge the model never had.
4. Consider fine-tuning when the facts are right and the format, tone or behaviour still fails after a good prompt.

You can only tell those failures apart with evals. Without them a knowledge problem looks like a wording problem, and a team keeps polishing words after the words stopped being the cause. On our DevOps agent, every validation failure is recorded, and that record goes into improving the validation loop rather than the prompt.