LLM applications

Server-Sent Events (SSE) let the model push each token to the client as soon as it is generated, rather than buffering the full response. The user sees output appearing in real time, which makes a ten-second generation feel far more responsive than the same content arriving all at once.

In practice, every production chat or co-pilot surface should stream. The implementation is straightforward with the official SDKs: open a stream, iterate over text events, and write them directly to the UI. The main gotcha is error handling: a 200 response can still carry an error event mid-stream, so you need to handle that case separately from the initial HTTP status.

Sources

• Streaming messagesAnthropic

Extended thinking gives the model a dedicated reasoning phase before it produces a final answer. On genuinely hard tasks, such as multi-step math, code architecture decisions, or complex comparisons, the accuracy lift is substantial compared to a direct answer.

The tradeoff is tokens and latency: thinking blocks are billed and add wall-clock time. The practical discipline is to use it selectively, on tasks where reasoning demonstrably helps, and to measure whether the quality gain justifies the cost. Modern Claude models support adaptive thinking, where the model decides when to think based on query complexity and your configured effort level.

Sources

• Extended thinkingAnthropic

The Message Batches API processes large volumes of requests asynchronously at half the per-token cost of synchronous calls. You submit a batch of up to ten thousand requests, poll for completion, and retrieve results when the batch is done, typically within an hour.

Batch mode is the right default for evaluations, content pipelines, data extraction at scale, and any workload where a user is not waiting for an immediate reply. The 50% cost reduction compounds quickly at volume, and the higher throughput means large jobs finish faster without hitting per-minute rate limits.

Sources

• Batch processingAnthropic

The Anthropic Messages API is the primary interface for frontier Claude models. It supports long context windows, native tool use, parallel function calls, streaming, extended thinking, and prompt caching. The Python and TypeScript SDKs provide typed wrappers with automatic retries and streaming helpers.

Key practitioner points: pin a specific model version to avoid silent behavior changes, enable streaming for any user-facing surface, and use the batch endpoint for high-volume async workloads to cut cost in half.

Sources

• Messages API overviewAnthropic
• Error handling and rate limitsAnthropic

The OpenAI API provides access to GPT-series frontier models with a broad ecosystem of compatible tooling, including libraries, proxies, and third-party services that implement the same request format. Many self-hosted inference servers (vLLM, Ollama) expose an OpenAI-compatible endpoint, making it a de facto interchange format.

When using it in production, track token spend per request from day one, handle 429 rate-limit responses with exponential backoff, and pin a specific model version rather than relying on a pointer like 'gpt-4' that may silently advance.

Sources

• Function calling guideOpenAI
• Structured outputs guideOpenAI

LLM APIs return 429 (rate limit) and 5xx (server error) responses that are transient and safe to retry. The correct pattern is jittered exponential backoff: wait a random fraction of an exponentially growing delay before each attempt, so a burst of simultaneous retries does not hit the provider in lockstep.

The critical addition is a hard budget: a maximum number of attempts or total elapsed time per request, beyond which you return an error rather than retrying forever. Without the budget, a sustained outage or a misconfigured loop can keep your application alive while draining spend or blocking downstream work. Most official SDKs include built-in retry logic; verify the defaults and add a cap if they do not.

Sources

• Error types and status codesAnthropic

Every API response includes the input and output token counts for that call. Logging those counts, attributed to the feature, user, and model that generated them, is the minimum instrumentation for cost visibility. Without it, spend is invisible until the billing cycle closes.

In practice, token metering feeds two things: a cost dashboard that lets you find the expensive paths in your product, and per-user or per-tenant quotas that prevent a single heavy user from driving up shared costs. Both are far easier to build from day one than to retrofit once usage grows.

Sources

• Error types and rate limitsAnthropic

Few-shot prompting places two to five worked examples directly in the prompt before the real input. The model infers the expected format, tone, and reasoning pattern from those examples rather than from abstract instructions. A well-chosen example set is usually more effective than a long written specification.

The key practitioner discipline is curation: examples should be diverse enough that the model generalizes the pattern rather than memorizing surface features, and they should be wrapped in XML tags so the model clearly separates them from instructions. When a new failure mode appears, adding a targeted example is often the fastest fix.

Sources

• Prompting best practicesAnthropic

Chain-of-thought prompting asks the model to write out its intermediate reasoning steps before giving a final answer. On tasks involving arithmetic, logic, or multi-step planning, showing the work consistently improves accuracy compared to asking for a direct answer.

The modern version of this technique is extended or adaptive thinking, where the reasoning happens in a dedicated thinking block rather than in-line text. For models without native thinking support, the instruction 'think step by step before answering' still provides a meaningful accuracy boost. The cost is additional tokens; the gain is reliability on the tasks where reasoning actually matters.

Sources

• Prompting best practicesAnthropic

A single prompt that asks the model to do too many things in one turn is brittle: each additional requirement increases the chance of the model dropping one of them. Prompt decomposition splits the task into a chain of smaller, focused prompts where each step has one clear job and its output feeds the next step.

Decomposed chains are also easier to debug and improve: when a step fails you replace just that step, not the whole prompt. The tradeoff is latency and cost from the extra round trips, but for high-stakes tasks the reliability gain is almost always worth it.

Sources

• Prompting best practicesAnthropic

When a prompt mixes instructions, context, user input, and examples, the model can confuse one section for another. Wrapping each section in XML tags such as `<instructions>`, `<context>`, `<example>`, and `<input>` gives the model unambiguous boundaries between them.

This is especially important for injection resistance: if user-supplied content is placed inside a tagged section, the model is less likely to interpret it as an instruction. It also makes prompts easier to maintain, because each section can be updated independently without accidentally affecting the others.

Sources

• Prompting best practicesAnthropic

A prompt is a software artifact: it has behavior, it can regress, and unreviewed changes can break production. Treating prompts like code means every change goes through review, is stored in version control with a meaningful commit message, and is linked to an eval run that proves the change is safe.

The practical mechanism is a prompt registry, either a dedicated tool like Langfuse or PromptLayer, or simply a directory of versioned prompt files in your repository. Without this discipline, a prompt fix made under pressure can silently break a related behavior, and the root cause is invisible.

Sources

• Prompting best practicesAnthropic

Model providers release new versions on a rolling basis, and pointer aliases like 'claude-latest' or 'gpt-4' can silently advance to a new model. Even beneficial improvements can shift tone, format, or behavior in ways that break downstream evals or user expectations.

The discipline is simple: always specify an exact, date-stamped model version in production. Upgrades are deliberate: you run your eval suite against the new version, confirm it passes, and promote the change. This way a model update is a planned event with a known quality baseline, not a surprise.

Sources

• Error handling and versioningAnthropic

Schema-forced output constrains the model to produce JSON that matches a schema your code already knows about. Instead of hoping the model formats its response correctly and then writing defensive parsing logic, you declare the shape up front and the API guarantees conformance.

This is the foundation of any LLM feature that feeds structured data into a downstream system, such as an extraction pipeline, a classification service, or a tool call result. It replaces a fragile 'parse and hope' pattern with a validated contract.

Sources

• Structured outputsAnthropic
• Structured outputs guideOpenAI

Constrained decoding enforces valid output at the token level: during generation, any token that would make the output invalid against the grammar is assigned zero probability and cannot be sampled. The result is that invalid JSON is literally impossible to produce, not merely unlikely.

This is stronger than schema-forced prompting, which still relies on the model following instructions. Constrained decoding is used in self-hosted serving engines such as SGLang and Outlines, and is what powers the strict mode in the Anthropic and OpenAI structured output APIs. For production systems that cannot tolerate malformed output, it is the right technical choice.

Sources

• Structured outputsAnthropic

Pydantic is Python's most widely used data validation library. You define a model as a class with typed fields, and Pydantic validates incoming data against that schema at parse time, raising a detailed error if anything does not fit. When used at the boundary where LLM output enters your system, it turns 'the model returned something' into 'the model returned exactly what we expected'.

For structured outputs specifically, you pass a Pydantic model class to the API client or validation layer and get back a typed Python object, not a raw string. This eliminates an entire class of runtime errors from malformed or missing fields.

Sources

• Pydantic documentationPydantic
• Structured outputs with PydanticAnthropic

Zod is a TypeScript-first schema declaration and validation library. You define a schema as a Zod object, call `.parse()` on the LLM output, and get back a fully typed value or a thrown error with precise field-level messages. The schema also doubles as the TypeScript type, so there is no separate type annotation to keep in sync.

In LLM applications, Zod is commonly used with the Vercel AI SDK's `generateObject()` to enforce that the model returns the exact shape a TypeScript component or API route expects. The zero-dependency, small-bundle design makes it practical on both server and client.

Sources

• Zod documentationZod
• Structured data generation with ZodVercel

Even with schema enforcement, edge cases and model updates can produce outputs that fail validation. The validate-then-repair pattern handles this gracefully: on a parse failure, send the model the original prompt plus the validation error and ask it to correct its output, then retry validation once. If the second attempt fails, surface an error to the caller.

This is more robust than a hard-fail on the first parse failure and far safer than passing an invalid value downstream. The single retry catches the most common failure mode (a field slightly out of range or an extra key), while the hard stop on the second failure prevents silent corruption.

Sources

• Structured outputsAnthropic

When a model response fails schema validation and the repair attempt also fails, the right behavior is to return an error to the caller, not to pass the malformed value through. Passing a bad parse downstream turns a visible API error into a silent data corruption, which is far harder to detect and fix.

Fail-closed means the system treats a bad parse exactly like a network error: it surfaces the failure, logs the raw response for inspection, and does not let the malformed data propagate. This is the same discipline that makes type systems useful: the invariant is enforced at the boundary, not optimistically hoped for throughout.

Sources

• Structured outputsAnthropic

A tool definition is the schema you expose to the model: a name, a description, and a JSON Schema for the parameters. The model uses the description to decide when to call the tool and uses the schema to fill in the arguments. A well-written definition is the primary lever for making tool use reliable.

The key practitioner insight is that the description does most of the work. Clear, specific descriptions of what the tool does and when to use it reduce both missed calls (model ignores the tool when it should use it) and hallucinated calls (model invents a tool call where none is needed). The parameter schema should match the TypeScript or Pydantic types your handler already validates, so invalid calls fail at the boundary.

Sources

• Tool use with ClaudeAnthropic
• AI SDK Core: tool callingVercel

When a task requires calling multiple independent tools, a capable model can issue all of them in a single turn rather than waiting for each result before making the next call. Fanning out three tool calls in one turn and then processing the three results together cuts round trips and wall-clock time substantially.

The production pattern is to always check whether returned tool calls are independent before executing them sequentially. If they are independent, execute in parallel. If one call's output feeds another's input, execute in order. The Vercel AI SDK and Anthropic SDK both handle the multi-tool response format; your dispatch code is responsible for the parallelism.

Sources

• Tool use with ClaudeAnthropic
• AI SDK Core: tool callingVercel

The Vercel AI SDK is a TypeScript library that provides a unified interface for calling LLMs from multiple providers, handling streaming, tool loops, and structured output generation from a single typed API. The `generateText`, `streamText`, and `generateObject` functions abstract over provider differences, so switching from one model to another is a one-line change.

For tool use specifically, the SDK manages the agentic loop: it calls the model, detects tool-use responses, dispatches the tool handlers you provide, sends results back to the model, and repeats until the model stops calling tools. This removes most of the boilerplate from building a reliable tool-using feature.

Sources

• AI SDK Core: tool callingVercel
• Structured data generationVercel

The Anthropic API's tool use feature lets you define a set of tools and have the model decide which to call, with what arguments, and in what order. The model returns a `tool_use` stop reason with one or more tool call blocks; your code executes them and sends results back as `tool_result` blocks in the next turn.

Parallel tool calls are supported natively: the model can return multiple `tool_use` blocks in one response, and the API accepts multiple `tool_result` blocks in one turn. The strict tool use option (adding `strict: true` to a tool definition) guarantees the arguments always match the declared JSON Schema exactly, removing the need for defensive validation in your handler.

Sources

• Tool use with ClaudeAnthropic

The model proposes tool calls; it does not execute them. That separation is where authorization lives. Before any tool action runs, your code must check whether the current user, in the current session context, is allowed to invoke that tool with those arguments.

Skipping this check turns tool use into an ambient authority attack surface: a prompt injection in retrieved content or user input can craft a tool call that the model faithfully forwards and your code faithfully executes. Authorization at the dispatch layer, not in the prompt, is the correct defense. This means every tool handler should receive the request context and verify permissions before touching any system.

Sources

• Tool use with ClaudeAnthropic

Tool calls can be retried: a network timeout, a model that re-issues a call after a context replay, or an agent loop that restarts from a checkpoint can all cause the same logical action to be dispatched more than once. If the tool handler is not idempotent, retries cause double-sends, double-charges, or duplicate writes.

The fix is to design every tool action as if it will be called twice. For external APIs this means using idempotency keys. For write operations it means checking whether the effect already exists before applying it. For anything financial it means tracking a transaction ID and deduplicating on it. Building idempotency in from the start is far cheaper than debugging the race conditions that appear when you do not.

Sources

• Tool use with ClaudeAnthropic