LLM Observability & Tracing

When a normal web request fails, you get a stack trace pointing at a line of code. When an LLM feature gives a bad answer, you get nothing: every component returned 200 OK, no exception fired, and the model simply produced something wrong. Run the same input again and you may get a different output. LLM observability is the discipline of recording enough about each call, the exact rendered prompt, the retrieved chunks, the tool arguments, the token counts, the latency, that you can reconstruct what happened and why, even though nothing technically errored.

Why ordinary logging falls short#

A production LLM feature is rarely one call. A single user question in a RAG assistant might embed the query, search a vector store, rerank the hits, assemble a prompt from templates, call the model, trigger a tool call, and call the model again with the result. Traditional logging gives you flat lines from each step, scattered across services and interleaved with every other concurrent request.

Now a customer reports: "the bot told me our refund window is 14 days, it is 30." You grep the logs. You find a model call at roughly the right timestamp, but not the rendered prompt, because logging full prompts felt expensive. You cannot tell which document chunks were retrieved, whether the reranker buried the right passage at position 12, or whether the model saw the correct clause and contradicted it anyway. Each of those diagnoses demands a completely different fix (index, reranker, prompt, model), and flat logs cannot distinguish them. What you need is the steps stitched into one causal story per request, with payloads attached.

Span-based tracing#

The fix is borrowed directly from distributed systems tracing. Each operation becomes a span with a start time, duration, and attributes; spans nest into a tree under one trace ID for the whole request; and you read the result as a waterfall. For LLM work the span attributes carry the domain-specific payload: model name, rendered prompt, completion, input and output token counts, cost, retrieved document IDs, tool arguments and results.

Read the waterfall above for the refund bug. Retrieval took 120 ms and its span attributes list the chunk IDs it returned: the 30-day clause is there, at rank 2. The generate span shows the rendered prompt: the clause made it in. So the model contradicted its context, which means the fix is prompt instructions or model choice, not the index. That diagnosis took two minutes instead of an afternoon of speculation. The same waterfall answers cost and latency questions for free: the two generation spans account for 2 of the 2.5 seconds, so caching or a faster model is the only optimization worth discussing.

OpenTelemetry GenAI semantic conventions#

For years every vendor invented its own span schema, so instrumentation locked you in. OpenTelemetry's GenAI semantic conventions fix that by standardizing the names: spans like chat and execute_tool, attributes like gen_ai.request.model, gen_ai.usage.input_tokens, and gen_ai.usage.output_tokens, plus metrics for operation duration and token throughput. There are provider-specific pages for OpenAI, Anthropic, and Bedrock, and newer conventions for agent and framework spans. Prompt and completion content is off by default and opt-in, which matters because traces routinely contain customer PII.

As of mid 2026 the conventions are still officially in development status: the core client-call conventions are settling while the agent spans remain experimental, and an environment flag (OTEL_SEMCONV_STABILITY_OPT_IN) controls which version your instrumentation emits. The practical takeaway is unchanged: instrument against the standard once, and your traces become portable across Langfuse, Phoenix, Datadog, Grafana, and whatever you migrate to in two years.

The 2026 tool landscape#

The platforms have converged on the same core (traces, costs, evals, prompt management) and differ in emphasis. Langfuse is the open-source default: MIT licensed, self-hostable, OTel-compatible, acquired by ClickHouse in January 2026 to deepen its analytics backend. LangSmith ships from the LangChain maintainers and auto-instruments LangChain and LangGraph apps, making it the path of least resistance for those stacks. Braintrust treats evaluation as the first-class object, with scores living directly on production traces; it raised an $80M Series B in early 2026. Arize Phoenix is open source and OpenTelemetry-native via the OpenInference instrumentation set, with Arize AX as the managed tier. The general-purpose APM vendors (Datadog, Grafana, New Relic) now ingest GenAI conventions too, which is convenient if your traces should live next to your infrastructure telemetry. Pick by deployment constraint (self-hosted vs SaaS), framework, and how much you care about evals being native rather than bolted on.

Trace before you guess#

The cultural payoff is a single rule: trace before you guess. The reflexive move when an LLM misbehaves is to rewrite the prompt on a hunch, redeploy, and eyeball a few outputs. That loop burns days and proves nothing, because you never confirmed the failure was in the prompt at all. The disciplined loop is: open the trace for the bad request, find the first span where reality diverged from intent, fix that component, then turn the failing case into an eval so the regression stays caught. Teams that adopt this stop having debates about why the bot misbehaved and start having lookups.

Practical takeaways#

Instrument every model call, retrieval, and tool invocation as spans under one trace ID from day one; retrofitting tracing during an incident is the worst time to do it. Emit OpenTelemetry GenAI conventions so the data outlives your current vendor. Capture full prompts and completions where privacy rules allow, with content capture consciously configured rather than defaulted. Record token counts and cost per span, because the waterfall is also your cost model. And enforce the discipline that no prompt change ships in response to a bug until someone has looked at the trace, because in an LLM system the trace is the only stack trace you get.