LLM Observability & Tracing

LLM observability is the practice of recording what happened inside each request to a language model feature: the exact rendered prompt, the chunks retrieval returned, the tool arguments and results, the token counts and the time each step took. The records are kept as spans in one trace per request, so when an answer is wrong you can find the step that made it wrong.

Ordinary monitoring cannot do that job. When a web request fails you get a stack trace pointing at a line of code. When an LLM feature gives a bad answer, every component returned 200 OK, nothing threw, and the model produced something wrong. Run the same input again and you may get a different output, so the only record of what the model was shown is the one you kept.

Why flat logs fall short#

A production LLM feature is rarely one call. One question to a RAG assistant can embed the query, search an index, rerank the hits, build a prompt from templates, call the model, make a tool call and call the model again with the result. Flat logs give you a line per step, spread across services and mixed in with every other request running at the time.

Suppose a customer reports that the assistant said the refund window is 14 days when it is 30. You find a model call near the right timestamp, but not the rendered prompt, because logging full prompts looked expensive. You cannot tell which chunks were retrieved, whether the reranker pushed the right passage to position 12, or whether the model saw the correct clause and contradicted it. Each of those needs a different fix (the index, the reranker, the prompt or the model), and flat logs cannot tell them apart.

Spans and the waterfall#

The fix comes from distributed tracing. Each operation becomes a span with a start time, a duration and attributes. Spans nest into a tree under one trace ID for the whole request, and you read the tree as a waterfall. For LLM work the attributes carry the payload: model name, rendered prompt, completion, input and output token counts, cost, retrieved document IDs, tool arguments and results.

Read the waterfall above for the refund bug. Retrieval took 120 ms and its span lists the chunk IDs it returned: the 30-day clause is there, at rank 2. The generate span shows the rendered prompt, and the clause made it in. So the model contradicted its context, and the fix is the prompt or the model, not the index. The same waterfall answers latency and cost questions: the two generation spans take 2 of the 2.5 seconds, so caching or a faster model is the only latency fix worth discussing.

What we put in a trace#

Trace every stage that can fail and the state the system ended in, not only the model calls. We trace our own builds that way because most of the failures we have seen were not model failures. On Skionis, our DevOps agent on live AWS, one record links the request, the account facts the agent read, the generated template, every validation failure and its revision, the architecture preview, the provider's response, the retries and the resources that resulted. The validation failures are the part worth keeping, because they are the system's own record of what the model gets wrong.

On Solarpunk, the executive operations desktop agent, each run's trace carries the goal, the plan, the tools discovered, every call and observation, each critic decision, the retries, any escalation, the final result and the resulting state in the target application. That last field matters most. Comparing the claimed completion with the actual state of the application is how the agent catches itself reporting success on work that did not land.

On AnyTeam, the on-device transcription build, the stages report their own state: detection, audio routes, interruptions, transcription timing, resource use, extraction and the CRM handoff. Without that, every support ticket would become a model investigation, when most of them are an audio routing problem. On the insurance voice agent we built, latency is instrumented per stage, because "the agent felt slow" cannot be checked otherwise. On the real-estate voice agent built on the same runtime, those per-stage timings showed the Hindi and Hinglish paths spending their extra time in transcription, not in the model.

OpenTelemetry's GenAI conventions#

Every observability vendor used to invent its own span schema, so instrumenting for one locked you to it. OpenTelemetry's semantic conventions for generative AI standardise the names. Operations are named chat, execute_tool and so on, attributes include gen_ai.request.model, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, and metrics cover operation duration and token usage. There are pages for OpenAI, Anthropic, AWS Bedrock and Azure AI Inference, and separate conventions for agent spans. Prompt and completion content is not captured by default; instrumentations are told to make it opt-in, which matters because traces routinely hold customer data.

As of October 2026 every part of the conventions is still marked Development, client calls and agent spans alike, and in June 2026 they moved out of the main specification into their own repository. Names can still change, and some instrumentations, such as OpenTelemetry's Python library for OpenAI, use the OTEL_SEMCONV_STABILITY_OPT_IN setting to choose which version they emit. Instrument against them anyway, because a rename in a standard is a migration you can plan, while a vendor's private schema ties every trace you have to that vendor.

Choosing where traces go#

The platforms share a core of traces, costs, evals and prompt management, and differ in where they run and who owns them. Langfuse is self-hostable with an MIT-licensed core, and joined ClickHouse in January 2026. LangSmith comes from the LangChain maintainers and traces LangChain and LangGraph apps with an environment variable. Arize Phoenix is built on OpenTelemetry and is self-hostable, under the Elastic License 2.0 rather than an open source licence, with Arize AX as the managed product. Datadog, Grafana and New Relic now accept GenAI-convention spans too, which helps if traces should sit next to your infrastructure telemetry.

Choose on the constraint you cannot move. If customer prompts may not leave your network, that decides it before features do: self-host, or keep content capture off. After that, the framework you already use and whether evals should run on production traces narrow the field. Emitting the OpenTelemetry conventions keeps the choice reversible.

Trace before you change the prompt#

When an LLM misbehaves, the usual move is to rewrite the prompt on a hunch, redeploy and look at a few outputs. That loop burns days and proves nothing, because nobody confirmed the failure was in the prompt. Open the trace for the bad request instead, find the first span where what happened differs from what should have happened, fix that component, then add the failing case to your evals so the regression stays caught. Instrument every model call, retrieval and tool call under one trace ID before launch, because adding tracing during an incident is the worst time to do it.