LLM Observability and Tracing: A Practical Guide for 2026

Traditional application monitoring answers "is the service up and how fast is it responding." LLM observability has to answer harder questions: what prompt actually went to the model, what did it decide to do with which tool, how many tokens did that cost, and why did it hallucinate on this specific input. Standard APM tools were never built for that. A new tooling category — LLM tracing — has formed around it, and in the last year it has largely converged on a shared standard instead of staying fragmented per vendor.

Why request logs aren't enough

A single user message to an agentic LLM app can fan out into a dozen internal operations: a retrieval call, three tool invocations, two model calls with different system prompts, and a final synthesis step. If you only log the final response, you cannot tell which internal step produced a bad answer, how much of the latency was model inference versus tool execution, or which step burned the token budget. Multi-step, multi-agent systems make this worse — a single user turn in a multi-agent pipeline can touch several models, each with different cost and latency profiles (Langfuse docs). Without structured tracing, debugging becomes reading raw JSON logs and guessing.

The OpenTelemetry GenAI semantic conventions

The industry settled on extending OpenTelemetry (OTel) — the existing standard for distributed tracing — with a GenAI-specific attribute schema rather than inventing a parallel standard. The OpenTelemetry GenAI Semantic Conventions define a standardized set of span attribute names, metric instrument names, and event schemas for AI operations (Greptime).

Key attributes every compliant span should carry:

A typical trace for an agent turn shows a top-level invoke_agent span with nested chat spans for each LLM call and execute_tool spans for each tool invocation (OpenTelemetry blog). Content capture is separated from structural attributes deliberately: prompt/completion text is stored in span events, not attributes, specifically so it can be filtered or dropped at the Collector level for compliance reasons without touching application code (OpenTelemetry blog).

Span: invoke_agent (root)
├─ Span: chat  gen_ai.request.model="gpt-4.1"  gen_ai.usage.input_tokens=812
│    └─ Event: gen_ai.output.messages (tool_call: search_docs)
├─ Span: execute_tool  gen_ai.tool.name="search_docs"  duration=340ms
└─ Span: chat  gen_ai.request.model="gpt-4.1"  gen_ai.usage.output_tokens=214
     └─ Event: gen_ai.output.messages (final answer)

Datadog's Agent Observability now natively supports these conventions rather than a proprietary schema (Datadog), which is the clearest signal that OTel GenAI has become the interoperability layer, not just an LLM-specific side standard.

The tool landscape: managed vs. self-hosted

LangSmith is LangChain's own managed platform — the path of least resistance if you're already building with LangChain or LangGraph, covering tracing, evaluation, and debugging through the full development lifecycle (MLflow).

Langfuse is the open-source leader, with an MIT license and no restrictions on self-hosting. It covers tracing with multi-turn conversation support, prompt versioning with a built-in playground, and evaluation via LLM-as-judge, human feedback, or custom metrics (Langfuse docs).

Arize Phoenix and TruLens lean OTel-native and framework-agnostic, useful when your stack spans LangChain, LlamaIndex, and raw API calls in the same system (MLflow).

MLflow has extended into end-to-end GenAI lifecycle management — tracing plus evaluation plus model registry in one platform, useful if you already run MLflow for traditional ML (MLflow).

What to actually trace in production

Token counts and latency are the baseline, but the attributes that catch real production issues are the ones tied to correctness:

• Tool call arguments and results — the single most common agent failure mode is a model calling a tool with malformed or hallucinated arguments; you can't diagnose this from a final-answer log alone.
• Retrieval context — for RAG systems, log which chunks were retrieved and their relevance scores alongside the generation span, so a bad answer can be traced to bad retrieval versus bad generation.
• Finish reason — length truncation is invisible in the rendered UI but shows up immediately in gen_ai.response.finish_reasons.
• Cost per trace — input/output token counts multiplied by model pricing, rolled up per trace, is the fastest way to catch a runaway agent loop before the bill does.

Debugging a real failure pattern

Consider a support-bot agent that occasionally gives an answer contradicting the retrieved document. Without tracing, you'd see only "wrong answer" in a user complaint. With a proper trace tree, you can see: the execute_tool span for retrieval returned the correct chunk, but the chat span's gen_ai.input.messages shows the chunk was truncated before it reached the model — a context-window overflow, not a retrieval or model quality problem. That's a five-minute fix once visible, versus hours of blind prompt tweaking without it.

Instrumenting with minimal code changes

Most teams don't hand-write span creation. Framework-level auto-instrumentation (via OpenLLMetry, Langfuse SDK decorators, or Phoenix's OTel exporters) wraps existing LangChain/LlamaIndex/raw SDK calls automatically. The typical integration path:

1. Add the SDK's tracing decorator or middleware around your model client.
2. Configure an OTel Collector endpoint (self-hosted Langfuse, Phoenix, or a vendor backend).
3. Opt into content capture only in non-production or with PII redaction configured — content events carry real prompt/response text.
4. Set up cost and latency dashboards keyed on gen_ai.request.model to catch per-model regressions after a version bump.

Actionable takeaway

Don't build tracing on a vendor-specific schema if you can avoid it. Instrument against the OpenTelemetry GenAI semantic conventions from day one — gen_ai.operation.name, gen_ai.request.model, token usage, and finish reasons at minimum — and pick a backend (Langfuse if you want self-hosted and open source, LangSmith if you're already deep in LangGraph) based on team fit rather than schema lock-in. The conventions are now stable enough, and adopted widely enough (Datadog, MLflow, Phoenix, Langfuse all support them), that OTel-first instrumentation is the safer long-term bet over a proprietary SDK.

Sources: Langfuse Docs, MLflow: Top LLM Observability Tools in 2026, MLflow: Top 5 Agent Observability Tools, Greptime: OpenTelemetry GenAI Semantic Conventions, OpenTelemetry Blog: Inside the LLM Call, Datadog: LLM OTel Semantic Convention