We've shipped enough production agents now to know which traces matter, which evals catch real regressions and which dashboards no one ever opens. Here's what we instrument and why.
The first production agent we shipped failed silently for nine days. It was a customer-support triage agent - a router with three tools and a small policy prompt. The standard application monitoring was green the entire time. Latency was fine, error rates were near zero, the model endpoint returned 200s on every call. What the monitoring couldn't see was that one of the tools had started returning truncated JSON, the agent had stopped calling it after the first failure and was now answering questions it shouldn't have been answering - confidently and politely. We only found out because a customer escalated.
That outage forced us to rebuild how we observe agents from scratch. What we had was logging dressed up as observability. What we needed was a different discipline.
Traditional application performance monitoring assumes failures are localised - a request fails, a span errors, a counter goes up. Agent failures are almost never like that. They are causal chains. The model picks the wrong tool at step two, which returns a degraded result at step three, which gets summarised into context at step four, which biases the final answer at step seven. Every individual step returned a 200. The system failed.
This is the single most important mental shift. You're not monitoring a request, you're monitoring a decision path. If your observability can't answer "why did the agent do that?" - not "what did the agent return?" - you don't have agent observability yet.
We settled on three layers, each answering a different question.
1. Structured traces with nested spans. Every agent run produces a trace tree. The root span is the user request. Underneath it sit spans for each model call, each tool invocation, each retrieval, each memory read or write. Parent-child relationships are preserved across handoffs between sub-agents. The trace tells the story of the decision. We use OpenTelemetry conventions so the same trace data feeds both our internal viewer and the broader APM stack.
The non-obvious thing here is what to put on each span as attributes. The bare minimum: the prompt or tool input, the output, the model or tool identifier, token counts, latency and a stable step type tag. The stable tag is what lets us aggregate later - "all retrieval steps in the last week, grouped by failure reason" only works if every retrieval step is tagged the same way.
2. Evals that run on production traces, not just in CI. This is the part most teams skip. Pre-deployment eval suites are necessary but they freeze in time the day you ship. The agent's real input distribution drifts within weeks. We run a sampled subset of production traces through the same eval suite - LLM-as-judge for subjective quality, deterministic checks for policy compliance, regression tests for known failure modes. Drift on any of these surfaces as an alert, not a dashboard tile no one reads.
When a production trace fails an eval, it gets one-click promoted into the regression suite. That feedback loop - failure becomes test case becomes deployment gate - is what keeps the eval suite alive.
3. Cost and step-count budgets per run. Every agent run has a hard budget on token spend and step count. Crossing the budget triggers an alert and, depending on environment, terminates the run. This sounds defensive but it is the single highest-value alert we have. Runaway loops are the most common agent failure in production and they are invisible to ordinary monitoring until the bill arrives.
# Simplified span attribute schema we standardised on
{
"step.type": "tool_call", # stable enum
"step.name": "search_knowledge_base",
"step.input": "...", # redacted in shared envs
"step.output": "...",
"step.tokens.input": 1248,
"step.tokens.output": 312,
"step.latency_ms": 840,
"step.eval.passed": True,
"run.budget.tokens_remaining": 14200,
}
Per-token logging at full fidelity in production. Tempting and ruinously expensive. We sample. The sampling rate is higher for new agents, lower for stable ones and goes to 100% temporarily when an eval starts flagging drift.
Synthetic dashboards no one looks at. The first version of our observability stack had eleven dashboards. Three were ever opened. We deleted the rest. If a metric isn't tied to an alert or a weekly review, it's noise.
LLM-as-judge on the critical path. Quality evals are slow and themselves cost tokens. Running them inline blocks the user response. All quality evals dispatch to a background worker. Only fail-fast heuristic checks - secret-leak detection, output-length sanity, schema validation - run on the critical path.
The single most useful trace attribute turned out to be the one we almost didn't add: the model's reasoning text on each step, when the model exposes it. Without it, you see what the agent did. With it, you see what the agent thought it was doing. The gap between those two is where almost every interesting failure lives.
The second surprise was how much of our debugging shifted from logs to trace replay. Our trace viewer lets us re-run any historical trace from any step with modified inputs. Most of the bugs we fix now are diagnosed by replaying, not by reading logs.
Agent observability is not LLM logging with extra steps. It's tracing the decision path, evaluating the trace on quality not just success and budgeting runs against cost and step count. Get those three layers in place and most production agent failures move from "discovered by a customer nine days later" to "alerted on within the hour." Skip them and you'll discover, as we did, that the monitoring being green tells you nothing about whether the agent is doing the right thing.