Our observability stack for agents

The single fastest way to lose trust in a production AI system is to ship it without observability and then have nobody, including the team that built it, able to answer the first operational question. “Why did the agent route that ticket to legal?” “Why did response times double yesterday?” “Are we still meeting the quality bar we committed to in the contract?”

Every agent we ship answers those questions on day one. The observability stack is not retrofitted. It is built before the agent is, and the agent’s structure is shaped by what needs to be observable.

This document is the surface we ship: what is captured, where it goes, what is shown to whom, and what pages the on-call.

What we capture, per call

Every call to a model, including retrieval, reranking, tool calls, intent classification, and the agent’s primary reasoning step, produces a structured event. The schema:

interface AgentCallEvent {
  trace_id: string;            // groups all calls within a single user-facing request
  span_id: string;             // unique per call
  parent_span_id: string | null;
  tenant_id: string;
  user_id: string | null;      // null when system-initiated
  intent: string;              // tagged by the upstream classifier
  step: 'classify' | 'retrieve' | 'rerank' | 'reason' | 'tool' | 'judge';
  provider: string;            // e.g. 'anthropic', 'openai', 'local'
  model: string;               // specific versioned model name
  prompt_hash: string;         // sha256 of the system+user prompt
  input_tokens: number;
  output_tokens: number;
  latency_ms: number;
  cost_cents: number;
  status: 'ok' | 'fallback_used' | 'error' | 'filtered';
  error_class: string | null;
  retrieved_doc_ids: string[]; // when step is 'retrieve' or 'rerank'
  eval_score: number | null;   // populated by the async judge worker
  created_at: string;
}

These events go to two destinations: an OpenTelemetry-compatible trace backend (so engineers can debug end-to-end), and a columnar warehouse (so the dashboards can aggregate cheaply). The choice of specific backend depends on what the client is already running. The stack runs on Honeycomb, Tempo, ClickHouse, BigQuery, or a client-internal system. The schema is the same regardless.

The prompt_hash is load-bearing. It lets us answer “are these two responses actually different because the prompt changed, or were the inputs different” without storing every prompt verbatim (which gets expensive and creates a separate retention problem).

Traces: end-to-end per request

Every user-facing request gets a trace. A trace shows every step the agent took, in order, with the latency, cost, and (where applicable) the retrieved context and the eval score. An engineer debugging a specific bad answer opens the trace, sees:

• The intent the request was classified as.
• The router decision and the reason for it.
• The retrieval call, the documents returned, the reranker output, and the final context passed to the model.
• The primary reasoning step, its prompt hash, and its raw output.
• Any tool calls the agent made, with their inputs and outputs.
• The eval score, if the async judge has scored it yet.

That trace, plus the production stream of similar traces from the same intent over the last 24 hours, is enough to diagnose almost every failure we encounter without rebuilding the agent state locally. We can do it locally, but we usually do not have to.

Live eval scoring in production

The eval suite described in our evals doc runs in CI to gate releases. A reduced version of it runs on a sampled stream of production responses to track quality over time.

The async judge worker:

1. Samples 5 to 10 percent of production responses, biased toward intents marked compliance_sensitive (which are sampled at 100%).
2. Scores each one against the rubric, using the same scoring logic that runs in CI.
3. Writes the score back to the trace and to the warehouse.
4. Flags any individual failure for human review through a queue that gets triaged daily.

This is the loop that catches drift. If the model provider silently updates a model version, if the corpus drifts because the client started adding new document types, if a prompt change interacts badly with a class of inputs we did not see in CI, it shows up here. First as a single failure, then as a pattern, before it becomes an incident.

Dashboards ops leaders look at

We build two surfaces. The engineering surface (traces, raw events, debugger views) is for the engineers. The ops surface is for the customer’s operations leader, and it is the one we put the most editorial work into.

The ops dashboard is one screen. Anything that doesn’t fit on it is wrong. It shows:

Panel	What it answers
Volume served (24h, 7d)	Are we still handling the workload we agreed to?
Quality score (rolling 7d, by intent)	Is the system meeting the quality bar?
Resolution rate (auto-resolved vs. escalated)	Are humans being recovered at the rate we promised?
p50 / p95 latency, by intent	Is the user-facing experience holding up?
Cost per resolution, by intent	Are unit economics holding?
Cost trend vs. budget cap	Are we going to need to talk about a budget change?
Active incidents	Is anyone working on anything right now?

That is the entire dashboard. Six panels and a banner. An ops leader can take it in over coffee. The discipline is in what is not on it. There is no “model utilisation by provider” panel, no “embedding throughput” panel, no “vector index size” panel. Those things are real and they are on the engineering dashboard, where engineers look. Ops leaders look at outcomes.

The ops dashboard is also the contract surface. When we set quality and cost targets in the engagement contract, those targets are panels on this dashboard. The ops leader can verify, themselves, in real time, whether the system is hitting the commitments. We do not get to mark our own homework.

Alerts that page the on-call

There are five things that page on-call. Anything more would be noise; anything less would be irresponsible.

1. Hard quality regression. Sustained quality score on any intent drops more than 5 points below baseline for 30 minutes. (The 30-minute window prevents single-batch noise from paging.)
2. Compliance-sensitive failure. Any single failure on a compliance_sensitive intent’s eval pages immediately. There is no acceptable rate of these. The on-call engineer reviews the failure, decides whether to roll back, and reports to compliance within the SLA.
3. Fallback rate spike. The router’s fallback chain is firing for more than 2% of traffic. This means a provider is degraded. Even if the system is “still working”, the cost, latency, and quality assumptions are violated until the primary recovers.
4. Cost ceiling breach. A tenant exceeds 80% of its agreed budget cap on a trajectory to hit 100% within 24 hours. The on-call engages with the customer success team to either adjust traffic or extend the cap before the system starts hard-rejecting requests.
5. Drift signal. A new failure mode appears in production that did not appear in any prior week. This is a soft alert (the system is still serving), but it kicks off the loop of adding the failure to the golden dataset and re-running CI.

Every alert has a runbook. The runbook is in the repo. The runbook is tested annually with a tabletop exercise. An alert with no runbook is not allowed to ship.

What we deliberately do not capture

We do not log raw user inputs or raw model outputs in the observability stream by default. We log the prompt hash, the response hash, and the structured metadata. The raw text is held in a separate, access-controlled, retention-limited store that is queryable for debugging by authorized engineers and is purged on the contract’s retention schedule.

This is a difference that matters. Observability events have to be cheap to store, fast to query, and broadly accessible to the team. Raw prompts and responses often contain PII, PHI, or commercially sensitive content, and storing them in the observability path commingles those concerns. We separate them.

When an engineer needs to see the raw text for a trace, the trace links to a “raw context viewer” that re-fetches the text from the retention-controlled store, logs the access, and times out the view after an hour.

The cost of this stack

Observability done this way runs at roughly 15% of the engagement budget. Most teams arrive expecting it to be 2% (a logging line at the end of every call). The 15% earns its keep the day it matters, because it means the first incident resolves in minutes rather than days, and the first compliance audit is answered in an afternoon instead of a quarter.

We do not negotiate the 15%. The observability stack is the thing that makes the system operable by a senior team that did not build it. If a client cannot operate the system without us, we have built the wrong system. The observability stack is the part that prevents that.