How we run evals in production

This document describes the eval suite that every agent we ship has to clear before it goes near a production environment, and the same suite that runs against it every day after it ships. It is the most load-bearing piece of engineering in a Svolta build. Nothing else we do is allowed to break it.

The suite has four layers: unit, regression, behaviour, and safety. Each layer answers a different question, runs at a different cadence, and blocks a different class of change.

The golden dataset comes first

Before any of the layers exist, the golden dataset exists. We will not start an engagement that does not produce one. It is the artifact that the rest of the suite is built around, and the artifact that the rest of the system is judged against.

A golden dataset for an agent is between 100 and 500 input/output pairs that represent the actual job the agent is going to do. Each pair is curated with at least one subject matter expert from the client’s team. Each one has:

• The input the agent will see (a ticket, a contract clause, a customer query, a transaction record).
• The correct output, written by the expert, in the form the agent is expected to produce.
• A rationale field explaining why that output is correct, so the next person reviewing the dataset can sanity-check it.
• An edge-case classification tag (happy_path, ambiguous, out_of_scope, compliance_sensitive, historical_anomaly).

The dataset is version controlled. It lives in the repo. It is the source of truth.

# evals/golden/ticket-triage.yaml (illustrative sample, not client data)
- id: t-0142
  input: |
    Customer reports their card was declined on a recurring payment.
    Account flag shows fraud_review_pending. Last login 14 days ago.
  expected:
    routing: tier2_fraud
    policy: P-FR-014
    escalation_window_seconds: 60
  rationale: |
    Fraud review pending + payment failure routes to tier 2 under P-FR-014.
    60s SLA because cardholder is currently locked out.
  tags: [compliance_sensitive, happy_path]

Subject matter experts hate writing these. They do it once, with one of our engineers in the room, and then the asset compounds for the life of the engagement.

Layer 1: unit evals

Unit evals check the smallest decisions the agent makes in isolation. If the agent is supposed to extract an account number from a free-text message, the unit eval is 200 messages and the account numbers we expect to see come out. If a tool wrapper is supposed to return ISO-formatted dates, the unit eval is 50 date strings and the expected normalisation.

These run on every commit. They are fast, deterministic, and they fail loudly. They catch the boring 80% of regressions that would otherwise leak into the higher layers and confuse the diagnosis.

// evals/unit/extract-account.spec.ts
import { extractAccount } from '../../src/tools/extract-account';
import golden from './fixtures/account-extraction.json';

for (const { input, expected, id } of golden) {
  test(`extract-account/${id}`, () => {
    expect(extractAccount(input)).toEqual(expected);
  });
}

Unit evals do not call a model. They are a regression net for the deterministic glue. If they fail, the system is broken in a way no prompt change can mask.

Layer 2: regression evals

Regression evals run the agent end-to-end against the full golden dataset and compare the output to the expected output, using rubric-graded scoring per field. They run on every pull request that touches an agent file, every model upgrade, every prompt change, every retrieval-pipeline change. They take five to twenty minutes depending on the dataset size.

The rubric matters more than the scoring code. We score each field as match, acceptable_variant, or mismatch, with acceptable_variant reserved for cases where the expert agrees the output is correct even though it differs from the canonical answer. A field-level rubric forces conversation about what changed, not just whether the score went up or down.

The PR is blocked if any of:

• Pass rate drops more than two points absolute on any tag.
• Pass rate on compliance_sensitive items drops at all.
• A previously-passing item now fails (regression, even if pass rate is unchanged in aggregate).

The third gate is the most important one. Aggregate scores hide a lot. A change that fixes ten edge cases and breaks ten different edge cases is not a wash. It is a behaviour change that needs an explicit decision.

Layer 3: behaviour evals

Behaviour evals are the layer that tests properties, not specific input/output pairs. They are how we know the system is doing the right kind of thing across a much larger surface than the golden dataset can practically cover.

A behaviour eval might be: “for any input where the policy ID is present and unambiguous, the agent’s response must cite that exact policy ID.” We generate a few thousand synthetic inputs that satisfy the precondition, run the agent across them, and assert the property holds for all of them. We do this for invariants like:

• Source citations always resolve to a document in the corpus.
• Numeric quantities in the output appear verbatim in the retrieved context (no hallucinated numbers).
• The agent never claims to have done a side-effecting action it didn’t actually do.
• Escalation paths comply with the policy hierarchy.

Behaviour evals are where the agent’s safety properties live. A regression eval can pass while a critical invariant is silently broken. Behaviour evals are the gate that catches that.

Layer 4: safety and adversarial

Safety evals are a dedicated set of prompts and inputs designed to surface failures we care about even when they are rare. Prompt injection through retrieved content. Attempted exfiltration of system instructions. Polite-but-disallowed requests. Inputs designed to trigger over-confident answers in cases where the system should refuse or escalate.

This layer runs less frequently (nightly, not per-PR) because the dataset is larger and slower. But it ships with every release, and it has its own block gate: any new failure here pages the on-call engineer.

Why we don’t trust LLM-as-judge alone

LLM-as-judge has its place. We use it to score open-ended free-text outputs where a rubric is too rigid. But it is a complement, not a replacement, for the four layers above. Three reasons:

1. It is correlated with the system under test. If we use the same model family to judge that we use to generate, both sides drift together. The judge becomes a poor estimator of human-perceived quality at exactly the moment we need a good estimator.
2. It is biased toward verbose, hedged, plausible-sounding outputs. Most of the production failure modes we care about look like that.
3. It is not auditable. When a regulated client asks “why did this release pass eval,” “the LLM judge gave it a 0.87” is not an answer that survives the conversation. A field-level rubric with subject matter expert sign-off is.

So we use LLM-as-judge as a noisy continuous signal in the regression layer for free-text fields, but the block/no-block decision is always made against the rubric, not the judge.

Drift detection in production

The suite does not stop running when the release ships. It runs against a sampled stream of production inputs every day, scored the same way as the regression layer, and the scores are tracked over time. We alert on:

• A statistically significant drop in pass rate week-over-week.
• A new failure mode that did not appear in the golden dataset (we add it).
• A model provider version change that we did not initiate (silent updates do happen).

When drift is detected, the failing examples get added to the golden dataset, and the regression layer is retriggered against the current production system. If it fails, we roll back. If it passes, the new behaviour was acceptable and the dataset is now stronger for it.

The discipline this enforces

The suite is the thing that lets a senior engineering team ship into a regulated environment with their name on it. It is also the thing that lets us walk into a diagnostic, look at a client’s existing agent, and have a falsifiable conversation about whether it works.

Most of the value is not in any individual layer. It is in the discipline of having every change in the system go through the same gate, scored the same way, against the same dataset. The gate is what makes the system reproducible. Reproducibility is what makes the system production-grade. Nothing else does.