Agent Auditor

Case study: Agent Auditor watching its own chat agent

I ran Agent Auditor against the demo agent in this repo (Auditor Chat) for about a day, switching between many models and providers, and read what the dashboard reported. This is what it showed, including where it fell short. Numbers come from the dashboard on 7 October 2026 (agent chat-agent, last 7 days). The run counts are small, so treat this as a worked example and not a benchmark.

Overview

1. The agent

2. Baseline

Measure Value Runs behind it
Runs with a problem 93.9% (24 errored, 38 answered badly) 66 runs
Quality checks passed 56% 168 of 302 checks
Median answer time 46 s (slowest 5%: 179 s) 66 runs
Top failure cause slow run, 32 of 62 flagged runs 62 flagged runs
Other causes llm error 24, tool error 3, user rejected 1 62 flagged runs
Error types other 13, timeout 6, rate limit 5, auth 2, not found 1, server error 1 failed runs

Each of the 62 flagged runs (24 errored plus 38 answered badly) gets one most-likely cause.

The headline number looks alarming, and the dashboard's own breakdown shows why it is misleading. Most flagged runs were slow, not wrong. For example, one run (e25be41e, AIRouter gpt-5-nano) answered the question correctly but took 23 s against the 10 s latency budget, so it was flagged as a slow run. Another model (deepseek-v4-flash) was flagged at 10.2 s for the same question. The rest were provider problems: timeouts, rate limits and rejected keys.

Failures

3. Failures turned into regression tests

Regression tests

4. A change gated by a canary

First attempt (7 October): no verdict. I used the Canary page to compare the active version (chat-8a662e2d) with three candidates. All four checks ended continue, "only N canary runs so far; need 20" (N was 0, 2, 3 and 0). That is the correct behaviour: with too few runs on both sides, the tool says so instead of guessing. It also meant my chat, which sends all traffic to production, never gave a candidate enough canary runs.

Second attempt (9 November): a real verdict. I wrote bench/canary.py, which asks a baseline model and a candidate model the same 24 questions, records them as production and canary runs, scores them, and asks the Auditor for a verdict. A first candidate (gemini-2.5-flash-lite) was refused by Google ("not available for this kind of request"), so it got 4 runs and the verdict was again continue. With a working candidate:

Baseline Candidate
Model Indos Deepseek/deepseek-v4-flash AIRouter openai/gpt-5-nano
Environment production canary
Answered / errored 24 / 0 24 / 0
Median answer time 1.7 s 4.4 s
95th percentile 2.9 s 9.0 s

Verdict: HOLD.

Check Result
evals pass: no significant pass-rate regression
latency fail: 3.16x the baseline (KS p = 4e-9)
causes pass: no failure cause became more common
drift warning: latency differs; "may come from routing rather than the candidate"
replay skipped: no regression replay for the candidate version
change pass: likely a model change (moderate), since only the model changed

How to read it: the candidate was not wrong or broken, it was slower. The Auditor held it back instead of rolling it back, because its rollback threshold is 4x on latency and this was 3.16x, and nothing else regressed. That is the decision I would have made by hand, but I got it with numbers and a named check behind it.

Third attempt (11 November): all three verdicts. I wrote bench/verdicts.py, which runs three checks on real models with the same 24 questions each and writes docs/real-verdicts.md. The numbers below are from that report.

Check Baseline (production) Candidate (canary) Median 95th percentile Verdict
A slower candidate Indos deepseek-v4-flash AIRouter gpt-5-nano 2.6 s vs 4.7 s 5.0 s vs 7.6 s promote (I expected hold)
A broken candidate (planted fault) Indos deepseek-v4-flash the same model through a wrong address 2.6 s vs 2.1 s 5.0 s vs 2.1 s rollback
A faster candidate AIRouter gpt-5-nano Indos deepseek-v4-flash 4.3 s vs 1.9 s 8.6 s vs 4.2 s promote

5. What it caught that I would have missed

6. What did not work

Audit trail

7. Reproduce it