Agent Auditor

Catching wrong answers when you have no answer key

A canary compares a new version of an agent with the live one. Errors and slowness are easy to see. A wrong answer is harder: a candidate that replies "I am not sure." to everything, or one that replies fluently and gets the facts wrong, throws no error and is not slow. This page says what Agent Auditor can catch, how, and what it still cannot.

What catches what

The candidate... Expected answers on the runs (tags.expect) No expected answers (most real traffic) Judge on (answer_judge)
errors or times out caught caught caught
is much slower caught caught caught
answers "I am not sure" or "I can't help with that" caught (expectations_met) caught (gives_an_answer) caught
answers fluently and wrongly caught (expectations_met) not caught caught, as well as the judge grades
is wrong in a way only a domain expert would notice only if you wrote that rule not caught not caught reliably

The middle column is the honest limit. With no answer key and no judge, a confident wrong answer looks like any other answer.

1. Give the runs an answer key

If you have questions with known answers (a regression suite, a benchmark, a support FAQ), tag each run with what a good answer must contain. This is exact, free and needs no model:

with auditor.run(input=question, tags={"expect": [{"type": "output_contains", "value": "Canberra"}]}):
    ...

The rule types are in agent_auditor/evals/expect.py (contains, does not contain, contains any, matches a pattern, word limits). The scorer is expectations_met. bench/canary.py does this for its 14 questions.

2. gives_an_answer: non-answers, with no labels

On by default. For a short answer (30 words or fewer, a setting) it flags "I am not sure", "I don't know", "I can't help with that", "as an AI" and similar, and the common Spanish and French equivalents ("no estoy seguro", "no lo sé", "no puedo ayudar", "je ne sais pas", "je ne peux pas vous aider", "désolé, mais"). Longer answers pass, because a long answer with one hedge is usually still an answer. Runs that carry tags.expect are skipped, because expectations_met knows the right answer.

What to expect from it:

3. answer_judge: a model grades each answer

Off by default, because it costs money, sends each run's question and answer to the model you choose, and can be wrong. Turn it on by pointing the Auditor server at any OpenAI-compatible chat endpoint:

AUDITOR_JUDGE_BASE_URL=https://api.openai.com/v1
AUDITOR_JUDGE_API_KEY=...
AUDITOR_JUDGE_MODEL=gpt-5-mini

Then ask for it by name, in an evaluation ("scorers": ["answer_judge"]) or in a canary check:

report = auditor.evaluate_canary(baseline="v1", candidate="v2",
                                 scorers=[*default_scorer_names(), "answer_judge"])

or, with the chat benchmark, python bench/canary.py --baseline a --candidate b --unlabeled --judge.

How it behaves:

See it work (offline, about 5 seconds)

python bench/wrong_answers.py

It plays two scripted candidates against a good baseline, 28 runs each, with the expected answers hidden:

candidate A: answers "I am not sure." to everything
  built-in checks before this change             PROMOTE
  built-in checks now (adds gives_an_answer)     ROLLBACK    caught by: gives_an_answer
  built-in checks plus answer_judge              ROLLBACK    caught by: answer_judge, gives_an_answer

candidate B: confident, fluent, and wrong
  built-in checks before this change             PROMOTE
  built-in checks now (adds gives_an_answer)     PROMOTE
  built-in checks plus answer_judge              ROLLBACK    caught by: answer_judge

"Before" is the same runs scored with the checks that existed before gives_an_answer. These are scripted models, and the judge here is a scripted stand-in that grades with the benchmark's own answer key, so it shows how the verdict behaves, not how well a real judge model grades. Candidate B's PROMOTE in the middle row is the point: no heuristic sees it.

What is still open