Agent Auditor

Documentation

Agent Auditor is an alpha open-source reliability layer for AI agents. This page gets you running; the pages on the left go deeper.

Quick start

You need Python 3.11 or newer. The core needs no API key.

pip install agent-auditor
agent-auditor demo

agent-auditor demo starts the dashboard with made-up sample data (a support bot with a bad afternoon, a good candidate and a bad one) and opens it in your browser. Nothing is saved. You can also click through a hosted copy.

To run it for real with your own database, use agent-auditor serve. To work on the code, clone the repository and run pip install -e ".[dev]".

On Windows, powershell -ExecutionPolicy Bypass -File .\run-all.ps1 starts the Auditor and the chat app together. Add -Check to run a 23-point self-check.

Use it with your own agent

from agent_auditor.sdk import Auditor

auditor = Auditor("http://127.0.0.1:8000", agent="support-bot")
auditor.register_version("v1", prompt=PROMPT, config={"top_k": 4})

@auditor.observe(kind="retrieval")
def kb_search(q): ...

@auditor.observe_run()
def answer(question): ...

Plain Python and LangGraph/LangChain are supported. See examples/quickstart_plain.py and examples/langgraph_demo.py in the repository.

The dashboard

Open /dashboard. Fifteen pages, grouped by what you are trying to do.

Group Pages
Watch Overview, Failures, Runs
Understand Quality, Root causes, Drift
Improve Versions, Regression tests, Fixes, Benchmark, Gap analysis
Ship safely Canary, Releases
Govern Audit log, Setup
Page What is in it
Reference Every phase in detail: evals, drift, failure-to-test, root causes, canary, releases, attribution
Architecture How the pieces fit together
Deployment Docker, PostgreSQL and secrets
OpenTelemetry import Send traces from any OpenTelemetry exporter to the Auditor, with no change to your agent's code
Wrong answers Catching "I am not sure" and confident wrong answers when you have no answer key, and what still slips through
Case study Agent Auditor watching its own chat agent, including what did not work

Known limitations