Every team I talk to has shipped an AI agent demo. Almost none of them have shipped it to production. The gap between "it worked when I tried it" and "I trust it with real users" is enormous, and it comes down to one uncomfortable truth: nobody knows how to test these things.
We have decades of tooling for testing code. You write a function, you write a test, pytest tells you green or red. But an AI agent isn't a function — it's a system that plans, calls tools, reasons over results, and produces text. When it fails, it usually fails by looking correct while doing the wrong thing. Your existing test suite has no idea.
So I built AgentProbe — an open-source, framework-agnostic testing platform that tests an agent's behavior, not just its output. This post is about why output testing isn't enough, the architecture I landed on, and the design decisions that made it work across any domain.
The Problem with Testing Outputs
The obvious way to test an AI agent is to check its final answer. Did it say the right thing? Tools like DeepEval do this well — they score relevance, faithfulness, and a dozen other qualities of the text the model produces.
But consider a customer support agent handling "I want a refund for order #12345." It replies:
"I've processed your refund of $79.99. It'll appear in 3-5 business days."
That output is perfect. Relevant, well-formed, grounded in the order data. An output-only evaluator gives it full marks. There's just one problem: the agent never actually called process_refund. It said it processed the refund. It didn't. The customer is now waiting for money that will never arrive.
This is the failure mode that matters most in production, and output testing is structurally blind to it. The text is correct. The behavior is broken. To catch it, you have to look at what the agent did — which tools it called, in what order, and whether its claims match its actions.
That insight is the whole reason AgentProbe exists.
What "Testing Behavior" Actually Means
AgentProbe evaluates an agent across four dimensions, each answering a question that output testing can't.
Quality asks: did the agent do its job correctly? Not just "is the answer good" but "did it call the right tools, in the right sequence, take an efficient path, and avoid fabricating actions it never performed?" The hallucination check here is the star — it cross-references claims against the tool trace. If the output says "refund processed" but process_refund never appears in the call log, that's flagged as fabrication.
Security asks: can the agent be manipulated? It runs a library of adversarial attacks — prompt injection, jailbreaks, PII extraction, system-prompt exfiltration, and tool abuse — and checks whether each one breached the agent's defenses. This dimension became more important after Promptfoo, the main open-source red-teaming tool, was acquired by OpenAI, leaving teams on other model providers without an independent option.
Cost asks: what does this actually cost to run? It profiles per-query token usage, latency, and dollar cost, attributes spend across planning, retrieval, and generation steps, and projects monthly cost at scale. Most teams discover their agent's true cost from the API bill. This surfaces it before deployment.
Regression asks: did my last change break anything? It captures a golden snapshot of the agent's behavior, then diffs against it after every change — flagging quality drops, tool-call changes, and cost increases per test case.
The Architecture Decision That Mattered Most
My first version was hardcoded around customer support. The metrics knew about refunds and orders. The attacks referenced specific order numbers. It worked beautifully for exactly one use case and was useless for everyone else.
The refactor that made AgentProbe actually valuable was replacing all domain knowledge with configurable assertions. Instead of the framework knowing what "correct" means, each test case declares it:
{
"input": "What is the capital of France?",
"expected_tools": ["search_knowledge"],
"expected_contains": ["paris"],
"expected_not_contains": ["error", "sorry"],
"action_claims": {"booked": "book_flight"},
"max_steps": 5
}
The action_claims field is the generalized version of the refund-hallucination check. It says: if the output contains this phrase, the mapped tool must have been called. "Booked your flight" requires book_flight. "Ticket created" requires create_ticket. "Refund processed" requires process_refund. It catches the universal failure mode — an agent claiming an action it never took — in any domain.
To prove the framework was truly generic, I shipped three sample agents in three unrelated domains: customer support, research/QA, and math. The same metrics and the same attacks score all of them. When all three pass at 100% for correct implementations and every injected bug gets caught, you know the abstraction is right.
Staying Framework-Agnostic
The other design constraint was that AgentProbe must work with any agent, regardless of how it was built — LangChain, CrewAI, the OpenAI Agents SDK, or a custom stack. The way to achieve that is to never depend on internals. AgentProbe talks to agents over HTTP.
This sounds simple until you realize every agent has a different request and response shape. One expects {"input": ...}, another {"message": ...}. One returns {"output": ...}, another {"reply": ...}, another buries the answer at choices.0.message.content. A rigid connector would force everyone to write an adapter.
Instead, the connector adapts through field mapping. You tell it where to put the query and where to read the answer:
agent = make_connector({
"type": "http",
"url": "http://localhost:5000/api/chat",
"input_field": "message",
"output_field": "reply",
"extra_body": {"provider": "local", "model": "llama3"},
})
Dot-paths handle nested responses, extra_body carries any extra parameters the endpoint needs, and non-JSON responses fall back to treating the whole body as output. The agent doesn't change at all — AgentProbe meets it where it is.
The Decision to Run Offline
Most evaluation tools default to using an LLM as a judge, which means every test run costs money and requires an API key. I made the opposite choice: AgentProbe's metrics run heuristically by default, with no external API.
This has real consequences. Tests are free to run, so you can run them on every commit without watching a meter. They're deterministic, so a passing test today passes tomorrow for the same reason. They work in air-gapped environments where you can't call out to a model provider. And they're fast — no network round-trip per assertion.
The heuristics aren't as nuanced as an LLM judge for open-ended quality, but for the things that matter most in agent testing — did it call the right tool, did it claim an action it didn't take, did it leak the system prompt — deterministic checks are actually better. You want those answers to be exact, not probabilistic. LLM-as-judge remains available as an optional upgrade for the fuzzy cases, but it's not a dependency.
Three Ways to Run It
AgentProbe meets developers in three places. There's a web dashboard for interactive exploration — pick an agent, hit run, drill into every failed case and blocked attack. There's a CLI for the terminal, styled after pytest: agentprobe test, agentprobe fuzz, agentprobe cost. And there's a pytest integration for CI/CD, so a pull request fails automatically if the agent regresses, gets less safe, or blows its cost budget:
def test_my_agent():
agent = make_connector({"type": "http", "url": "http://localhost:8000/agent"})
assert run_quality_suite(agent, my_cases)["pass_rate"] >= 0.95
assert run_security_suite(agent)["block_rate"] >= 0.95
That last one is where AgentProbe earns its keep. Testing infrastructure that lives only in a dashboard gets used once and forgotten. Testing that runs in CI becomes a gate nobody can ship around.
Why This Matters Now
The agent gold rush is producing thousands of demos and very few production systems. The bottleneck isn't model capability — frontier models are more than good enough. The bottleneck is confidence. Teams can't deploy what they can't verify, and they can't verify agents with tools built for testing deterministic code or scoring static text.
Evaluation and observability tooling has become one of the largest line items in AI engineering budgets for exactly this reason. The teams that figure out how to test agents will be the ones that ship them. The rest will keep building demos.
AgentProbe is my attempt at the missing piece: a way to point at any agent and ask the questions that actually predict production behavior. Did it do the right thing? Can it be attacked? What does it cost? Did my change break it? Answer those with confidence, and the gap between demo and production starts to close.
AgentProbe is open-source and MIT-licensed. It runs offline with no API key, works with any agent framework over HTTP, and ships with a web dashboard, a CLI, and pytest integration. Built with Python, Flask, and a lot of opinions about what testing should feel like.
No comments yet
Be the first reader to add one.