How to test an AI agent when no benchmark exists for your business

What do public AI agent benchmarks cover?

One of the best-known public benchmarks for customer-facing agents is Sierra's τ-bench, whose latest release is τ³-bench. It gives the agent a written policy, tools that change a real state (accounts, bookings, phone lines) and a simulated customer who needs help, so a good score means the agent did the right things, not just said them.

It covers three businesses: an airline, a retail store and a telecom company.

If you run insurance claims, IT access requests, card disputes or a field service desk, it tells you very little. Your refund limit isn't in it, and neither is the exception your team makes for customers in the middle of a contract. A score on someone else's rules tells you how a model handles someone else's business.

So if your business isn't one of the three, you have nothing to test against except what you write yourself.

Why not just write the tests by hand?

You can, and many teams start there. It gets hard quickly, for three reasons.

An agent that takes actions needs a world to act in. To test whether it suspends the right line or refunds the right amount, you need accounts, lines, bills and orders that change when the agent calls a tool, and a way to check what changed afterwards. Building that by hand for every rule is slow.

Quick mock-ups are too forgiving. A mock tool that returns a canned "success" gives a careless agent room to pass. The test world needs the same conditions your real systems check.

Hand-written tests can be impossible to pass. The expected answer contradicts the rules, or the right action can't be done with the tools available. The agent fails, and it counts against the agent. We described this and six other traps in Seven ways your AI agent's test results can mislead you.

How do you test an AI agent without a benchmark?

You build the benchmark from your own rules. This is the method in our paper, Compiling Operating Policies into Executable Test Environments for Tool-Calling Agents, published with its code.

It helps to know up front what each step can and can't tell you. A generated test environment earns confidence at three levels:

• Executable: the working copy and its tools run as code. This says the build works, not that it matches your business.
• Solvable: each test comes with a replay proof that a correct sequence of actions passes it in the working copy. This says a pass is possible, not that the test reflects your real policy.
• Validated against your real operation: the tests and their expected outcomes have been checked against how your systems and your team actually work. Only at this level can you say with confidence whether a failure is the agent's or the test's.

Steps 2 and 3 can get you to the first two levels without expert work, once the working copy runs and the replay proofs pass. The third takes your domain expert. In plain terms, the method has four steps.

Step 1: What do you need to describe?

Two things: the written policy your team works to, and the list of tools the agent can use, with their inputs and what each one does.

This is the domain expert's part, and it is the part only you can do. As we wrote in Anatomy of an AI agent, the rules and the definition of a correct result stay yours even when someone else builds the agent. If some of your rules live only in people's heads, this is where they get written down and approved.

Step 2: How do you get a working copy of your systems?

A language model reads the policy and the tool descriptions and writes a working copy of the system the agent operates in. The copy has:

• a model of the state: the customers, accounts, lines, orders or cases your business keeps;
• one function per tool, carrying the tool's documented conditions and effects: what must be true before it can be used, and what changes after it runs;
• checks that say whether the world is healthy;
• a starting world to run from.

Then execution decides what stays. A tool is kept only if its code parses, matches the inputs declared for it and runs against the starting world. Anything that doesn't run is not kept. That is the first level: executable.

Two things make this different from asking a model to play your system during the test. First, the model is used once, at build time. After that the working copy is ordinary code: the same call on the same state gives the same answer every time. Second, each tool carries the conditions written in your policy and tool descriptions, so it says no where those documents say no. Where your real systems enforce something the documents never mention, the copy won't know about it until someone adds it. The builder also records, for each tool, the assumptions it had to make, so you can see which behaviours rest on something your policy actually says and which were inferred.

Step 3: How are the tests made, and how do you know each one is fair?

The builder breaks the working world in the ways it can break: an overdue bill, a suspended line, a setting switched off. It records what each break does and searches for the actions that repair it. Then it combines breaks into scenarios, each with a short customer story ("no bars, nothing loads, and I can't send picture messages"), written without naming any tool.

The important part is the proof. A test is released only if a correct sequence of actions, replayed from the broken world one step at a time, brings everything back to healthy, with every step checked along the way. If the replay fails anywhere, the test is discarded.

That is the second level: every released test is solvable in the working copy, so no test is impossible to pass. Nobody writes the expected actions by hand: they are what the working repair actually did.

Read that proof for what it is. It shows a correct path exists in the generated world. It doesn't show the test matches your real policy: a test can be solvable and still rest on a wrong assumption, demand more than your team would, or miss an exception your team allows. When your agent fails a solvable test, you know the test isn't impossible. Whether the agent or the test is wrong is what the third level settles.

Step 4: Run, fix, re-test

You run your agent through the tests, with a simulated customer telling each story. Grading looks at what the agent did in the working copy (which tools it called, and on whose behalf), not at how polite the reply sounded.

When it fails, trace the failure first: look at the conversation and the rule the test checks, with your domain expert, and decide whether the agent broke the rule or the test expected the wrong thing. Fix the agent, the instructions or the test accordingly, and run again. This tracing is how a domain moves to the third level. Run each test more than once: an agent talking to a simulated customer does not take the same path every time, and one run is one roll of the dice.

Do the generated tests actually catch anything?

A test suite is only worth having if it tells a careful agent apart from a careless one. So that is what we measured, on the telecom domain of τ³-bench.

The working copy built from the telecom policy and tool list implements all 43 of the domain's tools. From it, the builder produced a suite of 24 scenarios. We ran each one five times with two versions of the same agent: GPT-4.1 with the full telecom policy, and GPT-4.1 with a single sentence telling it to help the customer with the tools available. Same model, same tools, same simulated customer. Only the instructions differed.

The agent given the policy completed 73.8% of the graded actions. The agent given one sentence completed 43.8%. That is a gap of 30.0 points, from one run of the suite. A graded action counts as completed when the agent made the call the test required (or had the customer make it, where the customer's phone is involved). Most graded actions check that the right call was made, not the values passed with it, so read the figure as "did the right things", not "did everything exactly right".

That gap is the point: the tests respond to whether the agent knows the rules. A suite both agents pass equally would tell you nothing.

What about businesses other than telecom?

The builder takes any written policy and tool list as input, so it doesn't need your business to appear in a public benchmark. What it has been shown to do is a different matter, and the three levels keep the two apart.

Telecom is the only domain we have checked end to end. There, the benchmark's own hand-built telecom system and tasks stand in for the real operation: we compared the generated tests with them, on top of the working copy, the replay proofs and a measured result. In our paper the builder also produced working copies for the airline and retail domains, but we did not carry them further, so they reached the first level only. Every new domain starts as a first draft: executable and solvable once built, and validated only after its runs have been traced back to either the agent or the test.

A first draft is useful from day one. Generating the working copy forces every rule to become specific enough to run. Where your policy is vague ("handle large refunds carefully"), the builder has to make an assumption, and it writes that assumption down. That list is often the most valuable thing a domain expert reads in the first week: it shows exactly where the rule book needs a decision.

Where should you start?

Start with what you already have: the written policy, even if it's spread across several documents, and the list of tools your agent can call. From there you can generate a working copy, read the assumptions it made, generate tests and run your agent through them. That is AI agent testing built on your business, not someone else's.

If your domain has no benchmark, we run custom-domain assessments. Send us the policy and tool list for one workflow, and we'll build the working copy and tests, show you which level of confidence each part has reached, and hand you the list of assumptions your expert needs to decide. Write to hello@constolve.com, or run a quick audit of your agent at app.constolve.com.

Questions people ask

Can you test an AI agent if there's no benchmark for your industry?

Yes. You build the test environment from your own written policy and tool list: a working copy of your systems, plus tests generated from it. Public benchmarks such as τ³-bench cover only an airline, a retail store and a telecom company.

What is a synthetic test environment for an AI agent?

A working copy of the systems the agent acts on: the state your business keeps, one function per tool that carries the conditions in your policy and tool descriptions, and a starting world. It runs as ordinary code, so the same action always gives the same result.

How do you know a generated test is fair to the agent?

Each test comes with proof: a correct sequence of actions, replayed from the broken starting state, brings everything back to healthy. Tests without that proof are discarded, so no test is impossible to pass. Whether a test matches your real policy is a separate check, done with your domain expert.

Does this work for domains other than telecom?

The builder takes any written policy as input. Telecom is the only domain we have checked end to end; a new domain is a first draft until its runs have been traced against the real operation. It reaches executable and solvable only once its working copy runs and its tests come with replay proofs.

What does the domain expert need to provide?

The written policy, the list of tools the agent can use, and approval of the rules and of what counts as a correct result. No code.