When the simulator covers for your agent

What is a test simulator for an AI agent?

An AI agent that does real work calls tools: look up a customer, change a plan, issue a credit, open a ticket. Running every test against your live systems is rarely an option: a test could refund real money or change a real customer's account. Some teams use isolated production-like environments or carefully controlled live tests, but most of the testing happens somewhere else.

So teams test against a stand-in: a simulated version of the systems the agent would touch. Sometimes it is a set of mock functions written by an engineer. Sometimes it is a language model told to "act like the billing system" and answer each tool call with something plausible. Either way, the agent talks to the simulator exactly as it would talk to production, and the test checks what happened.

That makes the simulator part of the test. If it behaves differently from your real systems, the test is measuring something other than the agent you will deploy. (If you want the bigger picture of where tools sit in an agent, see Anatomy of an AI agent.)

How can a simulator hide an agent's mistakes?

Here is the kind of failure we mean, as an illustration.

A test asks a support agent to help a customer with their mobile line. The agent needs to find the account first, so it calls the lookup tool with a phone number. The number it passes doesn't belong to any customer.

A real customer system answers "no customer found". The agent would have to notice, apologise and ask the customer to check the number.

A forgiving simulator answers with a customer. A name, an account, a plan, a line status. All plausible, all invented on the spot for a number that belongs to nobody. The agent carries on with the invented account, runs its troubleshooting steps, and the conversation ends politely. The test passes.

Nothing in a run like that looks broken: no error, no crash, no odd score. You only see it if you read the tool calls and ask: would our real system have answered that?

Why do forgiving tools make scores go up?

In a real operation, much of what stops an agent making a mess is done by your systems, not by the agent's good sense. The billing system rejects a payment against a bill that is already paid. The order system refuses to cancel an order that has shipped. The account system won't find a customer who doesn't exist.

A test can catch a mistake in two ways: something refuses it during the conversation, or a grader checks the end state afterwards and finds it wrong. When grading relies on tool calls succeeding, the refusals are doing most of the work. A forgiving simulator removes them, and each missing refusal is a chance for a false pass. Three kinds stand out.

Wrong inputs go unnoticed. A misread id, a wrong account or a made-up order number returns a sensible-looking answer, so the agent never has to recover. The skill you most want to test, noticing that something is off, is never exercised.

Rule-breaking actions succeed. If the simulator lets the agent resume a line whose contract has ended, or credit more than the limit, only a grader that checks the end state independently, for example by comparing the final account with what the policy allows, will catch the break. A grader that asks whether the right tools ran without error will not.

Missing steps don't matter. If the agent skips the identity check and goes straight to "change plan", a forgiving simulator just changes the plan. In production that call would fail, or worse, it would succeed on the wrong account.

Any of these can turn a failure into a pass unless an independent end-state check catches it. So a forgiving simulator doesn't just add noise to your scores. Its errors lean one way: up. And a simulator built on a language model can be forgiving by default, because answering helpfully is what the model was trained to do.

What checks must a simulator pass before you trust it?

You can test the simulator on its own, without the agent. Call each tool directly with inputs you have chosen, and compare what comes back with what your real system would do. Where the real system can't safely be probed, compare against its documented behaviour or a controlled test environment. It tells you whether your agent's scores mean anything.

1. Does it say no to bad inputs?

Send each tool a value that cannot be right: a phone number with too few digits, a date that doesn't exist, a negative amount, an empty customer id. A real system rejects these. The simulator should too, with an error the agent can see.

If the simulator answers every malformed call with a normal-looking result, it will hide every input mistake your agent makes.

2. Does it refuse what the real system refuses?

Make a list of the actions your real system blocks, and try each one. Refund an order twice. Pay a bill that is already paid. Resume a line that is suspended for a reason the system won't override. Add more data than the plan allows.

These refusals often encode your business rules. If the simulator allows them, a test can't tell an agent that follows your rules from one that breaks them, unless some other check catches it.

3. Does it keep state across calls?

Change something, then read it back. Add 1 GB of data, then check the balance. Change the plan, then look up the account. Cancel the order, then try to ship it.

A simulator that answers each call fresh, with no memory of the last one, lets an agent "fix" a problem without anything changing, or report a change that never happened. Tests that check the end state of your systems, not just the agent's reply, need a simulator that has an end state.

4. Are ids issued by one tool accepted by the next?

Create something and use it. Open a ticket, then add a note to that ticket number. Look up a customer, then fetch their bills with the account id the lookup returned.

If the ids don't line up, the agent fails tests for reasons that have nothing to do with it: the opposite problem, and just as misleading.

5. Are wrong ids refused?

Now take an id that was never issued, or one that belongs to a different customer, and use it. Fetch bills for an account that doesn't exist. Add a note to ticket 999999. Look up a phone number nobody has.

The real system says "not found" or "not yours". The simulator must say the same. This is the check that catches the example above, and it is an easy one to skip.

What does a test world that says no look like?

The simplest fix is to make the simulator code, not improvisation. Each tool is a function that keeps real state and checks its documented preconditions before it does anything. If the precondition fails, the call is refused. Run the same call on the same state twice and you get the same answer.

That is how we build test worlds at Constolve. We build the working copy of a system from its written policy and tool descriptions, and each tool enforces the preconditions those documents state. Every test we generate comes with a recorded sequence of correct steps that solves it inside that world, so no test is impossible to pass inside it. And because the world is code, the five checks above can be run as ordinary unit tests, every time the world changes.

A deterministic world is not automatically a faithful one. If a rule is written into it wrongly, it will be wrong the same way every time. That is why the probes above still matter: they compare the copy with the real thing.

How does this fit with the rest of your tests?

The simulator is one of several places an AI agent testing setup can go quietly wrong. The simulated customer, the number of repeat runs, what the checks actually look at and how the judge is briefed all matter too. We covered those in Seven ways your AI agent's test results can mislead you.

The aim is the same in every case. You want a failing agent to fail. To get that, the world your agent is tested in has to say no when the real one would, and you have to check that it does.

Run the five checks on your own test setup: one bad input, one action your real system refuses, one change read back, one id carried between tools and one id that was never issued, each called directly without the agent. If you would rather we ran them for you, ask us at app.constolve.com.

Questions people ask

What is an AI agent test simulator?

A stand-in for the systems an agent calls, such as billing, orders or accounts, so the agent can be tested without touching real data. It can be hand-written mock code or a language model that improvises tool responses.

Why is a forgiving simulator a problem?

Because it removes the refusals your real systems make. Wrong ids, rule-breaking actions and skipped steps all succeed, so failures can turn into passes and scores go up for the wrong reasons, especially when grading relies on tool calls succeeding.

How do I test the simulator itself?

Call each tool directly, without the agent, and compare the answers with your real system. Check that it rejects bad inputs, refuses what the real system refuses, keeps state across calls, accepts ids it issued and rejects ids it didn't.

Can a language model act as the simulator?

It can produce realistic responses, but it tends to answer helpfully, including when the real system would refuse. If you use one, run the five checks above on it before trusting the scores it produces.

Is a simulator written as code always accurate?

No. Code makes it consistent, not correct: a rule written wrongly is repeated faithfully. Compare it with the real system using the same probes, and re-run them whenever either one changes.