Seven ways your AI agent's test results can mislead you

Every team building an AI agent ends up trusting a number: a pass rate, a test score, an accuracy figure. That number is only as good as the test behind it, and tests mislead in ways that produce perfectly normal-looking numbers.

Here are seven we've run into while testing customer-support agents against their own operating rules. For each: what goes wrong, an example, and how to check your own setup.

1. The simulated customer isn't behaving like a customer.

To test a support agent you need someone to play the customer, usually another LLM. If that simulated customer misbehaves, every result is off, and nothing tells you.

Example: we found our simulated customer had been fed the conversation with the roles swapped: it read the agent's lines as its own. Every conversation still ran, and every score still came out. After we fixed it, the comparison between agents held, but every absolute score moved by about five points.

Check: read a few conversations from the customer's side. Does it sound like a customer with a problem?

2. One run is one roll of the dice.

An LLM-based test has randomness in it. The simulated customer is an LLM, usually set to vary its wording so it sounds natural. Small differences early in a conversation send the agent down different paths: a customer who mentions they're abroad in their first message gets a roaming check, and one who mentions it five messages later may never get one. So the same agent on the same tests gives a different score each run, and a single run can land high or low by chance.

Example: we ran the same telecom agent on the same 24 tests five times, comparing it with and without its rules. The measured benefit of the rules was 35, 22, 35, 32 and 26 points: from 22 to 35 depending on the run. Anyone who ran it once could have reported either number.

Check: run every test several times and report the range, not one number. For reliability, also count how many tests the agent passes every time. That's what a customer actually experiences.

3. A test that every agent passes tells you nothing.

A good test separates an agent that follows the rules from one that doesn't. Many tests don't: a careless agent passes them just as easily as a careful one.

Example: say a telecom company's rules cap a customer's extra mobile data at 2 GB. A test where the customer asks for 1 GB checks almost nothing: an agent that has never seen the rules adds 1 GB too, and passes. The test that matters is the customer who asks for 5 GB. The agent that knows the rule offers 2 GB and explains why. The agent that doesn't simply adds 5 GB. Most test suites are full of the first kind of test, which only inflates the score.

Check: run your tests on a deliberately weakened agent, such as one stripped of its instructions. Any test it still passes isn't testing your rules.

4. Some failures are the test's fault, not the agent's.

A test can be impossible to pass: the expected answer contradicts the rules, the starting setup is broken, or the "right" action can't be done with the tools available. The agent fails it, and it counts against the agent.

Example: a test expects the agent to restore a customer's phone line, but the rules say a line whose contract has ended must not be restored. The agent that follows the rules fails. The agent that breaks them passes.

Check: before a test can count against an agent, prove a correct solution passes it. And when a test fails, ask whether it's the agent or the test.

5. "Did the right thing" often means "pressed the right button".

Many graders check that the agent used the right tool, not what it put into it.

Example: a customer asks for 1 GB of extra data. The grader checks that the agent called "add data". The agent adds 5 GB, breaking the 2 GB limit, and the test passes. Or a test checks that a payment request was sent, not that it was for the right bill. When we looked at one test suite, most of its checks looked only at the action, never at the details.

Check: for each check, ask: if the agent did the right action with the wrong amount, account or date, would this still pass?

6. An LLM judge grades by impression unless you give it the rule.

Using an LLM as a judge to grade an agent is common. Without the specific rule in front of it, the judge grades on general helpfulness, and a polite rule-break looks like good service.

Example: an agent refunds a customer above its approval limit, politely and quickly. An LLM judge asked "did the agent handle this well?" says yes. The same judge, shown the rule "refunds above $100 need a manager's approval", says no.

Check: give the judge the exact rule each test is about. Better still, where the rule can be checked from data (amount, approval flag, account status), check it with code, not a judge.

7. An LLM reviewer can approve what it hasn't read.

Teams increasingly use an LLM as a reviewer for test code and results. It can produce confident, detailed approvals based on things that don't exist.

Example: we asked an LLM reviewer to audit a piece of test code. It approved it and quoted three lines from the code as evidence. None of the three lines exist in the file. The approval read as thorough, and it was invented.

Check: treat an LLM review as a lead, not a verdict. Check its quotes and line references against the actual files, especially when it says everything is fine.

What these have in common

None of them raises an error. Each produces a clean, believable number. And each is caught the same way: go back to the raw material (the conversations, the tests, the checks) and ask whether the number measures what you think it measures.

Before trusting your agent's score, ask:

• Does the simulated customer behave like a real one?
• Have you run the tests enough times to see the range, not just one number?
• Does each test separate an agent that follows the rules from one that doesn't?
• Is every test passable by a correct answer?
• Do the checks look at the details, or only the action?
• Does the LLM judge see the rule it's grading?
• Has someone verified what the LLM reviewer quoted?

A test score is only as good as the test behind it. And tests fail quietly.

Which of these have you run into? I'd like to hear what else has fooled your team.