How we check our own numbers before we publish them

Why should you care how a number was checked?

If you are choosing an agent, a model or a testing approach, you will be shown numbers. A pass rate. An accuracy figure. "Our rules made the agent far more reliable."

Each of those numbers came out of a chain: someone ran an experiment, a script scored the results, someone copied the score into a slide or a blog post. A mistake anywhere in that chain produces a number that looks exactly like a correct one. Nothing about "73.8%" tells you whether it was computed carefully.

So the useful question isn't "is this number good?" It is "how would I check it?" If the answer is "you can't", the number is a claim, not a result.

We hold our own numbers to that standard. Below is what that means in practice, step by step, with one real result as the example.

What is the result we'll use as the example?

We tested a customer-support agent on telecom troubleshooting, built from the telecom domain of Sierra's public τ³-bench benchmark. The agent, GPT-4.1, was run on 24 test scenarios (19 of them distinct), each repeated 5 times, in two versions: once with the written telecom policy in its instructions, and once with a one-sentence instruction instead.

With the policy, the agent completed 73.8% of the required actions (273 of 370). Without it, 43.8% (162 of 370). That is a gap of +30.0 percentage points, from one run of 240 conversations.

"Actions completed" mostly checks that the agent called the right tool for the situation; many checks don't look at the values it passed. We say that every time we quote the number, because it is part of what the number means.

This result is in our paper, published on 29 October 2026 on Zenodo (DOI 10.5281/zenodo.23741457). Here is how it got there.

Step 1: Why do we save every run?

Every conversation in the run is saved as a file: what the customer said, what the agent said, every tool call it made and what came back. Alongside them we save the run's settings: which model, which temperature, which test cases, how many repeats.

Then we take a fingerprint of the folder. Each file gets a checksum, and the list of checksums is stored with the run. If a single character in any file changes later, the fingerprint no longer matches, and the scoring script refuses to run.

This sounds like bookkeeping, and it is. It is also the foundation for everything else. A number you can't trace to its files can't be checked by anyone, including you.

Step 2: Why must a script recompute every number?

The number on the page is not typed in by hand. It is the output of a script that reads the saved files and computes it.

For the telecom result, the script reads the 240 saved conversations, applies the same deterministic checks to each one (did the agent call this tool, did it avoid that one), and adds them up. No AI model is involved in the scoring, so the script gives the same answer every time it runs on the same files.

Our rule is simple: a number without a script behind it doesn't get published. If we can't regenerate it from the saved files, we don't quote it.

Step 3: Why repeat runs instead of trusting one?

An AI agent test has randomness in it. The simulated customer is a language model and varies its wording; small differences early in a conversation lead the agent down different paths. Run the same agent on the same tests twice and you can get two different scores.

How much does that matter? In the reliability study in the same paper, 29.8% of telecom benchmark tasks changed verdict, pass to fail or fail to pass, across identical runs. A single run can land high or low by chance.

So every test case is repeated. For the telecom result that was five repeats per case per version. We report the pooled result and, in the paper, analyse it at the level of the 19 distinct test scenarios rather than individual conversations: the 95% interval for the gap is 24.7 to 35.4 points, and the version with the policy was ahead on 16 of the 19. We covered why single runs mislead in Seven ways your AI agent's test results can mislead you.

Step 4: Who checks the code and the number?

Whoever wrote the scoring code is the worst person to review it. They know what it is supposed to do, so they read what they meant rather than what is there.

So every scoring script and every number goes through a separate-model review with evidence verification. The reviewer is an AI model from a different provider than the one that wrote the code, given the code, the saved files and the claimed figure, and asked to confirm that the code measures what we say it measures and that the figure matches. We don't take its word on trust either. Its approvals have to quote real lines in real files, and we check each quote against the files.

Step 5: What goes in the facts ledger?

Once a number has passed the first four steps and been signed off, it goes into our facts ledger: a single file that lists every figure we may quote, where its evidence lives, the command that recomputes it, and how to say it.

That last column matters as much as the number. For the telecom result, the ledger says: one run, 24 cases × 5 repeats, GPT-4.1 agent, and "actions completed" mostly checks the tool called. Anyone writing about it, on our website, in a post or in a sales conversation, says it that way.

Before anything is published, every number in it is checked against the ledger. If a figure isn't there, it doesn't go out until it has been through the same steps.

Step 6: How can you re-run our numbers yourself?

The last step is to let anyone repeat the check. We released the code and data behind the paper in a public repository: github.com/vinsing09/constolve-policy-compilation.

It contains the saved conversations of every run behind the telecom results, the fingerprints, the scoring code, and a reproduce/ folder of scripts. The telecom result is one command from the repository root:

python3 -B reproduce/separation.py

It checks the fingerprints, re-scores the 240 saved conversations and prints actions completed of 73.8 with the policy and 43.8 without, a gap of +30.0. It runs with plain Python, needs no extra dependencies and no API key, and never calls a model. If any saved file has been modified, the seal check refuses to score it. bash reproduce/verify_all.sh runs every check in order against the values printed in the paper. The README lists, for each table and figure, which script recomputes it and which files it reads.

This is reproduction, and it is worth being exact about what it shows. Re-scoring the saved files confirms that the published number was computed correctly from the conversations we recorded. It is not the same as independently replicating the experiment: a fresh run of the agent and the simulated customer produces new conversations and a new score, and tests whether the effect holds up. The repeated runs and the scenario-level interval in Step 3 speak to replication; the command settles reproduction exactly.

How can you apply this to your own agent?

You don't need a research team to do this. For any number your team is about to share about an AI agent, ask:

• Are the conversations it came from saved, and would we notice if they changed?
• Does a script compute it from those files, or was it copied by hand?
• Was the test repeated, and do we know how much the score moves between repeats?
• Has someone who didn't write the code checked that it measures what we claim?
• Is there one place that says how this number may be described?

The same questions work on a vendor's numbers. Ask them for the run files and the script. Good AI agent testing produces numbers that survive that request.

The quickest way to see what this looks like is to run the command yourself. Clone the repository, run python3 -B reproduce/separation.py from its root, and compare what it prints with the numbers on this page. If you want the same standard applied to your own agent's results, you can start at app.constolve.com. For the parts of an agent those tests look at, see Anatomy of an AI agent.

Questions people ask

What does it mean for an AI agent result to be reproducible?

That someone else, given the same saved run files and the same scoring script, gets the same number. For results with randomness in them, it also means the run was repeated so the number isn't one lucky draw.

Why not just re-run the agent to check a number?

A new run of a language-model agent gives a new result, not the recorded one, because the agent and simulated customer vary. Re-scoring the saved conversations checks the published number exactly; re-running checks whether the effect holds up.

Can I check Constolve's telecom result myself?

Yes. Clone the public repository and run python3 -B reproduce/separation.py from its root. It re-scores the 240 saved conversations without calling a model and needs no API key.

What is a facts ledger?

A single file listing every number an organisation may quote, with its evidence, the command that recomputes it and the exact way it should be described. Anything published is checked against it first.

Why have a separate model review the scoring code?

The author of the code tends to read what they intended, not what they wrote. A reviewer that didn't write it, and whose claims are themselves checked against the files, catches mistakes the author misses.