See how Constolve tests an AI agent before it reaches customers.

Constolve turns an agent's policy, tools, and expected behaviour into repeatable sandbox tests. It shows what failed, keeps the evidence, and turns confirmed failures into regression tests.

Walk through a real example Browse research evidence

The workflow

How Constolve evaluates an agent

Six steps, from the rules an agent runs under to a fix that is only promoted once it is proven.

Step 1
Bring the agent policy and tool definitions
The rules it runs under, and every tool it is allowed to call.
Step 2
Declare the behaviour that matters
The specific things it must and must not do, written down before anything runs.
Step 3
Generate realistic customer situations
Each one designed to put those declared behaviours under pressure.
Step 4
Run the agent repeatedly in a controlled sandbox
Repeatedly, because an agent that handles something once can fail the next time.
Step 5
Inspect failures and lock confirmed ones as regression tests
A confirmed failure becomes a test that runs against every future version.
Step 6
Test a proposed fix against the same evidence
A change is promoted only if it fixes the failure without breaking what already worked.

Evidence

The kinds of evidence we hold

Not one number, but five different kinds of proof. We check the machinery before trusting it, generate coverage far beyond what a person would write, rerun everything to expose instability, refuse fixes that break what worked, and have the accounting audited by someone who did not produce it. Each block below is a separate measurement with its own source.

Pipeline validation

The simulated customer was checked before it was used

An agent is tested by talking to a simulated customer. If the simulator invents facts or breaks the conversation rules, the whole result is worthless. So we tested the simulator itself against declared telecom interaction scenarios first.

22 of 24: simulated interactions met every required check
97.1%: of declared behaviour checks passed
100%: of safety checks passed
0: execution errors

This is evidence for this telecom test environment and the configuration we selected. It is not a claim about customer simulation in general. The model used is not published.

Source: project_knowledge/candidate_agents_sweep/results/runtime_user_enactment_grid_v3_unicode_regrade_v1/summary.json · cells[] where clearance is true

Pipeline validation

Reading the policy correctly was tested before it was relied on

Turning a written policy into checkable requirements is the step everything else rests on. A reader that accepts a requirement the policy never stated would quietly invent the standard the agent is judged against.

0: false acceptances, in each of five repeats
170: labelled requirements in the development set
5: repeats, independently audited

On the locked telecom development set, the policy interpreter was tested against labelled requirements before it was used downstream. This is a development measurement, not a claim of universal policy understanding, and not a certification.

Source: project_knowledge/candidate_agents_sweep/results/final_sweep_report_v1/summary.json · role_findings.semantic_judge_composite

Coverage and repeatability

From one declared policy to 596 test situations

A policy of a few pages becomes a suite far larger than anyone would write by hand. Breadth is the point: the situations a person would not think to try are the ones that find problems.

596: generated telecom test situations
1,192: total attempts across two runs of the suite
0: execution errors, and none hit the turn limit

332situations came out differently on the second run

That is not a success rate and it is not a failure rate. It is the finding: more than half the suite was unstable between runs. A single run would have reported one of those outcomes and hidden the other, which is why we repeat runs and lock confirmed failures as regression tests rather than trusting one pass.

Part of this movement comes from the simulated customer, which runs with variation rather than a fixed seed. The disagreement is therefore not purely the agent's instability.

These were two independent runs of the same generated suite, not one native two-repeat command. All grading here is structural; no model judged it.

Source: project_knowledge/candidate_agents_sweep/results/parallel_telecom_generated_exhaustive_v1/manifest_accounting_failure_analysis.json · denominator, repeat_reconciliation.cases_with_solved_or_score_disagreement

Promotion gate

A fix is not promoted just because the score went up

The same safety fix was tried on two pinned deployments. On the first it cleared. On the second it raised the aggregate score and fixed both safety cases — and was still refused, because it broke two things that had worked.

16 → 18 of 18: on the deployment where the fix cleared
0 → 2 of 2: safety cases fixed on that deployment
13 → 16 of 18: on the second deployment, and still refused
2: new failures that blocked promotion

These are two pinned deployments of one agent framework, on an 18-task fixture. It shows how promotion is gated, not that any fix generalises. The refused transfer is recorded as refused; it is not reported as a partial success.

Source: project_knowledge/candidate_agents_sweep/results/deep_optimization_openhands_special_file_transfer_v1/summary.json · clearance, comparison; and final_sweep_report_v1/summary.json deep_optimization

Accounting and independent audit

The record was reconciled, and independently audited

Evidence is only worth what its accounting is worth. The sweep's own numbers were checked by someone who did not produce them.

35: public agent systems and benchmarks with repeated evidence
234: attempts on one shared nine-task fixture
203 / 31: passes and failures on that fixture
0: rows silently dropped
16 of 16: independent audit checks passed, with no findings

The audit covers the research report's internal accounting and reconciliation. It is not a certification of any product, model or benchmark.

Source: project_knowledge/candidate_agents_sweep/results/final_sweep_report_v1_audit/review.json · checks[], decision, findings; and final_sweep_report_v1/summary.json coverage, comparable_common_fixture

A worked example

Controlled policy comparison, separate from the public benchmark

We took a telecom support agent and the rules it is meant to follow. We generated 24 customer situations that test those rules and ran each one five times — 120 runs. Then we removed the rules and ran the same 120 again. The difference is what the rules were worth, and what they still could not prevent.

Constolve controlled demonstration

This is Constolve's own controlled experiment, not a public benchmark score. It uses 24 customer situations, rerun 5 times under each of two conditions — 240 attempts. The τ³ Telecom record in the research library below is a different experiment on the same domain: 24 situations rerun twice, 48 attempts, against a public benchmark's own grading. The two sets of numbers are not comparable and are never combined.

What was tested

Agent purpose: Resolve telecom customer-support requests through declared tools.
Inputs: One declared policy document and one tool schema set.
Customer situations tested: 24
Conditions compared: The supplied policy, against a minimal stub policy.
Execution: Each situation was repeated 5 times under each condition — 240 attempts.
Grading: Structural checks against 93 declared behaviours. No model judged the result.

What the policy covered

· Diagnose before handing over: try the troubleshooting steps before escalating to a human.
· Device and network steps: airplane mode, APN reset, mobile data, roaming, data saver, SIM status.
· Billing actions: send a payment request and take a payment when the situation calls for it.
· Escalation: transfer to a human agent only when the situation genuinely requires one.

Policy summary; the source text is not reproduced here.

The suite declares 93 behaviour checks across its 24 situations. Each complete pass through the suite evaluates all 93, so the two conditions at five repeats each — ten passes — evaluated 930 checks across 240 attempts.

Met every required behaviour check

The share of attempts in which every declared behaviour for that situation passed.

With the supplied policy29.2% of 120 attempts under each condition

With a minimal stub policy5.8% of 120 attempts under each condition

Required behaviour checks passed

Across all attempts, the share of individual declared behaviours that passed.

With the supplied policy70.9% of 93 declared checks, across 120 attempts under each condition

With a minimal stub policy36.8% of 93 declared checks, across 120 attempts under each condition

One behaviour, in detail

“Try to diagnose the problem before handing the customer to a human.”

With a stub policy

Missed in 53 of 120 attempts

With the supplied policy

Missed in 26 of 120 attempts

The supplied policy roughly halved how often this behaviour was missed. It did not eliminate it — the agent still handed over early in about one attempt in five. That is only visible because each situation ran 120 times rather than once.

The agent had to	Missed, stub	Missed, policy
Diagnose before handing the customer over	53 of 120	26 of 120
Toggle airplane mode when the steps call for it	31 of 120	not in the top misses
Reset the network settings	29 of 120	not in the top misses
Toggle mobile data	25 of 120	21 of 120
Top up data when asked	19 of 120	20 of 120

What was held constant

Both conditions used the same 24 situations, the same tools, and the same structural grading. Replacing the policy with a stub was the only intended change.

What this means, and does not

In this controlled environment, the supplied policy was associated with more attempts meeting every declared requirement. This does not certify the policy outside these tests.

Research evidence

Research runs in other agent environments

These are separate public benchmark evaluations. They use different situations and different grading, so they are shown as individual records rather than one leaderboard.

Customer support agent (telecom)

Evaluated on τ³-bench, telecom domainsource ↗

What the agent had to do: Handle a phone-support call: check the SIM, reset network settings, toggle data or roaming, take a payment, and escalate to a human only when the problem genuinely needs one.

Customer situations tested: 24 customer situations
Times each was rerun: each rerun twice
Total attempts: 48 attempts

Met every published requirement in 56.3% of attempts

13 of 24 situations in the first pass, 14 of 24 in the second.

What this tells us:: Repeating matters: 5 of the 24 situations came out differently on the second pass.
Limits:: One agent model, one customer simulator, one environment. Not a score for τ³-bench or for any model.

Customer support agent (airline)

Evaluated on τ³-bench, airline domainsource ↗

What the agent had to do: Handle an airline request end to end: find the booking, apply the fare rules, and make the change or refund the rules actually allow.

Customer situations tested: 50 customer situations
Times each was rerun: each rerun twice
Total attempts: 100 attempts

Passed the benchmark's published check in 71 of 100 attempts

The denominator counts every attempt, not every situation.

What this tells us:: The agent handled most airline requests correctly under the benchmark's own grading.
Limits:: One declared environment. Not a general result for the benchmark or for any model.

Customer support agent (retail)

Evaluated on τ³-bench, retail domainsource ↗

What the agent had to do: Handle a retail service request: look up the order, apply the return and exchange rules, and complete or refuse the request accordingly.

Customer situations tested: 114 customer situations
Times each was rerun: each rerun twice
Total attempts: 228 attempts

Passed the benchmark's published check in 185 of 228 attempts

80 of those decisions were made by a model rather than a structural check, and are counted here rather than hidden.

What this tells us:: Most retail requests were handled correctly, though part of this result depends on a model's judgement.
Limits:: Mixed grading. The 80 model-judged decisions are weaker evidence than the structural ones.

Personal assistant across several apps

Evaluated on AppWorld, development splitsource ↗

What the agent had to do: Complete everyday tasks across several simulated apps — messages, files, shopping, music — by calling their APIs in the right order.

Customer situations tested: 57 development tasks
Times each was rerun: each rerun twice
Total attempts: 114 attempts

Passed 6 of 57 tasks in each pass

AppWorld's own grader, unmodified. This benchmark is hard and this is what the runs recorded.

What this tells us:: Multi-application tool use is substantially harder than single-domain support, on the same method.
Limits:: Only the development split was opened. The 585 protected test tasks were not touched and nothing here describes them.

Patient administration assistant

Evaluated on PatientAgentBenchsource ↗

What the agent had to do: Handle patient-facing admin: book or move an appointment, deal with a prescription request, and route a message to the right place.

Workflow reliability only

No clinical claim is made or authorised from this evidence. The benchmark also scores rubrics it labels clinical safety and triage quality; those figures are deliberately not reported here, because an automated jury score is not a clinical assessment and no medical-correctness claim is authorised. Nothing here is medical advice, a diagnosis-quality measure, or a safety certification.

Customer situations tested: 20 patient-admin situations
Times each was rerun: each rerun twice
Total attempts: 40 attempts

Completed all 20 situations in both passes, with no errors

Task completion 100% in both passes; workflow accuracy 90% then 95%, scored by the benchmark's own automated jury.

What this tells us:: The assistant completed the administrative workflow reliably in this small sample.
Limits:: 20 bundled sample situations, one model. Small sample, single environment.

Your agent has rules too.

Give us those rules and we will show you which ones it misses, how often, and whether a fix actually holds.

Request a pilot How it works

All results are verified within the declared contract and environment · no outcome is guaranteed safe