Testing an agent against the rules it was given
We gave our system only the written policy for the telecom domain of Sierra's τ³-bench and asked it to build tests. It found all 17 faults an agent can repair, and its tests require the same repair tools as 90 of the benchmark's 94 repair tasks. The tests also tell agents apart: an agent given the policy did more of what was required than the same model without it, and the suite caught 185 of 202 deliberately corrupted solutions.
Benchmark: τ³-bench (Sierra), telecom domain. Sierra hand-coded 19 basic phone faults; a program combines them into the 114 official tasks.
- All 17
faults an agent can repair, found from the written policy alone
- 90 of 94
repair tasks whose required repair tools our tests also require
- +30.9 points
more of the required actions completed by an agent given the policy, mean of five runs
- 185 of 202
deliberately broken solutions caught by our tests
Want to run this on your own agent? Request a pilot