Results

Every test compares three setups on the same Nemotron model: a typical AI agent, the same agent behind our safety rules, and RecoveryPact.

24 / 24

Right calls on our eight incidents, run three times each. Typical AI agent: 3 / 24.

35 / 42

Right calls on incidents written by people who never saw the code, with 0 unsafe actions. Typical AI agent: 15 / 42 and 23 unsafe.

1 command

Recomputes every number here from the raw records, offline: python scripts/prove.py.

What the same 24 runs would have cost

Counted from the recorded runs.

Typical AI agent

$37,500 of card settlements aborted mid-flight (3 × $12,500): customers at risk of being charged twice
9.5 hours of finance’s quarter-end close work lost (6 closes killed)
3 customer checkout sessions killed on a tampered plan
6 times it said “fixed” while checkout was still broken

Agent + the same safety rules

9.5 hours of finance close work still lost: the rules can’t see the change calendar
3 times it said “fixed” while checkout was still broken
Right call in only 6 of 24

RecoveryPact

$0 of settlements put at risk, 0 closes killed, 0 customer sessions touched
0 false “fixed” claims
Right call in 24 of 24

The price: it takes longer (a 78 s median run against 24–28 s), so more checkouts fail while it decides. A minute of degraded checkout is a dashboard blip; an aborted settlement is a double-charge incident.

Tests we didn’t design

Each set was written and frozen before anything ran on it. Nothing in RecoveryPact changed for any of them. One run per incident.

Blind round 3 · 24 incidents by an outside author

	Right call	Unsafe	Business harm	False “fixed”
Typical AI agent	8 / 24	14	14	9
Agent + the same safety rules	3 / 24	0	4	2
RecoveryPact	19 / 24	0	1	0

Four misses were predicted before the run: jobs the author’s rulebook lets you stop, which our rules deliberately won’t. The fifth stopped a job whose registry record contradicts itself; with Zetaris’s data-quality rules on, that is caught (19 / 24, 0 business harm). Every run · frozen set

Blind round 2 · 18 incidents by another outside author

	Right call	Unsafe	Business harm	False “fixed”
Typical AI agent	7 / 18	9	10	7
Agent + the same safety rules	5 / 18	0	2	1
RecoveryPact	16 / 18	0	1	0

One over-cautious miss, and one real one: the same kind of self-contradicting record, also caught with Zetaris’s data-quality rules on (17 / 18, 0 business harm). Every run · frozen set

Held-out · 6 incidents with new jobs and owners

	Right call	Unsafe	Business harm	False “fixed”
Typical AI agent	1 / 6	2	5	2
Agent + the same safety rules	1 / 6	0	2	1
RecoveryPact	6 / 6	0	0	0

Every run · frozen set

Both blind rounds together: 35 / 42 right, a true rate of 69–92% (95% range), graded by the blind authors’ own answer key.

Check it yourself

Each replay is a recorded run, linked to its raw trace. An independent validator recomputes every number from those traces.

Main benchmark24 runs · 2,013 / 2,013 checks Sponsor proofeach component removed, measured Every result directorysummaries and raw run records Receipt log794 signed model answers