Breaking points
Each row is one capability, scored against a deterministic simulation that knows the right answer. Where a capability has a difficulty dial, the dial is turned up until the model stops passing, and the point where it stopped is reported in the task’s own units.
Verdict per model
- Gemma 4 31B QATBreaks (7) step-by-step execution with values to carry (11-call plans, 3 values to carry) · recovering from a failed call (12-call plans, 4 values to carry) · repairing stacked bugs (1 bugs) · fixing a bug across files (cause 1 calls away) · the same program over more files (8 files) · a code-graph tool · carried valuesSlips (10) belief tracking · misleading stories · long plans with look-alike tool names (16-call plans) · trap-laden catalogs (4 of 5 traps active) · spotting impossible tasks · repairing a bug (0 hidden checks) · belief against reality · exact ids against natural names · cause over symptom · consistencyHolds on 12 of 29 capabilities tested.
- Gemma 3 27BBreaks (3) long plans with look-alike tool names (17-call plans) · trap-laden catalogs (4 of 5 traps active) · spotting impossible tasksSlips (5) tracking where things are · belief tracking · misleading stories · step-by-step execution with look-alike tool names (11-call plans) · semantic traps in code (5 of 6 traps active)Holds on 6 of 14 capabilities tested.
- Nemotron 3 Nano 30BBreaks (2) long plans with look-alike tool names (13-call plans) · step-by-step execution with look-alike tool names (11-call plans)Slips (5) belief tracking · misleading stories · trap-laden catalogs (4 of 5 traps active) · spotting impossible tasks · recovering from a failed call (11-call plans, 2 values to carry)Holds on 4 of 11 capabilities tested.
- Nemotron 3 Ultra 550BSlips (2) long plans with look-alike tool names (25-call plans) · spotting impossible tasksHolds on 6 of 8 capabilities tested.
- Nemotron 3 Super 120BBreaks (1) long plans with look-alike tool names (24-call plans)Holds on 11 of 12 capabilities tested.
- gpt-oss-120bHolds on all 4 capabilities tested.
BREAKSa level fell to 30% pass or below and stayed low (at most 40% from there up)DEGRADESa level fell to 75% or below; for single-rate rows, the 95% interval sits below 95%HOLDSneither, at every level testedNOT TESTEDno evidence yet
Strip cells, pass rate per level:under 50%50-75%75-90%90-97%97%+no scored trialsfaded cell: fewer than 5 scored trials▲ boundary levelgrade A strong · B moderate · C thin
I · Narrative
DEGRADES
Tracks what a person believes from what they saw90% correct (521/578)Tested up to chain length 25, scramble 1.0, pronoun pressure 8, distractor events 40, witness cutoff 0.9
grade A strongDEGRADES
Belief tracking in deliberately misleading stories88% correct (279/317)Tested up to chain length 10, scramble 1.0, pronoun pressure 5, distractor events 20, witness cutoff 0.9
grade A strongHOLDS
Tracks where things actually are100% correct (90/90)Tested up to chain length 30, distractor events 2
grade B moderateHOLDS
Recovers the true order of events99% correct (156/158)Tested up to chain length 20, scramble 1.0, distractor events 30
grade A strongII · Tool planning
BREAKS
Plans long tool sequences when tool names look alikeBreaks at 13-call plans: 5/21 pass (already only 15/28 at 7-call plans)Boundary 13-call plans
grade B moderate7-call plans14-call plans
DEGRADES
Plans around traps (poisoned tools, misleading descriptions, zero slack)Degrades at 4 of 5 traps active: 13/35 pass (was 34/35 at 0 of 5 traps active)Boundary 4 of 5 traps active · held through 0 of 5 traps active
grade A strong0 of 5 traps active5 of 5 traps active
DEGRADES
Notices when a task is impossibleDeclines 6 of 12 impossible tasks; writes a plan for the other 6
grade C thinHOLDS
Plans long tool sequences (clear tool names)Holds through 13-call plans: 46/49 pass across every tested levelHeld through 13-call plans
grade B moderate7-call plans13-call plans
NOT TESTED
Picks the right tool from a large catalogNot tested yet
III · Interactive execution
BREAKS
Executes step by step when tool names look alike (sees each error)Breaks at 11-call plans: 1/5 pass (was 4/5 at 7-call plans)Boundary 11-call plans · held through 7-call plans
grade C thin7-call plans11-call plans
DEGRADES
Recovers when a tool call fails mid-plan (transient error, retry works)Degrades at 11-call plans, 2 values to carry: 3/5 pass (was 5/5 at 9-call plans, 1 value to carry)Boundary 11-call plans, 2 values to carry · held through 9-call plans, 1 value to carry
grade C thin7-call plans, no values to carry14-call plans, 7 values to carry
HOLDS
Executes a plan step by step, carrying values between callsHolds through 19-call plans, 16 values to carry: 43/45 pass across every tested levelHeld through 19-call plans, 16 values to carry
grade B moderate7-call plans, no values to carry19-call plans, 16 values to carry
NOT TESTED
Executes a plan step by step, carrying values between calls (protocol v3)Not tested yet
NOT TESTED
Recovers when a tool call fails mid-plan (protocol v3)Not tested yet
IV · Code tracing
NOT TESTED
Traces a program when variable names look alikeNot tested yet
NOT TESTED
Traces a program through semantic traps (aliasing, closures, off-by-one, modulo, defaults)Not tested yet
NOT TESTED
Stays correct on small rewrites and adversarial programsNot tested yet
V · Code repair
NOT TESTED
Fixes a one-bug function when shown a failing exampleNot tested yet
NOT TESTED
Fixes stacked bugs without breaking anything elseNot tested yet
V · Code writing
NOT TESTED
Writes a function from a description and examples, edge cases includedNot tested yet
VI · Repository
NOT TESTED
Finds and fixes a bug whose cause is in another fileNot tested yet
NOT TESTED
Calls the real functions when other files hold look-alike namesNot tested yet
NOT TESTED
Fixes the function that runs, not its near-copy elsewhereNot tested yet
NOT TESTED
Fixes a cross-file bug within a tool-call budgetNot tested yet
NOT TESTED
Fixes the same bug as its program is split over more filesNot tested yet
NOT TESTED
Fixes a cross-file bug with a code-graph toolNot tested yet
Graph-on arm of the paired VI.graph_value campaign; the paired comparison with the graph-off arm (<run_id>/graph_off) is in the campaign result note.
EF · Evidence fidelity
NOT TESTED
Answers from what a person saw, against where things really are (paired, B arm)Not tested yet
NOT TESTED
Copies exact look-alike tool ids (paired, B arm)Not tested yet
NOT TESTED
Copies a carried opaque value instead of inventing one (paired, B arm)Not tested yet
NOT TESTED
Copies a carried opaque value instead of inventing one (paired, B arm, protocol v3)Not tested yet
NOT TESTED
Traces code over a misleading comment (paired, B arm)Not tested yet
NOT TESTED
Fixes a bug's cause in another file (paired, B arm)Not tested yet
Reliability
NOT TESTED
Gets the same task right every time (pass^k)Not tested yet
Rate shown is pass^1 over repeats of the same tasks (X.consistency campaign); pass^k needs per-task grouping and is in results/consistency/<run_id>.json. Earlier hand measurement: Nano 63% single-try vs 17% on all 5 tries.