Breaking points
Each row is one capability, scored against a deterministic simulation that knows the right answer. Where a capability has a difficulty dial, the dial is turned up until the model stops passing, and the point where it stopped is reported in the task’s own units.
Verdict per model
- Gemma 4 31B QATBreaks (7) step-by-step execution with values to carry (11-call plans, 3 values to carry) · recovering from a failed call (12-call plans, 4 values to carry) · repairing stacked bugs (1 bugs) · fixing a bug across files (cause 1 calls away) · the same program over more files (8 files) · a code-graph tool · carried valuesSlips (10) belief tracking · misleading stories · long plans with look-alike tool names (16-call plans) · trap-laden catalogs (4 of 5 traps active) · spotting impossible tasks · repairing a bug (0 hidden checks) · belief against reality · exact ids against natural names · cause over symptom · consistencyHolds on 12 of 29 capabilities tested.
- Gemma 3 27BBreaks (3) long plans with look-alike tool names (17-call plans) · trap-laden catalogs (4 of 5 traps active) · spotting impossible tasksSlips (5) tracking where things are · belief tracking · misleading stories · step-by-step execution with look-alike tool names (11-call plans) · semantic traps in code (5 of 6 traps active)Holds on 6 of 14 capabilities tested.
- Nemotron 3 Nano 30BBreaks (2) long plans with look-alike tool names (13-call plans) · step-by-step execution with look-alike tool names (11-call plans)Slips (5) belief tracking · misleading stories · trap-laden catalogs (4 of 5 traps active) · spotting impossible tasks · recovering from a failed call (11-call plans, 2 values to carry)Holds on 4 of 11 capabilities tested.
- Nemotron 3 Ultra 550BSlips (2) long plans with look-alike tool names (25-call plans) · spotting impossible tasksHolds on 6 of 8 capabilities tested.
- Nemotron 3 Super 120BBreaks (1) long plans with look-alike tool names (24-call plans)Holds on 11 of 12 capabilities tested.
- gpt-oss-120bHolds on all 4 capabilities tested.
BREAKSa level fell to 30% pass or below and stayed low (at most 40% from there up)DEGRADESa level fell to 75% or below; for single-rate rows, the 95% interval sits below 95%HOLDSneither, at every level testedNOT TESTEDno evidence yet
Strip cells, pass rate per level:under 50%50-75%75-90%90-97%97%+no scored trialsfaded cell: fewer than 5 scored trials▲ boundary levelgrade A strong · B moderate · C thin
I · Narrative
HOLDS
Tracks where things actually are100% correct (30/30)Tested up to chain length 30, distractor events 2
grade B moderateHOLDS
Recovers the true order of events100% correct (80/80)Tested up to chain length 20, scramble 1.0, distractor events 30
grade B moderateHOLDS
Tracks what a person believes from what they saw99% correct (181/183)Tested up to chain length 25, scramble 1.0, pronoun pressure 8, distractor events 40, witness cutoff 0.9
grade A strongHOLDS
Belief tracking in deliberately misleading stories99% correct (127/128)Tested up to chain length 10, scramble 1.0, pronoun pressure 5, distractor events 20, witness cutoff 0.9
grade A strongII · Tool planning
DEGRADES
Plans long tool sequences when tool names look alikeDegrades at 25-call plans: 4/7 pass (was 7/7 at 21-call plans)Boundary 25-call plans · held through 21-call plans
grade A strong7-call plans31-call plans
DEGRADES
Notices when a task is impossibleDeclines 5 of 8 impossible tasks; writes a plan for the other 3
grade C thinHOLDS
Plans long tool sequences (clear tool names)Holds through 12-call plans: 28/28 pass across every tested levelHeld through 12-call plans
grade B moderate7-call plans12-call plans
HOLDS
Plans around traps (poisoned tools, misleading descriptions, zero slack)Holds through 5 of 5 traps active: 51/56 pass across every tested levelHeld through 5 of 5 traps active
grade B moderate0 of 5 traps active5 of 5 traps active
NOT TESTED
Picks the right tool from a large catalogNot tested yet
III · Interactive execution
NOT TESTED
Executes a plan step by step, carrying values between callsNot tested yet
NOT TESTED
Executes step by step when tool names look alike (sees each error)Too few trials to call
NOT TESTED
Recovers when a tool call fails mid-plan (transient error, retry works)Not tested yet
NOT TESTED
Executes a plan step by step, carrying values between calls (protocol v3)Not tested yet
NOT TESTED
Recovers when a tool call fails mid-plan (protocol v3)Not tested yet
IV · Code tracing
NOT TESTED
Traces a program when variable names look alikeNot tested yet
NOT TESTED
Traces a program through semantic traps (aliasing, closures, off-by-one, modulo, defaults)Not tested yet
NOT TESTED
Stays correct on small rewrites and adversarial programsNot tested yet
V · Code repair
NOT TESTED
Fixes a one-bug function when shown a failing exampleNot tested yet
NOT TESTED
Fixes stacked bugs without breaking anything elseNot tested yet
V · Code writing
NOT TESTED
Writes a function from a description and examples, edge cases includedNot tested yet
VI · Repository
NOT TESTED
Finds and fixes a bug whose cause is in another fileNot tested yet
NOT TESTED
Calls the real functions when other files hold look-alike namesNot tested yet
NOT TESTED
Fixes the function that runs, not its near-copy elsewhereNot tested yet
NOT TESTED
Fixes a cross-file bug within a tool-call budgetNot tested yet
NOT TESTED
Fixes the same bug as its program is split over more filesNot tested yet
NOT TESTED
Fixes a cross-file bug with a code-graph toolNot tested yet
Graph-on arm of the paired VI.graph_value campaign; the paired comparison with the graph-off arm (<run_id>/graph_off) is in the campaign result note.
EF · Evidence fidelity
NOT TESTED
Answers from what a person saw, against where things really are (paired, B arm)Not tested yet
NOT TESTED
Copies exact look-alike tool ids (paired, B arm)Not tested yet
NOT TESTED
Copies a carried opaque value instead of inventing one (paired, B arm)Not tested yet
NOT TESTED
Copies a carried opaque value instead of inventing one (paired, B arm, protocol v3)Not tested yet
NOT TESTED
Traces code over a misleading comment (paired, B arm)Not tested yet
NOT TESTED
Fixes a bug's cause in another file (paired, B arm)Not tested yet
Reliability
NOT TESTED
Gets the same task right every time (pass^k)Not tested yet
Rate shown is pass^1 over repeats of the same tasks (X.consistency campaign); pass^k needs per-task grouping and is in results/consistency/<run_id>.json. Earlier hand measurement: Nano 63% single-try vs 17% on all 5 tries.