Breaking points

Each row is one capability, scored against a deterministic simulation that knows the right answer. Where a capability has a difficulty dial, the dial is turned up until the model stops passing, and the point where it stopped is reported in the task’s own units.

Verdict per model6 models

Gemma 4 31B QATBreaks (7) step-by-step execution with values to carry (11-call plans, 3 values to carry) · recovering from a failed call (12-call plans, 4 values to carry) · repairing stacked bugs (1 bugs) · fixing a bug across files (cause 1 calls away) · the same program over more files (8 files) · a code-graph tool · carried valuesSlips (10) belief tracking · misleading stories · long plans with look-alike tool names (16-call plans) · trap-laden catalogs (4 of 5 traps active) · spotting impossible tasks · repairing a bug (0 hidden checks) · belief against reality · exact ids against natural names · cause over symptom · consistencyHolds on 12 of 29 capabilities tested.
Gemma 3 27BBreaks (3) long plans with look-alike tool names (17-call plans) · trap-laden catalogs (4 of 5 traps active) · spotting impossible tasksSlips (5) tracking where things are · belief tracking · misleading stories · step-by-step execution with look-alike tool names (11-call plans) · semantic traps in code (5 of 6 traps active)Holds on 6 of 14 capabilities tested.
Nemotron 3 Nano 30BBreaks (2) long plans with look-alike tool names (13-call plans) · step-by-step execution with look-alike tool names (11-call plans)Slips (5) belief tracking · misleading stories · trap-laden catalogs (4 of 5 traps active) · spotting impossible tasks · recovering from a failed call (11-call plans, 2 values to carry)Holds on 4 of 11 capabilities tested.
Nemotron 3 Ultra 550BSlips (2) long plans with look-alike tool names (25-call plans) · spotting impossible tasksHolds on 6 of 8 capabilities tested.
Nemotron 3 Super 120BBreaks (1) long plans with look-alike tool names (24-call plans)Holds on 11 of 12 capabilities tested.
gpt-oss-120bHolds on all 4 capabilities tested.

One model Compare all models

BREAKSa level fell to 30% pass or below and stayed low (at most 40% from there up)DEGRADESa level fell to 75% or below; for single-rate rows, the 95% interval sits below 95%HOLDSneither, at every level testedNOT TESTEDno evidence yet

Strip cells, pass rate per level:under 50%50-75%75-90%90-97%97%+no scored trialsfaded cell: fewer than 5 scored trials▲ boundary levelgrade A strong · B moderate · C thin

I · Narrative4 of 4 tested

Tracks what a person believes from what they saw58% correct (301/518)Tested up to chain length 12, scramble 1.0, pronoun pressure 5, distractor events 20, witness cutoff 0.9

Belief tracking in deliberately misleading stories62% correct (160/260)Tested up to chain length 10, scramble 1.0, pronoun pressure 5, distractor events 20, witness cutoff 0.9

Tracks where things actually are100% correct (125/125)Tested up to chain length 30, distractor events 2

Recovers the true order of events95% correct (199/209)Tested up to chain length 20, scramble 1.0, distractor events 30

II · Tool planning5 of 5 tested

Plans long tool sequences when tool names look alikeDegrades at 16-call plans: 7/13 pass (was 25/27 at 14-call plans)Boundary 16-call plans · held through 14-call plans

7-call plans25-call plans

Plans around traps (poisoned tools, misleading descriptions, zero slack)Degrades at 4 of 5 traps active: 14/56 pass (was 21/21 at 1 of 5 traps active)Boundary 4 of 5 traps active · held through 1 of 5 traps active

0 of 5 traps active5 of 5 traps active

Notices when a task is impossibleDeclines 18 of 22 impossible tasks; writes a plan for the other 4

grade B moderate

Plans long tool sequences (clear tool names)Holds through 12-call plans: 53/53 pass across every tested levelHeld through 12-call plans

7-call plans12-call plans

Picks the right tool from a large catalogHolds through 66-tool catalogs: 93/93 pass across every tested levelHeld through 66-tool catalogs

8-tool catalogs66-tool catalogs

III · Interactive execution5 of 5 tested

Executes a plan step by step, carrying values between callsBreaks at 11-call plans, 3 values to carry: 15/55 pass (was 30/38 at 7-call plans, no values to carry)Boundary 11-call plans, 3 values to carry · held through 7-call plans, no values to carry

7-call plans, no values to carry12-call plans, 4 values to carry

Recovers when a tool call fails mid-plan (transient error, retry works)Breaks at 12-call plans, 4 values to carry: 0/10 pass (was 7/7 at 8-call plans, no values to carry)Boundary 12-call plans, 4 values to carry · held through 8-call plans, no values to carry

7-call plans, no values to carry12-call plans, 4 values to carry

grade B moderate

Executes step by step when tool names look alike (sees each error)Holds through 25-call plans: 43/44 pass across every tested levelHeld through 25-call plans

7-call plans31-call plans

grade B moderate

Executes a plan step by step, carrying values between calls (protocol v3)Holds through 7-call plans, no values to carry: 5/5 pass across every tested levelHeld through 7-call plans, no values to carry

7-call plans, no values to carry19-call plans, 16 values to carry

Recovers when a tool call fails mid-plan (protocol v3)Holds through 7-call plans, no values to carry: 5/5 pass across every tested levelHeld through 7-call plans, no values to carry

7-call plans, no values to carry19-call plans, 16 values to carry

IV · Code tracing3 of 3 tested

Traces a program when variable names look alikeHolds through 42-statement programs: 40/40 pass across every tested levelHeld through 42-statement programs

28-statement programs42-statement programs

Traces a program through semantic traps (aliasing, closures, off-by-one, modulo, defaults)Holds through 6 of 6 traps active: 80/80 pass across every tested levelHeld through 6 of 6 traps active

0 of 6 traps active6 of 6 traps active

Stays correct on small rewrites and adversarial programs100% correct (183/183)

V · Code repair2 of 2 tested

Fixes stacked bugs without breaking anything elseBreaks at 1 bugs: 1/8 pass -- already weak at the easiest level testedBoundary 1 bugs

grade B moderate

Fixes a one-bug function when shown a failing exampleDegrades at 0 hidden checks: 3/5 pass -- already weak at the easiest level testedBoundary 0 hidden checks

0 hidden checks4 hidden checks

V · Code writing0 of 1 tested

Writes a function from a description and examples, edge cases includedNot tested yet

VI · Repository3 of 6 tested

Finds and fixes a bug whose cause is in another fileBreaks at cause 1 calls away: 1/11 pass -- already weak at the easiest level testedBoundary cause 1 calls away

cause 1 calls awaycause 6 calls away

grade B moderate

Fixes the same bug as its program is split over more filesBreaks at 8 files: 4/18 pass (already only 9/22 at 1 files)Boundary 8 files

grade B moderate

Fixes a cross-file bug with a code-graph tool28% correct (5/18)

Graph-on arm of the paired VI.graph_value campaign; the paired comparison with the graph-off arm (<run_id>/graph_off) is in the campaign result note.

Calls the real functions when other files hold look-alike namesNot tested yet

Fixes the function that runs, not its near-copy elsewhereNot tested yet

Fixes a cross-file bug within a tool-call budgetNot tested yet

EF · Evidence fidelity6 of 6 tested

Copies a carried opaque value instead of inventing one (paired, B arm)0% correct (0/10)

Answers from what a person saw, against where things really are (paired, B arm)37% correct (11/30)Tested up to chain length 8, distractor events 2

grade B moderate

Copies exact look-alike tool ids (paired, B arm)80% correct (8/10)

Fixes a bug's cause in another file (paired, B arm)33% correct (3/9)

Copies a carried opaque value instead of inventing one (paired, B arm, protocol v3)100% correct (10/10)

Traces code over a misleading comment (paired, B arm)100% correct (10/10)

Reliability1 of 1 tested

Gets the same task right every time (pass^k)58% correct (34/59)

Rate shown is pass^1 over repeats of the same tasks (X.consistency campaign); pass^k needs per-task grouping and is in results/consistency/<run_id>.json. Earlier hand measurement: Nano 63% single-try vs 17% on all 5 tries.

grade B moderate