Breaking points

Each row is one capability, scored against a deterministic simulation that knows the right answer. Where a capability has a difficulty dial, the dial is turned up until the model stops passing, and the point where it stopped is reported in the task’s own units.

Verdict per model6 models

Gemma 4 31B QATBreaks (7) step-by-step execution with values to carry (11-call plans, 3 values to carry) · recovering from a failed call (12-call plans, 4 values to carry) · repairing stacked bugs (1 bugs) · fixing a bug across files (cause 1 calls away) · the same program over more files (8 files) · a code-graph tool · carried valuesSlips (10) belief tracking · misleading stories · long plans with look-alike tool names (16-call plans) · trap-laden catalogs (4 of 5 traps active) · spotting impossible tasks · repairing a bug (0 hidden checks) · belief against reality · exact ids against natural names · cause over symptom · consistencyHolds on 12 of 29 capabilities tested.
Gemma 3 27BBreaks (3) long plans with look-alike tool names (17-call plans) · trap-laden catalogs (4 of 5 traps active) · spotting impossible tasksSlips (5) tracking where things are · belief tracking · misleading stories · step-by-step execution with look-alike tool names (11-call plans) · semantic traps in code (5 of 6 traps active)Holds on 6 of 14 capabilities tested.
Nemotron 3 Nano 30BBreaks (2) long plans with look-alike tool names (13-call plans) · step-by-step execution with look-alike tool names (11-call plans)Slips (5) belief tracking · misleading stories · trap-laden catalogs (4 of 5 traps active) · spotting impossible tasks · recovering from a failed call (11-call plans, 2 values to carry)Holds on 4 of 11 capabilities tested.
Nemotron 3 Ultra 550BSlips (2) long plans with look-alike tool names (25-call plans) · spotting impossible tasksHolds on 6 of 8 capabilities tested.
Nemotron 3 Super 120BBreaks (1) long plans with look-alike tool names (24-call plans)Holds on 11 of 12 capabilities tested.
gpt-oss-120bHolds on all 4 capabilities tested.

One model Compare all models

BREAKSa level fell to 30% pass or below and stayed low (at most 40% from there up)DEGRADESa level fell to 75% or below; for single-rate rows, the 95% interval sits below 95%HOLDSneither, at every level testedNOT TESTEDno evidence yet

Strip cells, pass rate per level:under 50%50-75%75-90%90-97%97%+no scored trialsfaded cell: fewer than 5 scored trials▲ boundary levelgrade A strong · B moderate · C thin

I · Narrative6 models

Tracks where things actually are

Gemma 4 31B QAT

100% correct (125/125)grade A strongTested up to chain length 30, distractor events 2

57% correct (26/46)grade B moderateTested up to chain length 30, distractor events 2

Nemotron 3 Nano 30B

100% correct (90/90)grade B moderateTested up to chain length 30, distractor events 2

Nemotron 3 Ultra 550B

100% correct (30/30)grade B moderateTested up to chain length 30, distractor events 2

Nemotron 3 Super 120B

96% correct (91/95)grade B moderateTested up to chain length 30, distractor events 2

100% correct (60/60)grade B moderateTested up to chain length 30, distractor events 2

Recovers the true order of events

Gemma 4 31B QAT

95% correct (199/209)grade A strongTested up to chain length 20, scramble 1.0, distractor events 30

99% correct (79/80)grade B moderateTested up to chain length 20, scramble 1.0, distractor events 30

Nemotron 3 Nano 30B

99% correct (156/158)grade A strongTested up to chain length 20, scramble 1.0, distractor events 30

Nemotron 3 Ultra 550B

100% correct (80/80)grade B moderateTested up to chain length 20, scramble 1.0, distractor events 30

Nemotron 3 Super 120B

99% correct (79/80)grade B moderateTested up to chain length 20, scramble 1.0, distractor events 30

99% correct (79/80)grade B moderateTested up to chain length 20, scramble 1.0, distractor events 30

Tracks what a person believes from what they saw

Gemma 4 31B QAT

58% correct (301/518)grade A strongTested up to chain length 12, scramble 1.0, pronoun pressure 5, distractor events 20, witness cutoff 0.9

49% correct (80/162)grade A strongTested up to chain length 10, scramble 1.0, pronoun pressure 5, distractor events 20, witness cutoff 0.9

Nemotron 3 Nano 30B

90% correct (521/578)grade A strongTested up to chain length 25, scramble 1.0, pronoun pressure 8, distractor events 40, witness cutoff 0.9

Nemotron 3 Ultra 550B

99% correct (181/183)grade A strongTested up to chain length 25, scramble 1.0, pronoun pressure 8, distractor events 40, witness cutoff 0.9

Nemotron 3 Super 120B

97% correct (283/291)grade A strongTested up to chain length 25, scramble 1.0, pronoun pressure 8, distractor events 40, witness cutoff 0.9

97% correct (154/159)grade A strongTested up to chain length 10, scramble 1.0, pronoun pressure 5, distractor events 20, witness cutoff 0.9

Belief tracking in deliberately misleading stories

Gemma 4 31B QAT

62% correct (160/260)grade A strongTested up to chain length 10, scramble 1.0, pronoun pressure 5, distractor events 20, witness cutoff 0.9

43% correct (62/145)grade A strongTested up to chain length 10, scramble 1.0, pronoun pressure 5, distractor events 20, witness cutoff 0.9

Nemotron 3 Nano 30B

88% correct (279/317)grade A strongTested up to chain length 10, scramble 1.0, pronoun pressure 5, distractor events 20, witness cutoff 0.9

Nemotron 3 Ultra 550B

99% correct (127/128)grade A strongTested up to chain length 10, scramble 1.0, pronoun pressure 5, distractor events 20, witness cutoff 0.9

Nemotron 3 Super 120B

95% correct (152/160)grade A strongTested up to chain length 10, scramble 1.0, pronoun pressure 5, distractor events 20, witness cutoff 0.9

98% correct (41/42)grade B moderateTested up to chain length 10, scramble 0.8, distractor events 2

II · Tool planning6 models

Plans long tool sequences (clear tool names)

Gemma 4 31B QAT

Holds through 12-call plans: 53/53 pass across every tested levelgrade A strongHeld through 12-call plans

Holds through 13-call plans: 34/42 pass across every tested levelgrade B moderateHeld through 13-call plans

Nemotron 3 Nano 30B

Holds through 13-call plans: 46/49 pass across every tested levelgrade B moderateHeld through 13-call plans

Nemotron 3 Ultra 550B

Holds through 12-call plans: 28/28 pass across every tested levelgrade B moderateHeld through 12-call plans

Nemotron 3 Super 120B

Holds through 12-call plans: 28/28 pass across every tested levelgrade B moderateHeld through 12-call plans

7-call plans13-call plans

Plans long tool sequences when tool names look alike

Gemma 4 31B QAT

Degrades at 16-call plans: 7/13 pass (was 25/27 at 14-call plans)grade A strongBoundary 16-call plans · held through 14-call plans

Breaks at 17-call plans: 0/7 pass (already only 7/14 at 7-call plans)grade B moderateBoundary 17-call plans

Nemotron 3 Nano 30B

Breaks at 13-call plans: 5/21 pass (already only 15/28 at 7-call plans)grade B moderateBoundary 13-call plans

Nemotron 3 Ultra 550B

Degrades at 25-call plans: 4/7 pass (was 7/7 at 21-call plans)grade A strongBoundary 25-call plans · held through 21-call plans

Nemotron 3 Super 120B

Breaks at 24-call plans: 8/28 pass (was 6/7 at 11-call plans)grade A strongBoundary 24-call plans · held through 11-call plans

7-call plans31-call plans

Plans around traps (poisoned tools, misleading descriptions, zero slack)

Gemma 4 31B QAT

Degrades at 4 of 5 traps active: 14/56 pass (was 21/21 at 1 of 5 traps active)grade A strongBoundary 4 of 5 traps active · held through 1 of 5 traps active

Breaks at 4 of 5 traps active: 4/35 pass (was 20/21 at 1 of 5 traps active)grade A strongBoundary 4 of 5 traps active · held through 1 of 5 traps active

Nemotron 3 Nano 30B

Degrades at 4 of 5 traps active: 13/35 pass (was 34/35 at 0 of 5 traps active)grade A strongBoundary 4 of 5 traps active · held through 0 of 5 traps active

Nemotron 3 Ultra 550B

Holds through 5 of 5 traps active: 51/56 pass across every tested levelgrade B moderateHeld through 5 of 5 traps active

Nemotron 3 Super 120B

Holds through 5 of 5 traps active: 51/52 pass across every tested levelgrade B moderateHeld through 5 of 5 traps active

0 of 5 traps active5 of 5 traps active

Picks the right tool from a large catalog

Gemma 4 31B QAT

Holds through 66-tool catalogs: 93/93 pass across every tested levelgrade A strongHeld through 66-tool catalogs

Holds through 66-tool catalogs: 46/56 pass across every tested levelgrade B moderateHeld through 66-tool catalogs

Nemotron 3 Nano 30B

Nemotron 3 Ultra 550B

Nemotron 3 Super 120B

Holds through 66-tool catalogs: 112/112 pass across every tested levelgrade A strongHeld through 66-tool catalogs

8-tool catalogs66-tool catalogs

Notices when a task is impossible

Gemma 4 31B QAT

Declines 18 of 22 impossible tasks; writes a plan for the other 4grade B moderate

Declines 0 of 16 impossible tasks; writes a plan for the other 16grade C thin

Nemotron 3 Nano 30B

Declines 6 of 12 impossible tasks; writes a plan for the other 6grade C thin

Nemotron 3 Ultra 550B

Declines 5 of 8 impossible tasks; writes a plan for the other 3grade C thin

Nemotron 3 Super 120B

Declines 10 of 12 impossible tasks; writes a plan for the other 2grade C thin

III · Interactive execution6 models

Executes a plan step by step, carrying values between calls

Gemma 4 31B QAT

Breaks at 11-call plans, 3 values to carry: 15/55 pass (was 30/38 at 7-call plans, no values to carry)grade A strongBoundary 11-call plans, 3 values to carry · held through 7-call plans, no values to carry

Holds through 19-call plans, 16 values to carry: 40/40 pass across every tested levelgrade A strongHeld through 19-call plans, 16 values to carry

Nemotron 3 Nano 30B

Holds through 19-call plans, 16 values to carry: 43/45 pass across every tested levelgrade B moderateHeld through 19-call plans, 16 values to carry

Nemotron 3 Ultra 550B

Nemotron 3 Super 120B

Holds through 19-call plans, 16 values to carry: 53/53 pass across every tested levelgrade A strongHeld through 19-call plans, 16 values to carry

7-call plans, no values to carry19-call plans, 16 values to carry

Executes step by step when tool names look alike (sees each error)

Gemma 4 31B QAT

Holds through 25-call plans: 43/44 pass across every tested levelgrade B moderateHeld through 25-call plans

Degrades at 11-call plans: 3/5 pass (was 5/5 at 7-call plans)grade C thinBoundary 11-call plans · held through 7-call plans

Nemotron 3 Nano 30B

Breaks at 11-call plans: 1/5 pass (was 4/5 at 7-call plans)grade C thinBoundary 11-call plans · held through 7-call plans

Nemotron 3 Ultra 550B

Too few trials to call

Nemotron 3 Super 120B

Holds through 24-call plans: 43/47 pass across every tested levelgrade B moderateHeld through 24-call plans

7-call plans31-call plans

Recovers when a tool call fails mid-plan (transient error, retry works)

Gemma 4 31B QAT

Breaks at 12-call plans, 4 values to carry: 0/10 pass (was 7/7 at 8-call plans, no values to carry)grade B moderateBoundary 12-call plans, 4 values to carry · held through 8-call plans, no values to carry

Holds through 19-call plans, 16 values to carry: 40/40 pass across every tested levelgrade A strongHeld through 19-call plans, 16 values to carry

Nemotron 3 Nano 30B

Degrades at 11-call plans, 2 values to carry: 3/5 pass (was 5/5 at 9-call plans, 1 value to carry)grade C thinBoundary 11-call plans, 2 values to carry · held through 9-call plans, 1 value to carry

Nemotron 3 Ultra 550B

Nemotron 3 Super 120B

Holds through 16-call plans, 10 values to carry: 30/30 pass across every tested levelgrade B moderateHeld through 16-call plans, 10 values to carry

7-call plans, no values to carry19-call plans, 16 values to carry

Executes a plan step by step, carrying values between calls (protocol v3)

Gemma 4 31B QAT

Holds through 7-call plans, no values to carry: 5/5 pass across every tested levelgrade C thinHeld through 7-call plans, no values to carry

Nemotron 3 Nano 30B

Nemotron 3 Ultra 550B

Nemotron 3 Super 120B

7-call plans, no values to carry19-call plans, 16 values to carry

Recovers when a tool call fails mid-plan (protocol v3)

Gemma 4 31B QAT

Holds through 7-call plans, no values to carry: 5/5 pass across every tested levelgrade C thinHeld through 7-call plans, no values to carry

Nemotron 3 Nano 30B

Nemotron 3 Ultra 550B

Nemotron 3 Super 120B

7-call plans, no values to carry19-call plans, 16 values to carry

IV · Code tracing6 models

Traces a program when variable names look alike

Gemma 4 31B QAT

Holds through 42-statement programs: 40/40 pass across every tested levelgrade A strongHeld through 42-statement programs

Holds through 42-statement programs: 25/28 pass across every tested levelgrade B moderateHeld through 42-statement programs

Nemotron 3 Nano 30B

Nemotron 3 Ultra 550B

Nemotron 3 Super 120B

28-statement programs42-statement programs

Traces a program through semantic traps (aliasing, closures, off-by-one, modulo, defaults)

Gemma 4 31B QAT

Holds through 6 of 6 traps active: 80/80 pass across every tested levelgrade A strongHeld through 6 of 6 traps active

Degrades at 5 of 6 traps active: 26/49 pass (was 41/42 at 0 of 6 traps active)grade A strongBoundary 5 of 6 traps active · held through 0 of 6 traps active

Nemotron 3 Nano 30B

Nemotron 3 Ultra 550B

Nemotron 3 Super 120B

0 of 6 traps active6 of 6 traps active

Stays correct on small rewrites and adversarial programs

Gemma 4 31B QAT

100% correct (183/183)grade A strong

Nemotron 3 Nano 30B

Nemotron 3 Ultra 550B

Nemotron 3 Super 120B

V · Code repair6 models

Fixes a one-bug function when shown a failing example

Gemma 4 31B QAT

Degrades at 0 hidden checks: 3/5 pass -- already weak at the easiest level testedgrade C thinBoundary 0 hidden checks

Nemotron 3 Nano 30B

Nemotron 3 Ultra 550B

Nemotron 3 Super 120B

0 hidden checks4 hidden checks

Fixes stacked bugs without breaking anything else

Gemma 4 31B QAT

Breaks at 1 bugs: 1/8 pass -- already weak at the easiest level testedgrade B moderateBoundary 1 bugs

Nemotron 3 Nano 30B

Nemotron 3 Ultra 550B

Nemotron 3 Super 120B

V · Code writing6 models

Writes a function from a description and examples, edge cases included

Gemma 4 31B QAT

Nemotron 3 Nano 30B

Nemotron 3 Ultra 550B

Nemotron 3 Super 120B

VI · Repository6 models

Finds and fixes a bug whose cause is in another file

Gemma 4 31B QAT

Breaks at cause 1 calls away: 1/11 pass -- already weak at the easiest level testedgrade B moderateBoundary cause 1 calls away

Nemotron 3 Nano 30B

Nemotron 3 Ultra 550B

Nemotron 3 Super 120B

cause 1 calls awaycause 6 calls away

Calls the real functions when other files hold look-alike names

Gemma 4 31B QAT

Nemotron 3 Nano 30B

Nemotron 3 Ultra 550B

Nemotron 3 Super 120B

Fixes the function that runs, not its near-copy elsewhere

Gemma 4 31B QAT

Nemotron 3 Nano 30B

Nemotron 3 Ultra 550B

Nemotron 3 Super 120B

Fixes a cross-file bug within a tool-call budget

Gemma 4 31B QAT

Nemotron 3 Nano 30B

Nemotron 3 Ultra 550B

Nemotron 3 Super 120B

Fixes the same bug as its program is split over more files

Gemma 4 31B QAT

Breaks at 8 files: 4/18 pass (already only 9/22 at 1 files)grade B moderateBoundary 8 files

Nemotron 3 Nano 30B

Nemotron 3 Ultra 550B

Nemotron 3 Super 120B

1 files8 files

Fixes a cross-file bug with a code-graph tool

Gemma 4 31B QAT

28% correct (5/18)grade C thin

Nemotron 3 Nano 30B

Nemotron 3 Ultra 550B

Nemotron 3 Super 120B

EF · Evidence fidelity6 models

Answers from what a person saw, against where things really are (paired, B arm)

Gemma 4 31B QAT

37% correct (11/30)grade B moderateTested up to chain length 8, distractor events 2

Nemotron 3 Nano 30B

Nemotron 3 Ultra 550B

Nemotron 3 Super 120B

Copies exact look-alike tool ids (paired, B arm)

Gemma 4 31B QAT

80% correct (8/10)grade C thin

Nemotron 3 Nano 30B

Nemotron 3 Ultra 550B

Nemotron 3 Super 120B

Copies a carried opaque value instead of inventing one (paired, B arm)

Gemma 4 31B QAT

0% correct (0/10)grade C thin

Nemotron 3 Nano 30B

Nemotron 3 Ultra 550B

Nemotron 3 Super 120B

Copies a carried opaque value instead of inventing one (paired, B arm, protocol v3)

Gemma 4 31B QAT

100% correct (10/10)grade C thin

Nemotron 3 Nano 30B

Nemotron 3 Ultra 550B

Nemotron 3 Super 120B

Traces code over a misleading comment (paired, B arm)

Gemma 4 31B QAT

100% correct (10/10)grade C thin

Nemotron 3 Nano 30B

Nemotron 3 Ultra 550B

Nemotron 3 Super 120B

Fixes a bug's cause in another file (paired, B arm)

Gemma 4 31B QAT

33% correct (3/9)grade C thin

Nemotron 3 Nano 30B

Nemotron 3 Ultra 550B

Nemotron 3 Super 120B

Reliability6 models

Gets the same task right every time (pass^k)

Gemma 4 31B QAT

58% correct (34/59)grade B moderate

Nemotron 3 Nano 30B

Nemotron 3 Ultra 550B

Nemotron 3 Super 120B