II · Tool planning

Plans around traps (poisoned tools, misleading descriptions, zero slack)

Nemotron 3 Nano 30B

Finding:DEGRADES· grade A(research conclusion)

Degrades at 4 of 5 traps active: 13/35 pass (was 34/35 at 0 of 5 traps active)

Boundary 4 of 5 traps active · held through 0 of 5 traps active

Pass rate by level3 levels tested

Dashed lines: a level at or below 30% pass counts toward BREAKS, at or below 75% toward DEGRADES. Labels above points: passed/scored.

Level	Passed	Pass rate	Note
0 of 5 traps active	34/35	97%
4 of 5 traps active	13/35	37%	▲ boundary
5 of 5 traps active	18/56	32%

Why it slips

The model calls a real tool before the step it depends on is done.

Example failureat 4 of 5 traps active

Trial result: FAIL (one model answer)

FAILprecondition_violation:step=9:adva-nce_1

[ {"tool":"set_flag_0","args":{"flag":"flag_0"}}, {"tool":"set_-flag_1","args":{"flag":"flag_1"}}, {"tool":"open_interleave_window","args":{}}, {"tool":"build_art_o0","args":{"artifact":"art_0"}}, {"tool":"publis-h_art_0","args":{"artifact":"art_0"}}, {"tool":"advance_o0","args":{"service":"svc_0"}}, {"tool":"adva-nce_1","args":{"service":"svc_0"}}, {"tool":"adva-nce_2","args":{"service":"svc_0"}}

Open this trial →Open the run it came from →

Evidencewhat this finding rests on

Ground truthDeterministic simulation: the right answer is computed, not judged
StatisticsWilson 95% interval 23%–54%p = 1.2×10^-11, before vs after the boundary (one-sided Fisher exact)Grade A: strong evidence
Sample size35 scored trials at the boundary
Fresh-seed confirmationNot run yet
ReplicationNot run yet

Raw evidenceruns and trials

Runs behind this row
- camp-tool_bp-nano-002 126 trials
Every tool-planning trial for this model → broader than this row
The example failure, in full →

Back to the capability map →