II · Tool planning

Plans around traps (poisoned tools, misleading descriptions, zero slack)

Gemma 4 31B QAT

Finding:DEGRADES· grade A(research conclusion)

Degrades at 4 of 5 traps active: 14/56 pass (was 21/21 at 1 of 5 traps active)

Boundary 4 of 5 traps active · held through 1 of 5 traps active

Pass rate by level4 levels tested

Dashed lines: a level at or below 30% pass counts toward BREAKS, at or below 75% toward DEGRADES. Labels above points: passed/scored.

Level	Passed	Pass rate	Note
0 of 5 traps active	74/74	100%
1 of 5 traps active	21/21	100%
4 of 5 traps active	14/56	25%	▲ boundary
5 of 5 traps active	63/94	67%

Why it slips

The model calls a real tool before the step it depends on is done.

Example failureat 4 of 5 traps active

Trial result: FAIL (one model answer)

FAILprecondition_violation:step=9:adva-nce_1

[ {"tool": "set_flag_0", "args": {"flag": "flag_0"}}, {"tool": "set_-flag_1", "args": {"flag": "flag_1"}}, {"tool": "build_art_0", "args": {"artifact": "art_0"}}, {"tool": "publish_art_o0", "args": {"artifact": "art_0"}}, {"tool": "open_interleave_window", "args": {}}, {"tool": "adva-nce_0", "args": {"service": "svc_0"}}, {"tool": "adva-nce_1", "args": {"service": "svc_0"}}, {"tool": "advance_o2",

Open this trial →Open the run it came from →

Evidencewhat this finding rests on

Ground truthDeterministic simulation: the right answer is computed, not judged
StatisticsWilson 95% interval 16%–38%p = 2.4×10^-20, before vs after the boundary (one-sided Fisher exact)Grade A: strong evidence
Sample size56 scored trials at the boundary
Fresh-seed confirmationNot run yet
ReplicationNot run yet

Raw evidenceruns and trials

Runs behind this row
- camp-tool_bp-gemma4qat-001 245 trials
Every tool-planning trial for this model → broader than this row
The example failure, in full →

Back to the capability map →