II · Tool planning

Plans around traps (poisoned tools, misleading descriptions, zero slack)

Gemma 3 27B

Finding:BREAKS· grade A(research conclusion)

Breaks at 4 of 5 traps active: 4/35 pass (was 20/21 at 1 of 5 traps active)

Boundary 4 of 5 traps active · held through 1 of 5 traps active

Pass rate by level4 levels tested

Dashed lines: a level at or below 30% pass counts toward BREAKS, at or below 75% toward DEGRADES. Labels above points: passed/scored.

Level	Passed	Pass rate	Note
0 of 5 traps active	37/49	76%
1 of 5 traps active	20/21	95%
4 of 5 traps active	4/35	11%	▲ boundary
5 of 5 traps active	11/49	22%

Why it breaks

The model calls a real tool before the step it depends on is done.

Example failureat 4 of 5 traps active

Trial result: FAIL (one model answer)

FAILprecondition_violation:step=7:advance_o2

```json [ {"tool": "open_interleave_window", "args": {}}, {"tool": "set_flag_0", "args": {"flag": "flag_0"}}, {"tool": "adva-nce_0", "args": {"service": "svc_0"}}, {"tool": "adva-nce_0", "args": {"service": "svc_1"}}, {"tool": "advance_o1", "args": {"service": "svc_1"}}, {"tool": "build-_art_0", "args": {"artifact": "art_0"}}, {"tool": "publis_h_art_0", "args": {"artifact": "art_0"}}, {"tool": "ad

Open this trial →Open the run it came from →

Evidencewhat this finding rests on

Ground truthDeterministic simulation: the right answer is computed, not judged
StatisticsWilson 95% interval 4%–26%p = 7.0×10^-16, before vs after the boundary (one-sided Fisher exact)Grade A: strong evidence
Sample size35 scored trials at the boundary
Fresh-seed confirmationNot run yet
ReplicationNot run yet

Raw evidenceruns and trials

Runs behind this row
- camp-tool_bp-gemma-001 154 trials
Every tool-planning trial for this model → broader than this row
The example failure, in full →

Back to the capability map →