II · Tool planning

Plans long tool sequences when tool names look alike

Gemma 3 27B

Finding:BREAKS· grade B(research conclusion)

Breaks at 17-call plans: 0/7 pass (already only 7/14 at 7-call plans)

Boundary 17-call plans

Pass rate by level7 levels tested

Dashed lines: a level at or below 30% pass counts toward BREAKS, at or below 75% toward DEGRADES. Labels above points: passed/scored.

Level	Passed	Pass rate	Note
7-call plans	7/14	50%
11-call plans	5/7	71%
13-call plans	13/28	46%
14-call plans	7/14	50%
16-call plans	12/28	43%
17-call plans	0/7	0%	▲ boundary
18-call plans	1/7	14%

Why it breaks

The model calls a tool that does not exist -- usually a misspelled look-alike of a real tool name.

Example failureat 17-call plans

Trial result: FAIL (one model answer)

FAILunknown_tool:step=4:set_dep-th_flag_1

```json [ {"tool": "build-_art_0", "args": {"artifact": "art_0"}}, {"tool": "publish_art_o0", "args": {"artifact": "art_0"}}, {"tool": "set_-flag_0", "args": {"flag": "flag_0"}}, {"tool": "set_dep-th_flag_0", "args": {"flag": "depth_flag_0"}}, {"tool": "set_dep-th_flag_1", "args": {"flag": "depth_flag_1"}}, {"tool": "set_dep_th_flag_2", "args": {"flag": "depth_flag_2"}}, {"tool": "set_dep_th_flag_

Open this trial →Open the run it came from →

Evidencewhat this finding rests on

Ground truthDeterministic simulation: the right answer is computed, not judged
StatisticsWilson 95% interval 0%–35%Grade B: moderate evidence
Sample size7 scored trials at the boundary: small sample (under 15)
Fresh-seed confirmationNot run yet
ReplicationNot run yet

Raw evidenceruns and trials

Runs behind this row
- camp-tool-deep-gemma-001 70 trials
- camp-tool-depth-gemma-003 35 trials
Every tool-planning trial for this model → broader than this row
The example failure, in full →

Back to the capability map →