Data to …… trials
II · Tool planning

Plans long tool sequences when tool names look alike

Gemma 4 31B QAT

Finding:DEGRADES· grade A(research conclusion)

Degrades at 16-call plans: 7/13 pass (was 25/27 at 14-call plans)

Boundary 16-call plans · held through 14-call plans

Pass rate by level10 levels tested
Pass rate by level for Plans long tool sequences when tool names look alike0%30%50%75%100%Boundary: degrades at 16-call plans7-call plans7-call plans: 26/28 pass (93%)26/2811-call plans11-call plans: 13/13 pass (100%)13/1314-call plans14-call plans: 25/27 pass (93%)25/2716-call plans16-call plans: 7/13 pass (54%)7/1318-call plans18-call plans: 16/26 pass (62%)16/2619-call plans19-call plans: 7/13 pass (54%)7/1321-call plans21-call plans: 11/13 pass (85%)11/1323-call plans23-call plans: 10/14 pass (71%)10/1424-call plans24-call plans: 9/12 pass (75%), 2 errored9/1225-call plans25-call plans: 31/56 pass (55%)31/56

Dashed lines: a level at or below 30% pass counts toward BREAKS, at or below 75% toward DEGRADES. Labels above points: passed/scored.

LevelPassedPass rateNote
7-call plans26/2893%
11-call plans13/13100%
14-call plans25/2793%
16-call plans7/1354%▲ boundary
18-call plans16/2662%
19-call plans7/1354%
21-call plans11/1385%
23-call plans10/1471%
24-call plans9/1275%2 errored (excluded)
25-call plans31/5655%
Why it slips

The model calls a tool that does not exist -- usually a misspelled look-alike of a real tool name.

in 100% of failures (56 failures at and beyond the boundary)

Example failureat 16-call plans
Trial result: FAIL (one model answer)
FAILunknown_tool:step=2:set_dep-th_flag_1
[ {"tool": "set_flag_o0", "args": {"flag": "flag_0"}}, {"tool": "set_dep-th_flag_0", "args": {"flag": "depth_flag_0"}}, {"tool": "set_dep-th_flag_1", "args": {"flag": "depth_flag_1"}}, {"tool": "set_dep_th_flag_2", "args": {"flag": "depth_flag_2"}}, {"tool": "set_depth_flag_o3", "args": {"flag": "depth_flag_3"}}, {"tool": "set_dep-th_flag_4", "args": {"flag": "depth_flag_4"}}, {"tool": "set_dep-th
Open this trial →Open the run it came from →
Evidencewhat this finding rests on
  • Ground truthDeterministic simulation: the right answer is computed, not judged
  • StatisticsWilson 95% interval 29%–77%p = 1.4×10-7, before vs after the boundary (one-sided Fisher exact)Grade A: strong evidence
  • Sample size13 scored trials at the boundary: small sample (under 15)
  • Fresh-seed confirmationNot run yet
  • ReplicationNot run yet
Raw evidenceruns and trials

Back to the capability map →