III · Interactive execution
Executes step by step when tool names look alike (sees each error)
Gemma 3 27B
Finding:DEGRADES· grade C(research conclusion)
Degrades at 11-call plans: 3/5 pass (was 5/5 at 7-call plans)
Boundary 11-call plans · held through 7-call plans
Pass rate by level
Dashed lines: a level at or below 30% pass counts toward BREAKS, at or below 75% toward DEGRADES. Labels above points: passed/scored. Hollow points: fewer than 5 scored trials, not used for the call.
| Level | Passed | Pass rate | Note |
|---|---|---|---|
| 7-call plans | 5/5 | 100% | |
| 11-call plans | 3/5 | 60% | ▲ boundary |
| 14-call plans | 4/5 | 80% | |
| 18-call plans | 3/5 | 60% | |
| 21-call plans | 4/5 | 80% | |
| 25-call plans | 3/5 | 60% | |
| 29-call plans | 2/2 | 100% | fewer than 5 scored, not used for the call; 3 errored (excluded) |
Why it slips
The model calls a tool that does not exist -- usually a misspelled look-alike of a real tool name.
in 62% of failures (8 failures at and beyond the boundary)
Example failure
Trial result: FAIL (one model answer)
FAILunknown_tool
step 3: set_dep_th_flag_1({'flag': 'depth_flag_1'}) -> unknown_tool:set_dep_th_flag_1Evidence
- Ground truthDeterministic simulation: the right answer is computed, not judged
- StatisticsWilson 95% interval 23%–88%
- Sample size5 scored trials at the boundary: small sample (under 15)
- Fresh-seed confirmationNot run yet
- ReplicationNot run yet
Raw evidence