III · Interactive execution
Executes step by step when tool names look alike (sees each error)
Gemma 4 31B QAT
Finding:HOLDS· grade B(research conclusion)
Holds through 25-call plans: 43/44 pass across every tested level
Held through 25-call plans
Pass rate by level
Dashed lines: a level at or below 30% pass counts toward BREAKS, at or below 75% toward DEGRADES. Labels above points: passed/scored. Hollow points: fewer than 5 scored trials, not used for the call.
| Level | Passed | Pass rate | Note |
|---|---|---|---|
| 7-call plans | 9/9 | 100% | 9 errored (excluded) |
| 11-call plans | 7/7 | 100% | 1 errored (excluded) |
| 14-call plans | 8/8 | 100% | |
| 18-call plans | 8/8 | 100% | |
| 21-call plans | 6/7 | 86% | 1 errored (excluded) |
| 25-call plans | 5/5 | 100% | |
| 29-call plans | 2/2 | 100% | fewer than 5 scored, not used for the call; 3 errored (excluded) |
| 30-call plans | 2/2 | 100% | fewer than 5 scored, not used for the call; 3 errored (excluded) |
| 31-call plans | 1/1 | 100% | fewer than 5 scored, not used for the call; 4 errored (excluded) |
What the few failures have in common
The model stops acting before the goal is reached.
in 100% of failures (1 failures)
Evidence
- Ground truthDeterministic simulation: the right answer is computed, not judged
- StatisticsWilson 95% interval 88%–100%
- Sample size44 scored trials in total
- Fresh-seed confirmationNot run yet
- ReplicationNot run yet
Raw evidence