III · Interactive execution
Recovers when a tool call fails mid-plan (transient error, retry works)
Gemma 4 31B QAT
Finding:BREAKS· grade B(research conclusion)
Breaks at 12-call plans, 4 values to carry: 0/10 pass (was 7/7 at 8-call plans, no values to carry)
Boundary 12-call plans, 4 values to carry · held through 8-call plans, no values to carry
Pass rate by level
Dashed lines: a level at or below 30% pass counts toward BREAKS, at or below 75% toward DEGRADES. Labels above points: passed/scored.
| Level | Passed | Pass rate | Note |
|---|---|---|---|
| 7-call plans, no values to carry | 30/30 | 100% | 9 errored (excluded) |
| 8-call plans, no values to carry | 7/7 | 100% | 2 errored (excluded) |
| 9-call plans, 1 value to carry | 5/10 | 50% | 3 errored (excluded) |
| 11-call plans, 3 values to carry | 24/70 | 34% | |
| 12-call plans, 4 values to carry | 0/10 | 0% | ▲ boundary |
Why it breaks
The model fails to pass on a value returned by an earlier call.
in 100% of failures (10 failures at and beyond the boundary)
Example failure
Trial result: FAIL (one model answer)
FAILinvalid_ticket
step 2: advance_1({'service': 'svc_0', 'ticket': 'ticket_advance_0_svc_0'}) -> invalid_ticket: `ticket` must be the exact ticket returned by the most recent successful call to advance_0.Evidence
- Ground truthDeterministic simulation: the right answer is computed, not judged
- StatisticsWilson 95% interval 0%–28%
- Sample size10 scored trials at the boundary: small sample (under 15)
- Fresh-seed confirmationNot run yet
- ReplicationNot run yet
Raw evidence