III · Interactive execution
Executes a plan step by step, carrying values between calls
Gemma 4 31B QAT
Finding:BREAKS· grade A(research conclusion)
Breaks at 11-call plans, 3 values to carry: 15/55 pass (was 30/38 at 7-call plans, no values to carry)
Boundary 11-call plans, 3 values to carry · held through 7-call plans, no values to carry
Pass rate by level
Dashed lines: a level at or below 30% pass counts toward BREAKS, at or below 75% toward DEGRADES. Labels above points: passed/scored.
| Level | Passed | Pass rate | Note |
|---|---|---|---|
| 7-call plans, no values to carry | 30/38 | 79% | 10 errored (excluded) |
| 9-call plans, 1 value to carry | 6/10 | 60% | |
| 11-call plans, 3 values to carry | 15/55 | 27% | ▲ boundary |
| 12-call plans, 4 values to carry | 2/10 | 20% |
Why it breaks
The model fails to pass on a value returned by an earlier call.
in 100% of failures (48 failures at and beyond the boundary)
Example failure
Trial result: FAIL (one model answer)
FAILinvalid_ticket
step 8: set_depth_flag_1({'flag': 'depth_flag_1', 'ticket': 'ticket_depth_flag_0_123'}) -> invalid_ticket: `ticket` must be the exact ticket returned by the most recent successful call to set_depth_flag_0.Evidence
- Ground truthDeterministic simulation: the right answer is computed, not judged
- StatisticsWilson 95% interval 17%–40%
- Sample size55 scored trials at the boundary
- Fresh-seed confirmationNot run yet
- ReplicationNot run yet
Raw evidence