III · Interactive execution
Recovers when a tool call fails mid-plan (transient error, retry works)
Nemotron 3 Nano 30B
Finding:DEGRADES· grade C(research conclusion)
Degrades at 11-call plans, 2 values to carry: 3/5 pass (was 5/5 at 9-call plans, 1 value to carry)
Boundary 11-call plans, 2 values to carry · held through 9-call plans, 1 value to carry
Pass rate by level
Dashed lines: a level at or below 30% pass counts toward BREAKS, at or below 75% toward DEGRADES. Labels above points: passed/scored. Hollow points: fewer than 5 scored trials, not used for the call.
| Level | Passed | Pass rate | Note |
|---|---|---|---|
| 7-call plans, no values to carry | 5/5 | 100% | |
| 9-call plans, 1 value to carry | 5/5 | 100% | |
| 11-call plans, 2 values to carry | 3/5 | 60% | ▲ boundary |
| 12-call plans, 4 values to carry | 4/5 | 80% | |
| 14-call plans, 7 values to carry | 3/3 | 100% | fewer than 5 scored, not used for the call; 2 errored (excluded) |
Why it slips
The model stops acting before the goal is reached.
in 67% of failures (3 failures at and beyond the boundary)
Example failure
Trial result: FAIL (one model answer)
FAILstopped_short
[]
Evidence
- Ground truthDeterministic simulation: the right answer is computed, not judged
- StatisticsWilson 95% interval 23%–88%
- Sample size5 scored trials at the boundary: small sample (under 15)
- Fresh-seed confirmationNot run yet
- ReplicationNot run yet
Raw evidence