III · Interactive execution
Executes step by step when tool names look alike (sees each error)
Nemotron 3 Super 120B
Finding:HOLDS· grade B(research conclusion)
Holds through 24-call plans: 43/47 pass across every tested level
Held through 24-call plans
Pass rate by level
Dashed lines: a level at or below 30% pass counts toward BREAKS, at or below 75% toward DEGRADES. Labels above points: passed/scored. Hollow points: fewer than 5 scored trials, not used for the call.
| Level | Passed | Pass rate | Note |
|---|---|---|---|
| 15-call plans | 4/5 | 80% | |
| 17-call plans | 4/5 | 80% | |
| 18-call plans | 17/17 | 100% | 3 errored (excluded) |
| 20-call plans | 5/5 | 100% | |
| 22-call plans | 4/5 | 80% | |
| 23-call plans | 4/5 | 80% | |
| 24-call plans | 5/5 | 100% | |
| 25-call plans | 4/4 | 100% | fewer than 5 scored, not used for the call; 6 errored (excluded) |
What the few failures have in common
The model calls a tool that does not exist -- usually a misspelled look-alike of a real tool name.
in 75% of failures (4 failures)
Evidence
- Ground truthDeterministic simulation: the right answer is computed, not judged
- StatisticsWilson 95% interval 80%–97%
- Sample size47 scored trials in total
- Fresh-seed confirmationNot run yet
- ReplicationNot run yet
Raw evidence