II · Tool planning
Plans long tool sequences (clear tool names)
Nemotron 3 Nano 30B
Finding:HOLDS· grade B(research conclusion)
Holds through 13-call plans: 46/49 pass across every tested level
Held through 13-call plans
Pass rate by level
Dashed lines: a level at or below 30% pass counts toward BREAKS, at or below 75% toward DEGRADES. Labels above points: passed/scored.
| Level | Passed | Pass rate | Note |
|---|---|---|---|
| 7-call plans | 12/14 | 86% | |
| 9-call plans | 6/7 | 86% | |
| 11-call plans | 7/7 | 100% | |
| 12-call plans | 7/7 | 100% | |
| 13-call plans | 14/14 | 100% |
What the few failures have in common
The model passes the wrong entity to a tool.
in 67% of failures (3 failures)
Evidence
- Ground truthDeterministic simulation: the right answer is computed, not judged
- StatisticsWilson 95% interval 84%–98%
- Sample size49 scored trials in total
- Fresh-seed confirmationNot run yet
- ReplicationNot run yet
Raw evidence
- Runs behind this row
- Every tool-planning trial for this model →