II · Tool planning
Picks the right tool from a large catalog
Gemma 3 27B
Finding:HOLDS· grade B(research conclusion)
Holds through 66-tool catalogs: 46/56 pass across every tested level
Held through 66-tool catalogs
Pass rate by level
Dashed lines: a level at or below 30% pass counts toward BREAKS, at or below 75% toward DEGRADES. Labels above points: passed/scored.
| Level | Passed | Pass rate | Note |
|---|---|---|---|
| 8-tool catalogs | 7/7 | 100% | |
| 17-tool catalogs | 5/7 | 71% | |
| 25-tool catalogs | 6/7 | 86% | |
| 34-tool catalogs | 4/7 | 57% | |
| 43-tool catalogs | 4/7 | 57% | |
| 52-tool catalogs | 7/7 | 100% | |
| 60-tool catalogs | 7/7 | 100% | |
| 66-tool catalogs | 6/7 | 86% |
What the few failures have in common
The model calls a real tool before the step it depends on is done.
in 100% of failures (10 failures)
Evidence
- Ground truthDeterministic simulation: the right answer is computed, not judged
- StatisticsWilson 95% interval 70%–90%
- Sample size56 scored trials in total
- Fresh-seed confirmationNot run yet
- ReplicationNot run yet
Raw evidence
- Runs behind this row
- Every tool-planning trial for this model →