II · Tool planning

Plans long tool sequences when tool names look alike

Nemotron 3 Nano 30B

Finding:BREAKS· grade B(research conclusion)

Breaks at 13-call plans: 5/21 pass (already only 15/28 at 7-call plans)

Boundary 13-call plans

Pass rate by level3 levels tested

Dashed lines: a level at or below 30% pass counts toward BREAKS, at or below 75% toward DEGRADES. Labels above points: passed/scored.

Level	Passed	Pass rate	Note
7-call plans	15/28	54%
13-call plans	5/21	24%	▲ boundary
14-call plans	0/7	0%

Why it breaks

The model calls a tool that does not exist -- usually a misspelled look-alike of a real tool name.

Example failureat 13-call plans

Trial result: FAIL (one model answer)

FAILunknown_tool:step=2:adva_nce_2

[ {"tool": "adva_nce_0", "args": {"service": "svc_0"}}, {"tool": "advance_o1", "args": {"service": "svc_0"}}, {"tool": "adva_nce_2", "args": {"service": "svc_0"}}, {"tool": "build_art_o0", "args": {"artifact": "art_0"}}, {"tool": "publish_art_o0", "args": {"artifact": "art_0"}}, {"tool": "set_flag_o0", "args": {"flag": "flag_0"}}, {"tool": "set_depth_flag_o0", "args": {"flag": "depth_flag_0"}}, {"

Open this trial →Open the run it came from →

Evidencewhat this finding rests on

Ground truthDeterministic simulation: the right answer is computed, not judged
StatisticsWilson 95% interval 11%–45%Grade B: moderate evidence
Sample size21 scored trials at the boundary
Fresh-seed confirmationNot run yet
ReplicationNot run yet

Raw evidenceruns and trials

Runs behind this row
- camp-tool-deep-nano-001 7 trials
- camp-tool-depth-nano-003 49 trials
Every tool-planning trial for this model → broader than this row
The example failure, in full →

Back to the capability map →