Data to …… trials
II · Tool planning

Plans long tool sequences when tool names look alike

Nemotron 3 Ultra 550B

Finding:DEGRADES· grade A(research conclusion)

Degrades at 25-call plans: 4/7 pass (was 7/7 at 21-call plans)

Boundary 25-call plans · held through 21-call plans

Pass rate by level10 levels tested
Pass rate by level for Plans long tool sequences when tool names look alike0%30%50%75%100%Boundary: degrades at 25-call plans7-call plans7-call plans: 6/7 pass (86%)6/711-call plans11-call plans: 7/7 pass (100%)7/714-call plans14-call plans: 11/14 pass (79%)11/1416-call plans16-call plans: 5/7 pass (71%)5/718-call plans18-call plans: 13/14 pass (93%)13/1419-call plans19-call plans: 7/7 pass (100%)7/721-call plans21-call plans: 7/7 pass (100%)7/725-call plans25-call plans: 4/7 pass (57%)4/729-call plans29-call plans: 4/7 pass (57%)4/731-call plans31-call plans: 3/7 pass (43%)3/7

Dashed lines: a level at or below 30% pass counts toward BREAKS, at or below 75% toward DEGRADES. Labels above points: passed/scored.

LevelPassedPass rateNote
7-call plans6/786%
11-call plans7/7100%
14-call plans11/1479%
16-call plans5/771%
18-call plans13/1493%
19-call plans7/7100%
21-call plans7/7100%
25-call plans4/757%▲ boundary
29-call plans4/757%
31-call plans3/743%
Why it slips

The model calls a tool that does not exist -- usually a misspelled look-alike of a real tool name.

in 100% of failures (10 failures at and beyond the boundary)

Example failureat 25-call plans
Trial result: FAIL (one model answer)
FAILunknown_tool:step=10:set_dep-th_flag_9
[ {"tool": "set_flag_o0", "args": {"flag": "flag_0"}}, {"tool": "set_depth_flag_o0", "args": {"flag": "depth_flag_0"}}, {"tool": "set_depth_flag_o1", "args": {"flag": "depth_flag_1"}}, {"tool": "set_dep-th_flag_2", "args": {"flag": "depth_flag_2"}}, {"tool": "set_depth_flag_o3", "args": {"flag": "depth_flag_3"}}, {"tool": "set_dep-th_flag_4", "args": {"flag": "depth_flag_4"}}, {"tool": "set_dep-th
Open this trial →Open the run it came from →
Evidencewhat this finding rests on
  • Ground truthDeterministic simulation: the right answer is computed, not judged
  • StatisticsWilson 95% interval 25%–84%p = 4.2×10-4, before vs after the boundary (one-sided Fisher exact)Grade A: strong evidence
  • Sample size7 scored trials at the boundary: small sample (under 15)
  • Fresh-seed confirmationNot run yet
  • ReplicationNot run yet
Raw evidenceruns and trials

Back to the capability map →