Data to …… trials
II · Tool planning

Plans long tool sequences when tool names look alike

Nemotron 3 Super 120B

Finding:BREAKS· grade A(research conclusion)

Breaks at 24-call plans: 8/28 pass (was 6/7 at 11-call plans)

Boundary 24-call plans · held through 11-call plans

Pass rate by level10 levels tested
Pass rate by level for Plans long tool sequences when tool names look alike0%30%50%75%100%Boundary: breaks at 24-call plans7-call plans7-call plans: 20/21 pass (95%)20/2111-call plans11-call plans: 6/7 pass (86%)6/714-call plans14-call plans: 16/22 pass (73%)16/2216-call plans16-call plans: 9/15 pass (60%)9/1518-call plans18-call plans: 16/22 pass (73%)16/2219-call plans19-call plans: 7/15 pass (47%)7/1521-call plans21-call plans: 4/7 pass (57%)4/723-call plans23-call plans: 5/7 pass (71%)5/724-call plans24-call plans: 8/28 pass (29%)8/2825-call plans25-call plans: 2/7 pass (29%)2/7

Dashed lines: a level at or below 30% pass counts toward BREAKS, at or below 75% toward DEGRADES. Labels above points: passed/scored.

LevelPassedPass rateNote
7-call plans20/2195%
11-call plans6/786%
14-call plans16/2273%
16-call plans9/1560%
18-call plans16/2273%
19-call plans7/1547%
21-call plans4/757%
23-call plans5/771%
24-call plans8/2829%▲ boundary
25-call plans2/729%
Why it breaks

The model calls a tool that does not exist -- usually a misspelled look-alike of a real tool name.

in 96% of failures (25 failures at and beyond the boundary)

Example failureat 24-call plans
Trial result: FAIL (one model answer)
FAILunknown_tool:step=10:set_dep-th_flag_9
[ {"tool":"set_flag_0","args":{"flag":"flag_0"}}, {"tool":"set_dep_th_flag_0","args":{"flag":"depth_flag_0"}}, {"tool":"set_dep-th_flag_1","args":{"flag":"depth_flag_1"}}, {"tool":"set_dep-th_flag_2","args":{"flag":"depth_flag_2"}}, {"tool":"set_depth_flag_o3","args":{"flag":"depth_flag_3"}}, {"tool":"set_dep-th_flag_4","args":{"flag":"depth_flag_4"}}, {"tool":"set_dep-th_flag_5","args":{"flag":"d
Open this trial →Open the run it came from →
Evidencewhat this finding rests on
  • Ground truthDeterministic simulation: the right answer is computed, not judged
  • StatisticsWilson 95% interval 15%–47%p = 1.5×10-7, before vs after the boundary (one-sided Fisher exact)Grade A: strong evidence
  • Sample size28 scored trials at the boundary
  • Fresh-seed confirmationNot run yet
  • ReplicationNot run yet
Raw evidenceruns and trials

Back to the capability map →