III · Interactive execution

Executes step by step when tool names look alike (sees each error)

Nemotron 3 Nano 30B

Finding:BREAKS· grade C(research conclusion)

Breaks at 11-call plans: 1/5 pass (was 4/5 at 7-call plans)

Boundary 11-call plans · held through 7-call plans

Pass rate by level4 levels tested

Dashed lines: a level at or below 30% pass counts toward BREAKS, at or below 75% toward DEGRADES. Labels above points: passed/scored.

Level	Passed	Pass rate	Note
7-call plans	4/5	80%	5 errored (excluded)
9-call plans	22/36	61%	4 errored (excluded)
10-call plans	2/5	40%
11-call plans	1/5	20%	▲ boundary

Why it breaks

The model stops acting before the goal is reached.

Example failureat 11-call plans

Trial result: FAIL (one model answer)

FAILstopped_short

[{"step": 1, "tool": "set_flag_o0", "args": {"flag": "flag_0"}, "ok": true, "error_kind": null, "result": {"status": "ok"}, "distance_after": 10}, {"step": 2, "tool": "set_dep-th_flag_0", "args": {"flag": "depth_flag_0"}, "ok": true, "error_kind": null, "result": {"status": "ok"}, "distance_after": 9}, {"step": 3, "tool": "build_art_o0", "args": {"artifact": "art_0"}, "ok": true, "error_kind": nul

Open this trial →Open the run it came from →

Evidencewhat this finding rests on

Ground truthDeterministic simulation: the right answer is computed, not judged
StatisticsWilson 95% interval 4%–62%p = 0.103, before vs after the boundary (one-sided Fisher exact)Grade C: thin evidence
Sample size5 scored trials at the boundary: small sample (under 15)
Fresh-seed confirmationNot run yet
ReplicationNot run yet

Raw evidenceruns and trials

Runs behind this row
- camp-interactive-lookalike-nano-001 45 trials
- camp-interactive-lookalike-nano-002 15 trials
Every interactive-execution trial for this model → broader than this row
The example failure, in full →

Back to the capability map →