Data to …… trials

Run camp-propose-toolworld-001

79% PASS

What this run revealed, then every individual trial as evidence.

watch live →compare across models →
40
7 errors
Trials
79%
26/33 scored
over trials
Pass rate
7
18%
Failures
$0.0194
Run cost

Run camp-propose-toolworld-001, 2 arms: clean on none; below 97% on control, round3.

Arm control: 79% pass rate across 20 trials.

(control) 3 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Arm round3: 79% pass rate across 20 trials.

(round3) 4 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record 79dac819 — asked "You may use at most 20 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

  • record 3628cbcd — asked "You may use at most 6 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

  • record afad433b — asked "You may use at most 8 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

Trials40 trials
Per-trial table (40 trials -- prompt, reasoning, verdict, answer key)
modelstageverdictkinddifficultycost ($)tokens in/out
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bcontrol?FAILanswered "]" (world seeded for: poison_traps=0)unverifiedtool_plan-0.000598884/2770full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bcontrol✓PASSanswered "]"tool_plan-0.000769893/3620full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bcontrol✓PASSanswered "]"tool_plan-0.000673897/3139full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bcontrol✓PASSanswered "]"tool_plan-0.000440892/1978full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bcontrol✓PASSanswered "]"tool_plan-0.000636889/2957full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bcontrol✕ERRORlengthtool_plan-0.000841827/4000full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bcontrol✕ERRORlengthtool_plan-0.000842831/4000full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bcontrol✕ERRORlengthtool_plan-0.000841824/4000full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bcontrol✕ERRORlengthtool_plan-0.000841819/4000full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bcontrol✕ERRORlengthtool_plan-0.000843852/4000full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bcontrol✓PASSanswered "]"tool_plan-0.000314964/1330full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bcontrol✕ERRORlengthtool_plan-0.000348950/1500full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bcontrol✓PASSanswered "]"tool_plan-0.000323962/1376full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bcontrol✓PASSanswered "]"tool_plan-0.000334955/1433full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bcontrol✓PASSanswered "]"tool_plan-0.000316953/1342full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bcontrol✓PASSanswered "]"tool_plan-0.000320809/1400full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bcontrol?FAILanswered "]" (world seeded for: poison_traps=0)unverifiedtool_plan-0.000304816/1316full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bcontrol✓PASSanswered "]"tool_plan-0.000340807/1500full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bcontrol✓PASSanswered "]"tool_plan-0.000277805/1186full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bcontrol?FAILanswered "]" (world seeded for: poison_traps=0)unverifiedtool_plan-0.000276806/1180full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bround3✓PASSanswered "]"tool_plan-0.0006071045/2772full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bround3✓PASSanswered "[{"tool":"build_art_0","args":{"artifact":"art_0"}},{"too..."tool_plan-0.0007341053/3406full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bround3✓PASSanswered "]"tool_plan-0.0005831050/2653full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bround3✓PASSanswered "]"tool_plan-0.0004561040/2019full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bround3✓PASSanswered "]"tool_plan-0.0005691047/2583full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bround3✓PASSanswered "[{"tool":"advance_o0","args":{"service":"svc_0"}},{"tool"..."tool_plan-0.000200507/875full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bround3✓PASSanswered "]"tool_plan-0.000157521/653full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bround3?FAILanswered "]" (world seeded for: poison_traps=0)unverifiedtool_plan-0.000263510/1186full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bround3✓PASSanswered "]"tool_plan-0.000208508/914full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bround3✓PASSanswered "]"tool_plan-0.000154515/639full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bround3✓PASSanswered "]"tool_plan-0.0005561069/2515full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bround3✓PASSanswered "]"tool_plan-0.0008311054/3890full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bround3✓PASSanswered "]"tool_plan-0.0003901053/1688full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bround3✓PASSanswered "]"tool_plan-0.0007071067/3270full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bround3✓PASSanswered "]"tool_plan-0.0005661073/2564full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bround3✓PASSanswered "]"tool_plan-0.000592813/2756full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bround3?FAILanswered "]" (world seeded for: poison_traps=0)unverifiedtool_plan-0.000459814/2093full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bround3✕ERRORlengthtool_plan-0.000341823/1500full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bround3?FAILanswered "]" (world seeded for: poison_traps=0)unverifiedtool_plan-0.000279810/1193full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3Bround3?FAILanswered "]" (world seeded for: poison_traps=0)unverifiedtool_plan-0.000313807/1361full transcript →