Data to …… trials

Run camp-tool_planning-nvidia-nemotron-3-nano-30b-a3b-001

98% PASS

What this run revealed, then every individual trial as evidence.

40
Trials
98%
39/40 scored
over trials
Pass rate
1
2%
Failures
$0.0057
Run cost

Run camp-tool_planning-nvidia-nemotron-3-nano-30b-a3b-001: passed essentially everything (98%) -- this test did not challenge the model.

Of 40 scored trials (tool query kind), the model passed 98%. That's saturation: the difficulty range probed here isn't finding this model's limits. Consider a harder preset (breaking_point or adversarial_belief) or a wider dimension range.

1 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record 78829e71 — asked "You may use at most 5 tool calls."
    expected: (unknown)
    actual: [{"tool":"provision","args":{"service":"svc_0"}}]

    answer did not match the verified ground truth

Trials40 trials
Per-trial table (40 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "[{"tool":"advance_0","args":{"service":"svc_0"}}]"tool_plan-0.000072387/264full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "[{"tool":"advance_0","args":{"service":"svc_0"}}]"tool_plan-0.000087387/339full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "[{"tool":"advance_0","args":{"service":"svc_0"}}]"tool_plan-0.000090384/355full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "[{"tool":"advance_0","args":{"service":"svc_0"}}]"tool_plan-0.000090387/354full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "[{"tool":"advance_0","args":{"service":"svc_0"}}]"tool_plan-0.000123385/518full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000166434/721full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "[{"tool":"advance_0","args":{"service":"svc_0"}},{"tool":..."tool_plan-0.000126433/524full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000122431/504full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000144435/611full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000149431/637full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000213525/935full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000209524/915full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000182525/779full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000192526/829full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000239526/1065full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "[{"tool":"advance_0","args":{"service":"svc_0"}},{"tool":..."tool_plan-0.000227525/1003full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000158527/659full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000203525/886full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000209523/914full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000175527/741full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "[{"tool":"advance_0","args":{"service":"svc_0"}},{"tool":..."tool_plan-0.000224522/990full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000155526/643full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "[{"tool":"advance_0","args":{"service":"svc_0"}},{"tool":..."tool_plan-0.000248525/1111full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000131526/524full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000197527/853full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "[{"tool":"advance_0","args":{"service":"svc_0"}}]"tool_plan-0.000076387/281full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "[{"tool":"advance_0","args":{"service":"svc_0"}}]"tool_plan-0.000086386/333full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "[{"tool":"advance_0","args":{"service":"svc_0"}}]"tool_plan-0.000088384/343full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "[{"tool":"advance_0","args":{"service":"svc_0"}}]"tool_plan-0.000092384/365full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "[{"tool":"provision","args":{"service":"svc_0"}}]"unverifiedtool_plan-0.000103425/410full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "[{"tool":"advance_0","args":{"service":"svc_0"}}]"tool_plan-0.000120383/502full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "[{"tool":"advance_0","args":{"service":"svc_0"}}]"tool_plan-0.000076425/273full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "[{"tool":"advance_0","args":{"service":"svc_0"}}]"tool_plan-0.000095426/366full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "[{"tool":"advance_0","args":{"service":"svc_0"}}]"tool_plan-0.000103424/407full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "[{"tool":"advance_0","args":{"service":"svc_0"}}]"tool_plan-0.000115428/469full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "[{"tool":"advance_0","args":{"service":"svc_0"}}]"tool_plan-0.000118469/475full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "[{"tool":"advance_0","args":{"service":"svc_0"}}]"tool_plan-0.000116471/461full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "[{"tool":"advance_0","args":{"service":"svc_0"}}]"tool_plan-0.000087472/318full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "[{"tool":"advance_0","args":{"service":"svc_0"}}]"tool_plan-0.000119471/476full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "[{"tool":"advance_0","args":{"service":"svc_0"}}]"tool_plan-0.000128469/525full transcript →