Data to …… trials

Run camp-tool-depth-nano-001

66% PASS

What this run revealed, then every individual trial as evidence.

watch live →compare across models →
44
Trials
66%
29/44 scored
over trials
Pass rate
15
34%
Failures
$0.0329
Run cost

Run camp-tool-depth-nano-001: 66% pass rate on tool queries.

Of 44 scored trials (tool query kind), the model passed 66% and failed 34%.

15 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Sharp boundary detected on: catalog_size (high-end pass rate 12%), chain_length (high-end pass rate 12%), realized_plan_depth (high-end pass rate 12%).

Notable failures

  • record 7ecb774c — asked "You may use at most 24 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

  • record 1a00508c — asked "You may use at most 28 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

  • record 8bebb59c — asked "You may use at most 28 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

Trials44 trials
Per-trial table (44 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000386773/1738full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000483773/2220full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000444771/2029full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000338774/1498full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000269773/1150full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000429772/1953full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000250772/1055full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000765910/3599full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000625909/2899full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000478907/2165full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000499910/2266full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000412909/1835full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000606909/2805full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000645911/2999full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0008061050/3767full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0006101048/2790full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0007091051/3280full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0005991049/2733full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0008161048/3819full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0007461051/3468full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "[{"tool":"advance_0","args":{"service":"svc_0"}},{"tool":..."tool_plan-0.0008031049/3753full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0008991123/4213full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0006731121/3083full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0005361120/2398full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0008171122/3806full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0006911120/3176full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=0)unverifiedtool_plan-0.0007511122/3475full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "[{"tool":"advance_0","args":{"service":"svc_0"}},{"tool":..."tool_plan-0.0013461122/6448full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0009001277/4183full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0008131273/3746full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0013721282/6540full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0009071276/4218full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "[{"tool":"set_-flag_0","args":{"flag":"flag_0"}},{"tool":..." (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0014041280/6702full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0011761275/5559full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0009271276/4314full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0008431208/3912full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0006781203/3091full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0012901215/6148full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0006401206/2898full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0008381209/3890full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0010231206/4813full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0010111204/4753full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0008261192/3833full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0008381191/3894full transcript →