Data to …… trials

Run camp-tool-deep-super-001

60% PASS

What this run revealed, then every individual trial as evidence.

watch live →compare across models →
91
Trials
60%
55/91 scored
over trials
Pass rate
36
40%
Failures
$0.1513
Run cost

Run camp-tool-deep-super-001: 60% pass rate on tool queries.

Of 91 scored trials (tool query kind), the model passed 60% and failed 40%.

36 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record bbb384f7 — asked "You may use at most 22 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

  • record 62c1598f — asked "You may use at most 28 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

  • record 2459fd68 — asked "You may use at most 28 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

Trials91 trials
Per-trial table (91 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"tool":"advance_o0","args":{"service":"svc_0"}},{"tool"..."tool_plan-0.000527783/1033full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000557784/1099full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"tool":"set_flag_0","args":{"flag":"flag_0"}},{"tool":"..."tool_plan-0.000647786/1298full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"tool":"adva-nce_0","args":{"service":"svc_0"}},{"tool"..."tool_plan-0.000615783/1227full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"tool":"adva_nce_0","args":{"service":"svc_0"}},{"tool"..."tool_plan-0.000716788/1452full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000733787/1489full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000627784/1253full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0008801069/1765full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0010071068/2048full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0010601060/2167full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"tool":"adva_nce_0","args":{"service":"svc_0"}},{"tool"..."tool_plan-0.0012191072/2518full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0012061064/2490full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0013521067/2815full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0013781067/2872full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0013781280/2835full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0015171272/3145full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0014081280/2901full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0014731277/3046full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0016741278/3492full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"tool":"set_flag_0","args":{"flag":"flag_0"}},{"tool":"..."tool_plan-0.0017081281/3568full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0009411279/1864full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "[{"tool":"set_-flag_0","args":{"flag":"flag_0"}},{"tool":..." (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0012901571/2588full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0014421572/2924full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0016181574/3316full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0016371573/3358full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0022211568/4657full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0020981568/4383full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0016991559/3498full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0017491790/3569full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0017591788/3590full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0018881782/3878full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0020971795/4340full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0021981796/4566full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0025331796/5309full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0023781794/4966full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0013732098/2678full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0020822100/4253full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0025952097/5393full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0028152091/5883full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0029832098/6256full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0036932103/7833full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0024262102/5017full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0019241934/3931full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0019771945/4048full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0022421954/4635full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0022311940/4613full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0024821945/5169full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0030901926/6524full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0021121941/4348full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0016442018/3294full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0019122011/3891full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0018842018/3827full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0023922028/4955full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0025962020/5410full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0025582014/5326full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0024692018/5128full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0015912009/3179full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0018332020/3714full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0018662017/3788full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0020572022/4212full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0022162025/4565full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0028502018/5975full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0016632026/3336full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0020162022/4120full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0019982020/4082full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0021562020/4432full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0024222018/5024full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0019952015/4076full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0027192014/5685full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0020252021/4141full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000564780/1115full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000719781/1458full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.000613788/1223full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000643786/1290full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000644784/1292full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000553781/1090full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000768786/1567full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"tool":"adva-nce_0","args":{"service":"svc_0"}},{"tool"..."tool_plan-0.000641785/1284full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000716783/1453full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"tool":"advance_o0","args":{"service":"svc_0"}},{"tool"..."tool_plan-0.000642782/1288full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"tool":"adva-nce_0","args":{"service":"svc_0"}},{"tool"..."tool_plan-0.000439779/837full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000707783/1431full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000570784/1128full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000631780/1263full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0019422021/3956full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0020312010/4155full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0022052016/4542full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0025272019/5256full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0017262020/3476full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0022212012/4578full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0017062021/3432full transcript →