Data to …… trials

Run camp-tool-depth-ultra-002

91% PASS

What this run revealed, then every individual trial as evidence.

watch live →compare across models →
56
Trials
91%
51/56 scored
over trials
Pass rate
5
9%
Failures
$0.2320
Run cost

Run camp-tool-depth-ultra-002, 1 arms: clean on none; below 97% on topup15.

Arm topup15: 86% pass rate across 7 trials.

(topup15) 1 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record 9623c7da — asked "You may use at most 36 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

Trials56 trials
Per-trial table (56 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.001657771/578full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.002025773/745full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.002219772/833full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.002396774/913full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.002168773/810full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.002323773/880full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.002051772/757full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.002351909/862full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.002660907/1003full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.002806909/1069full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.002910910/1116full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.003110909/1207full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.003990910/1607full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.003888911/1560full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0024251049/864full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0030461050/1146full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "[{"tool": "advance_0", "args": {"service": "svc_0"}}, {"t..."tool_plan-0.0031951048/1214full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0033201048/1271full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0039691051/1565full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0045761051/1841full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0045731049/1840full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0030361122/1125full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0032081122/1203full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0031601120/1182full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0035091121/1340full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0033751123/1279full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0037681120/1458full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0056801122/2327full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0044441280/1729full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0036331282/1360full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0034851276/1294full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0037991275/1437full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0046621273/1830full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0041611277/1601full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0035791276/1337full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0040801419/1532full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0045551423/1747full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0046911426/1808full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0047911424/1854full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0053221426/2095full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0062051419/2498full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0069781424/2848full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0051571579/1985full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0050321571/1930full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0052091566/2012full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0058221573/2289full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0063531574/2530full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0065931565/2641full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0053511567/2076full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0046851644/1756full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0048081643/1812full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0050201641/1909full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0056171642/2180full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0073371645/2961full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "[{"tool": "set_-flag_0", "args": {"flag": "flag_0"}}, {"t..."tool_plan-0.0080941642/3306full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0051421646/1963full transcript →