Data to …… trials

Run camp-tool-depth-super-003

86% PASS

What this run revealed, then every individual trial as evidence.

watch live →compare across models →
56
Trials
86%
48/56 scored
over trials
Pass rate
8
14%
Failures
$0.0656
Run cost

Run camp-tool-depth-super-003, 1 arms: clean on none; below 97% on k15.

Arm k15: 56% pass rate across 32 trials.

(k15) 14 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record e63588e2 — asked "You may use at most 28 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

  • record 9de901cf — asked "You may use at most 28 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

Trials56 trials
Per-trial table (56 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"tool":"build_art_0","args":{"artifact":"art_0"}},{"too..."tool_plan-0.000479773/927full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000454771/872full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000465773/897full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000525772/1030full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000571773/1131full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000567774/1122full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000510772/997full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000487907/922full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"tool":"advance_0","args":{"service":"svc_0"}},{"tool":..."tool_plan-0.000648909/1279full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000660910/1305full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000712910/1420full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000729909/1458full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"tool":"advance_0","args":{"service":"svc_0"}},{"tool":..."tool_plan-0.000976909/2007full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"tool":"set_flag_0","args":{"flag":"flag_0"}},{"tool":"..."tool_plan-0.000636911/1252full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0005951051/1135full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0007931048/1577full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0008441048/1690full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0008191049/1633full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0008401050/1681full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0009181051/1853full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0006971049/1363full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0008661120/1725full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0009421122/1893full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0009031121/1807full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0010571120/2149full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0009731122/1962full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0010961123/2237full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0010301122/2090full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0011731275/2381full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0012131273/2470full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0012471280/2543full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0013161276/2697full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0015601277/3239full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "[{"tool":"adva_nce_0","args":{"service":"svc_0"}},{"tool"..." (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0015631282/3246full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0011621276/2355full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0012791419/2590full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0013631426/2776full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0013861419/2828full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0015721424/3241full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0015001426/3080full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0016521423/3418full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0014251424/2913full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0011801571/2343full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0013461566/2712full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0016121565/3305full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0019151574/3976full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0019641573/4085full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0018391579/3805full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0016791567/3453full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0016341643/3338full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0016241641/3318full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0018711644/3865full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "[{"tool":"set_flag_0","args":{"flag":"flag_0"}},{"tool":"..." (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0019841645/4116full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0020681642/4304full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0022641642/4739full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0024151646/5075full transcript →