Data to …… trials

Run camp-tool-depth-super-002

84% PASS

What this run revealed, then every individual trial as evidence.

watch live →compare across models →
84
10 errors
Trials
84%
62/74 scored
over trials
Pass rate
12
14%
Failures
$0.1100
Run cost

Run camp-tool-depth-super-002, 1 arms: clean on none; below 97% on captokens.

Arm captokens: 60% pass rate across 10 trials.

(captokens) 4 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record f5392068 — asked "You may use at most 32 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

  • record c248372a — asked "You may use at most 36 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

Trials84 trials
Per-trial table (84 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"tool":"advance_0","args":{"service":"svc_0"}},{"tool":..."tool_plan-0.000491774/954full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000500773/973full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000392773/734full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000636771/1277full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000546772/1076full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000534773/1049full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000581772/1153full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000721910/1441full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000708907/1413full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"tool":"set_flag_0","args":{"flag":"flag_0"}},{"tool":"..."tool_plan-0.000786910/1584full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000914909/1869full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000823909/1668full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000924909/1891full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000528911/1012full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0007531051/1486full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0008321048/1662full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0008721051/1750full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"tool":"advance_0","args":{"service":"svc_0"}},{"tool":..."tool_plan-0.0008161049/1626full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0008151050/1624full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0011801048/2437full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0009061049/1826full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0008981122/1797full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0008711123/1737full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"tool":"advance_0","args":{"service":"svc_0"}},{"tool":..."tool_plan-0.0009971120/2016full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0009461122/1903full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0010351120/2101full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0010901121/2224full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0008621122/1716full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"tool":"advance_o0","args":{"service":"svc_0"}},{"tool"..."tool_plan-0.0012731275/2603full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0012991282/2658full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0013201280/2705full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0016241277/3381full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"tool":"set_flag_o0","args":{"flag":"flag_0"}},{"tool":..."tool_plan-0.0012761273/2610full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0015141276/3138full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "[{"tool":"advance_o0","args":{"service":"svc_0"}},{"tool"..." (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0012081276/2457full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0014241426/2911full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0014721426/3018full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0016771423/3473full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0018141424/3779full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0018601419/3880full transcript →
nvidia/nemotron-3-super-120b-a12b✕ERRORlengthtool_plan-0.0019141419/4000full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"tool":"adva_nce_0","args":{"service":"svc_0"}},{"tool"..."tool_plan-0.0017811424/3705full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0015591579/3183full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0017281566/3561full transcript →
nvidia/nemotron-3-super-120b-a12b✕ERRORlengthtool_plan-0.0019261571/4000full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0019101574/3964full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0018731565/3883full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0015331567/3128full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0017371573/3581full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0016071644/3279full transcript →
nvidia/nemotron-3-super-120b-a12b✕ERRORlengthtool_plan-0.0019311642/4000full transcript →
nvidia/nemotron-3-super-120b-a12b✕ERRORlengthtool_plan-0.0019311641/4000full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0016591645/3395full transcript →
nvidia/nemotron-3-super-120b-a12b✕ERRORlengthtool_plan-0.0019311643/4000full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0024261642/5099full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0024491646/5149full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0014371567/2915full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"tool":"set_-flag_0","args":{"flag":"flag_0"}},{"tool":..."tool_plan-0.0017491573/3608full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0016921559/3482full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0018041569/3729full transcript →
nvidia/nemotron-3-super-120b-a12b✕ERRORlengthtool_plan-0.0019261572/4000full transcript →
nvidia/nemotron-3-super-120b-a12b✕ERRORlengthtool_plan-0.0019261573/4000full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0014001563/2834full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0016961563/3490full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0018441562/3821full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0019051570/3954full transcript →
nvidia/nemotron-3-super-120b-a12b✕ERRORlengthtool_plan-0.0019261571/4000full transcript →
nvidia/nemotron-3-super-120b-a12b✕ERRORlengthtool_plan-0.0019261573/4000full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0019061570/3957full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0017141576/3529full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0016151571/3310full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0015441575/3150full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0017151569/3533full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0013651575/2753full transcript →
nvidia/nemotron-3-super-120b-a12b✕ERRORlengthtool_plan-0.0019261569/4000full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0017001564/3500full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0016731565/3439full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000521774/1021full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000697774/1412full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000516769/1010full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"tool":"advance_0","args":{"service":"svc_0"}},{"tool":..."tool_plan-0.000487773/944full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.000605773/1206full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"tool":"advance_0","args":{"service":"svc_0"}},{"tool":..."tool_plan-0.000554771/1094full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"tool":"advance_0","args":{"service":"svc_0"}},{"tool":..."tool_plan-0.000646773/1298full transcript →