Data to …… trials

Run camp-tool-depth-nano-003

65% PASS

What this run revealed, then every individual trial as evidence.

watch live →compare across models →
98
Trials
65%
64/98 scored
over trials
Pass rate
34
35%
Failures
$0.0711
Run cost

Run camp-tool-depth-nano-003: 65% pass rate on tool queries.

Of 98 scored trials (tool query kind), the model passed 65% and failed 35%.

34 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record ecf15f86 — asked "You may use at most 14 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

  • record e7a3787b — asked "You may use at most 18 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

  • record 482e710a — asked "You may use at most 28 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

Trials98 trials
Per-trial table (98 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000234774/978full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000310773/1358full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000325771/1434full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000344772/1529full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000375773/1683full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000476773/2189full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=0)unverifiedtool_plan-0.000299772/1300full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000542909/2485full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000576909/2652full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000640910/2974full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=0)unverifiedtool_plan-0.000648910/3013full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000677907/3159full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000684909/3193full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000517911/2355full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0006511049/2994full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0006941051/3206full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0007751048/3614full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0007961051/3719full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0008631050/4051full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0009791048/4631full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0005111049/2294full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0005001120/2222full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0007581122/3508full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0008561121/3999full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0013091120/6264full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "[{"tool":"build_art_0","args":{"artifact":"art_0"}},{"too..."tool_plan-0.0010391122/4914full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0005391123/2414full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0006361122/2898full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0009991275/4675full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0010251276/4808full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0010351273/4855full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0010831282/5092full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0011471277/5418full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0010991276/5178full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "[{"tool":"adva-nce_0","args":{"service":"svc_0"}},{"tool"..." (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0009421280/4388full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0008581203/3990full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0009591215/4489full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0009851209/4621full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0010731206/5065full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0011461208/5427full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0012191206/5793full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0012981204/6191full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0007171193/3286full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0007341189/3374full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0008861191/4134full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0008911189/4158full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0010121190/4760full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0010281192/4840full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "[{"tool":"set_flag_0","args":{"flag":"flag_0"}},{"tool":"..."tool_plan-0.0008631192/4016full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0008291200/3847full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0009441212/4416full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0009501210/4449full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0010211217/4799full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0011281212/5336full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0009381216/4384full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0012441213/5915full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0009391214/4391full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0013971206/6681full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0010791211/5094full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0017521209/8456full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0009441206/4420full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0012321206/5860full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0009241214/4316full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000289774/1251full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000359774/1603full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000316769/1389full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=0)unverifiedtool_plan-0.000448773/2046full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000318773/1396full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000313771/1372full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000312773/1366full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0007371190/3388full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0007001190/3201full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0007591192/3497full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0009681192/4542full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "[{"tool":"set_flag_0","args":{"flag":"flag_0"}},{"tool":"..."tool_plan-0.0015551192/7476full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0007901190/3655full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.0010051191/4728full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000362782/1617full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000404783/1824full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000395782/1778full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000340786/1501full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000353780/1569full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000307779/1338full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000443780/2021full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.000443779/2022full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.000434777/1974full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000323780/1418full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000459777/2100full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000348783/1546full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.000420784/1906full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.000413785/1870full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.000551787/2559full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.000415780/1878full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000464783/2123full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000374786/1674full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "]"tool_plan-0.000393781/1772full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.000393788/1767full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.000354786/1573full transcript →