Run camp-tool-depth-ultra-003
95% PASSWhat this run revealed, then every individual trial as evidence.
56
Trials
95%
53/56 scored
over trials
Pass rate
3
Failures
$0.2359
Run cost
Run camp-tool-depth-ultra-003: 95% pass rate on tool queries.
Of 56 scored trials (tool query kind), the model passed 95% and failed 5%.
3 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).
Diffuse weakness (below 90% but no outright flip) on: catalog_size (pass rate 86% at the high end), chain_length (pass rate 86% at the high end), id_collisions_on (pass rate 89% at the high end), realized_plan_depth (pass rate 86% at the high end).
Notable failures
- record c3190c93 expected: (unknown)actual: ]
- record 9fe6a4d9 expected: (unknown)actual: ]
- record 4a2e0ba7 expected: (unknown)actual: ]
Trials
Per-trial table
| model | verdict | kind | difficulty | cost ($) | tokens in/out | |
|---|---|---|---|---|---|---|
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.001657 | 771/578 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.002025 | 773/745 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.002219 | 772/833 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.002396 | 774/913 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.002168 | 773/810 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.002323 | 773/880 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.002051 | 772/757 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.002644 | 910/995 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.002512 | 909/935 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.002597 | 909/974 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.002895 | 910/1109 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.003081 | 909/1194 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.003133 | 907/1218 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.003888 | 911/1560 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.002425 | 1049/864 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.003046 | 1050/1146 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "[{"tool": "advance_0", "args": {"service": "svc_0"}}, {"t..." | tool_plan | - | 0.003195 | 1048/1214 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.003185 | 1051/1209 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.003650 | 1048/1421 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.003991 | 1051/1575 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.006354 | 1049/2650 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.002911 | 1122/1068 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.003036 | 1122/1125 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.003092 | 1120/1151 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.003539 | 1121/1354 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.003375 | 1123/1279 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.003768 | 1120/1458 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.005680 | 1122/2327 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.003931 | 1276/1497 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.004114 | 1275/1580 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.004319 | 1277/1673 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.004781 | 1282/1882 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.005295 | 1280/2116 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.006244 | 1273/2549 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.005938 | 1276/2409 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ?FAILanswered "]" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.003948 | 1419/1472 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.004110 | 1426/1544 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.004168 | 1423/1571 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ?FAILanswered "]" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.004300 | 1419/1632 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.005993 | 1426/2400 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.006896 | 1424/2811 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.006034 | 1424/2419 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.004248 | 1574/1573 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.004899 | 1579/1868 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ?FAILanswered "]" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.005064 | 1566/1946 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.005814 | 1573/2285 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.006146 | 1565/2438 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.007458 | 1571/3033 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.004997 | 1567/1915 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "[{"tool": "set_flag_0", "args": {"flag": "flag_0"}}, {"to..." | tool_plan | - | 0.003771 | 1642/1341 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.004222 | 1641/1546 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.004272 | 1645/1568 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.006235 | 1642/2461 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.006241 | 1644/2463 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.005370 | 1646/2067 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "]" | tool_plan | - | 0.010266 | 1643/4293 | full transcript → |