Data to …… trials

Run camp-tool-deep-ultra-001

75% PASS

What this run revealed, then every individual trial as evidence.

watch live →compare across models →
56
Trials
75%
42/56 scored
over trials
Pass rate
14
25%
Failures
$0.3314
Run cost

Run camp-tool-deep-ultra-001: 75% pass rate on tool queries.

Of 56 scored trials (tool query kind), the model passed 75% and failed 25%.

14 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record a99706c8 — asked "You may use at most 14 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

  • record ff8a5f7d — asked "You may use at most 28 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

  • record 5074079d — asked "You may use at most 28 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

Trials56 trials
Per-trial table (56 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.001683783/587full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.002384786/905full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.002924787/1150full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.003485783/1406full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.003765784/1533full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.004182788/1722full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.002733784/1064full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0030181064/1130full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0030491060/1145full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0034511067/1326full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0041471069/1642full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0039971072/1573full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0056181068/2311full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0033341067/1273full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0043711280/1696full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0036171272/1355full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0039251280/1493full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0043871281/1703full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0046011277/1801full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0057871278/2340full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0035191279/1309full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0038241571/1381full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0043151572/1604full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0055801568/2180full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0054291574/2110full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0055211568/2153full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0065261573/2609full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0048651559/1857full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0047831796/1766full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0052471782/1980full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0053291788/2016full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0054291790/2061full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0063601795/2483full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0060661796/2349full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0085901794/3497full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0057402091/2134full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0056952098/2112full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0066372103/2539full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "[{"tool":"set_flag_0","args":{"flag":"flag_0"}},{"tool":"..."tool_plan-0.0063912098/2428full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0099092100/4027full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0103902097/4246full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0072402102/2813full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0069742402/2624full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0078462395/3022full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0083962396/3272full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0083492397/3250full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0096862383/3861full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0087902401/3450full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0102332390/4108full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0088182548/3429full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0082892546/3189full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0092272548/3615full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0091372536/3577full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0110582549/4447full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0102622537/4088full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "]"tool_plan-0.0065402551/2393full transcript →