Data to …… trials

Run camp-tool-depth-super-002/captokens

60% PASS

What this run revealed, then every individual trial as evidence.

10
Trials
60%
6/10 scored
over trials
Pass rate
4
40%
Failures
$0.0179
Run cost

Run camp-tool-depth-super-002/captokens: 60% pass rate on tool queries.

Of 10 scored trials (tool query kind), the model passed 60% and failed 40%.

4 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record f5392068 — asked "You may use at most 32 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

  • record c248372a — asked "You may use at most 36 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

  • record 5b377c71 — asked "You may use at most 36 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

Trials10 trials
Per-trial table (10 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0013571419/2763full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0018071571/3736full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0017191643/3528full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0018161641/3743full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0018691642/3861full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0022391572/4695full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0018771573/3892full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0014621571/2969full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0017281573/3561full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0020641569/4308full transcript →