Data to …… trials

Run camp-tool-depth-super-003/k15

56% PASS

What this run revealed, then every individual trial as evidence.

watch live →compare across models →
32
Trials
56%
18/32 scored
over trials
Pass rate
14
44%
Failures
$0.0519
Run cost

Run camp-tool-depth-super-003/k15: 56% pass rate on tool queries.

Of 32 scored trials (tool query kind), the model passed 56% and failed 44%.

14 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record e63588e2 — asked "You may use at most 28 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

  • record 9de901cf — asked "You may use at most 28 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

  • record 2036ac31 — asked "You may use at most 32 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

Trials32 trials
Per-trial table (32 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0011741281/2382full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0013541278/2782full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0014211281/2929full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0015981271/3326full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0016021281/3332full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0017351278/3629full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0011291419/2256full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"tool":"adva-nce_0","args":{"service":"svc_0"}},{"tool"..."tool_plan-0.0014741280/3048full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0010061425/1983full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0015201279/3151full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0015361422/3161full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0019171420/4008full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0012791424/2588full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0015941420/3290full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0014611419/2994full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0018001417/3747full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0014251577/2887full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0015511563/3169full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0021351564/4467full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0015461573/3157full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0013871570/2803full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0016361569/3357full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0023341567/4909full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0012881644/2571full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0019011561/3948full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0017221638/3536full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0021761644/4544full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0018101649/3730full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0021181651/4414full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "]"tool_plan-0.0018481643/3814full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0015841655/3226full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0018721651/3866full transcript →