Data to …… trials

Run camp-interactive-lookalike-super-001

85% PASS

What this run revealed, then every individual trial as evidence.

25
5 errors
Trials
85%
17/20 scored
over trials
Pass rate
3
12%
Failures
$0.2089
Run cost

Run camp-interactive-lookalike-super-001: 85% pass rate on interactive queries.

Of 20 scored trials (interactive query kind), the model passed 85% and failed 15%.

5 additional trial(s) ended in ERROR (infra failures, cost cap, or cancellation) and are excluded from the pass rate above.

3 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Sharp boundary detected on: final_distance (high-end pass rate 0%).

Diffuse weakness (below 90% but no outright flip) on: first_deviation_step (pass rate 71% at the high end), n_errors (pass rate 57% at the high end).

Notable failures

  • record b0d02a0e — asked "You may make at most 30 tool calls."
    expected: (unknown)
    actual: [{"step": 1, "tool": "set_-flag_0", "args": {"flag": "flag_0"}, "ok": true, "error_kind": null, "...

    answer did not match the verified ground truth

  • record 10bca94a — asked "You may make at most 34 tool calls."
    expected: (unknown)
    actual: [{"step": 1, "tool": "set_-flag_0", "args": {"flag": "flag_0"}, "ok": true, "error_kind": null, "...

    answer did not match the verified ground truth

  • record 43d30d0e — asked "You may make at most 44 tool calls."
    expected: (unknown)
    actual: [{"step": 1, "tool": "advance_o0", "args": {"service": "svc_0"}, "ok": true, "error_kind": null, ...

    answer did not match the verified ground truth

Trials25 trials
Per-trial table (25 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"step": 1, "tool": "set_flag_o0", "args": {"flag": "fla..."interactive-0.00625756449/3869full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"step": 1, "tool": "build_art_0", "args": {"artifact": ..."interactive-0.00634956413/4079full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"step": 1, "tool": "set_-flag_0", "args": {"flag": "fla..."interactive-0.00487848747/2173full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"step": 1, "tool": "set_-flag_0", "args": {"flag": "fla..."interactive-0.00494648887/2300full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "[{"step": 1, "tool": "set_-flag_0", "args": {"flag": "fla..."unverifiedinteractive-0.014489110884/12484full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"step": 1, "tool": "adva_nce_0", "args": {"service": "s..."interactive-0.00682268034/3066full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"step": 1, "tool": "build-_art_0", "args": {"artifact":..."interactive-0.00669663650/3564full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag..."interactive-0.00731468119/4144full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "[{"step": 1, "tool": "set_-flag_0", "args": {"flag": "fla..."unverifiedinteractive-0.00399324451/4527full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"step": 1, "tool": "set_-flag_0", "args": {"flag": "fla..."interactive-0.00670863921/3544full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"step": 1, "tool": "set_-flag_0", "args": {"flag": "fla..."interactive-0.00902292520/3601full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"step": 1, "tool": "set_flag_o0", "args": {"flag": "fla..."interactive-0.00982397717/4458full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"step": 1, "tool": "set_-flag_0", "args": {"flag": "fla..."interactive-0.013807130797/7430full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag..."interactive-0.00927992506/4174full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"step": 1, "tool": "set_flag_o0", "args": {"flag": "fla..."interactive-0.00910892617/3775full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"step": 1, "tool": "set_flag_o0", "args": {"flag": "fla..."interactive-0.011382121770/3645full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag..."interactive-0.011624121690/4198full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"step": 1, "tool": "set_flag_o0", "args": {"flag": "fla..."interactive-0.014267140627/6705full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "[{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag..."interactive-0.011479115902/4904full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "[{"step": 1, "tool": "advance_o0", "args": {"service": "s..."unverifiedinteractive-0.026648261908/12657full transcript →
nvidia/nemotron-3-super-120b-a12b✕ERRORtool_callsinteractive-0.00515753060/2028full transcript →
nvidia/nemotron-3-super-120b-a12b✕ERRORinfra failureinteractive-0.0000000/0full transcript →
nvidia/nemotron-3-super-120b-a12b✕ERRORinfra failureinteractive-0.0000000/0full transcript →
nvidia/nemotron-3-super-120b-a12b✕ERRORtool_callsinteractive-0.00460740905/2966full transcript →
nvidia/nemotron-3-super-120b-a12b✕ERRORtool_callsinteractive-0.00426340905/2202full transcript →