Data to …… trials

Run camp-ticket-length-gemma4qat-001/ablate

56% PASS

What this run revealed, then every individual trial as evidence.

watch live →compare across models →
54
Trials
56%
30/54 scored
over trials
Pass rate
24
44%
Failures
$0.0420
Run cost

Run camp-ticket-length-gemma4qat-001/ablate: 56% pass rate on interactive queries.

Of 54 scored trials (interactive query kind), the model passed 56% and failed 44%.

24 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Sharp boundary detected on: final_distance (high-end pass rate 0%).

Notable failures

  • record 0dd95acc — asked "You may make at most 22 tool calls."
    expected: (unknown)
    actual: [{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag_0"}, "ok": true, "error_kind": null, "r...

    answer did not match the verified ground truth

  • record 59d124a7 — asked "You may make at most 22 tool calls."
    expected: (unknown)
    actual: [{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag_0"}, "ok": true, "error_kind": null, "r...

    answer did not match the verified ground truth

  • record f3efb2d7 — asked "You may make at most 22 tool calls."
    expected: (unknown)
    actual: [{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag_0"}, "ok": true, "error_kind": null, "r...

    answer did not match the verified ground truth

Trials54 trials
Per-trial table (54 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag..."unverifiedinteractive-0.0007015350/554full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag..."unverifiedinteractive-0.0007015350/554full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag..."unverifiedinteractive-0.0007015350/554full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag..."unverifiedinteractive-0.0007015350/554full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag..."unverifiedinteractive-0.0007015350/554full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag..."unverifiedinteractive-0.0007015350/554full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag..."interactive-0.0007925592/777full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag..."interactive-0.0007925592/777full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag..."interactive-0.0007925592/777full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag..."interactive-0.0007925592/777full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag..."interactive-0.0007925592/777full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag..."interactive-0.0007925592/777full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag..."unverifiedinteractive-0.0007055386/554full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag..."unverifiedinteractive-0.0007055386/554full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag..."unverifiedinteractive-0.0007055386/554full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag..."unverifiedinteractive-0.0007055386/554full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag..."unverifiedinteractive-0.0007055386/554full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag..."unverifiedinteractive-0.0007055386/554full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[]"unverifiedinteractive-0.0002302245/17full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[]"unverifiedinteractive-0.0002302245/17full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[]"unverifiedinteractive-0.0002302245/17full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[]"unverifiedinteractive-0.0002302245/17full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[]"unverifiedinteractive-0.0002302245/17full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[]"unverifiedinteractive-0.0002302245/17full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0007805392/801full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0007805392/801full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0007805392/801full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0007805392/801full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0007805392/801full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0007805392/801full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007195389/601full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007195389/601full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007195389/601full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007195389/601full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007195389/601full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007195389/601full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0007895513/792full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0007895513/792full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0007895513/792full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0007895513/792full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0007895513/792full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0007895513/792full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.00153014133/389full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.00153014133/389full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.00153014133/389full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.00153014133/389full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.00153014133/389full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.00153014133/389full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0007495242/750full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0007495242/750full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0007495242/750full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0007495242/750full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0007495242/750full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0007495242/750full transcript →