Data to …… trials

Run camp-ef-tickets-gemma4qat-001

16% PASS

What this run revealed, then every individual trial as evidence.

watch live →compare across models →
50
Trials
16%
8/50 scored
over trials
Pass rate
42
84%
Failures
$0.0362
Run cost

Run camp-ef-tickets-gemma4qat-001, 5 arms: clean on none; below 97% on A, B, C, D, X.

Arm A: 0% pass rate across 10 trials.

(A) 10 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Arm B: 0% pass rate across 10 trials.

(B) 10 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Arm C: 50% pass rate across 10 trials.

(C) 5 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Arm D: 10% pass rate across 10 trials.

(D) 9 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Arm X: 20% pass rate across 10 trials.

(X) 8 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record f2738901 — asked "You may make at most 22 tool calls."
    expected: (unknown)
    actual: [{"step": 1, "tool": "advance_0", "args": {"service": "svc_0"}, "ok": true, "error_kind": null, "...

    answer did not match the verified ground truth

  • record 5adf4516 — asked "You may make at most 22 tool calls."
    expected: (unknown)
    actual: [{"step": 1, "tool": "advance_0", "args": {"service": "svc_0"}, "ok": true, "error_kind": null, "...

    answer did not match the verified ground truth

  • record 75dde693 — asked "You may make at most 22 tool calls."
    expected: (unknown)
    actual: [{"step": 1, "tool": "advance_0", "args": {"service": "svc_0"}, "ok": true, "error_kind": null, "...

    answer did not match the verified ground truth

Trials50 trials
Per-trial table (50 trials -- prompt, reasoning, verdict, answer key)
modelstageverdictkinddifficultycost ($)tokens in/out
gemma-4-31b-it-qat-w4a16-ctA?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007095356/578full transcript →
gemma-4-31b-it-qat-w4a16-ctA?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007185378/600full transcript →
gemma-4-31b-it-qat-w4a16-ctA?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0006775177/532full transcript →
gemma-4-31b-it-qat-w4a16-ctA?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007225402/606full transcript →
gemma-4-31b-it-qat-w4a16-ctA?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0006555251/433full transcript →
gemma-4-31b-it-qat-w4a16-ctA?FAILanswered "[{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag..."unverifiedinteractive-0.0007025353/557full transcript →
gemma-4-31b-it-qat-w4a16-ctA?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007225402/606full transcript →
gemma-4-31b-it-qat-w4a16-ctA?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007115338/592full transcript →
gemma-4-31b-it-qat-w4a16-ctA?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007165380/594full transcript →
gemma-4-31b-it-qat-w4a16-ctA?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007125345/593full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007095359/578full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007185378/600full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0006795180/536full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007245406/610full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0006575255/437full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "[{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag..."unverifiedinteractive-0.0007025353/557full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007235405/609full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007125341/592full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007165381/594full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007135348/593full transcript →
gemma-4-31b-it-qat-w4a16-ctC?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007355588/587full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0007475497/657full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0007005410/529full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0008265603/885full transcript →
gemma-4-31b-it-qat-w4a16-ctC?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0006815496/437full transcript →
gemma-4-31b-it-qat-w4a16-ctC?FAILanswered "[{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag..."unverifiedinteractive-0.0007225544/557full transcript →
gemma-4-31b-it-qat-w4a16-ctC?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007445614/609full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0007465550/635full transcript →
gemma-4-31b-it-qat-w4a16-ctC?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007315505/601full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0007355531/605full transcript →
gemma-4-31b-it-qat-w4a16-ctD?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007095360/578full transcript →
gemma-4-31b-it-qat-w4a16-ctD?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007185379/600full transcript →
gemma-4-31b-it-qat-w4a16-ctD?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0006805181/539full transcript →
gemma-4-31b-it-qat-w4a16-ctD✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0007745407/777full transcript →
gemma-4-31b-it-qat-w4a16-ctD?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0006575256/439full transcript →
gemma-4-31b-it-qat-w4a16-ctD?FAILanswered "[{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag..."unverifiedinteractive-0.0007035354/559full transcript →
gemma-4-31b-it-qat-w4a16-ctD?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007245406/611full transcript →
gemma-4-31b-it-qat-w4a16-ctD?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007125342/592full transcript →
gemma-4-31b-it-qat-w4a16-ctD?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007165382/594full transcript →
gemma-4-31b-it-qat-w4a16-ctD?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007135349/593full transcript →
gemma-4-31b-it-qat-w4a16-ctX?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007425655/587full transcript →
gemma-4-31b-it-qat-w4a16-ctX?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007545718/606full transcript →
gemma-4-31b-it-qat-w4a16-ctX?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0006745476/423full transcript →
gemma-4-31b-it-qat-w4a16-ctX?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007555724/610full transcript →
gemma-4-31b-it-qat-w4a16-ctX?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0006865551/437full transcript →
gemma-4-31b-it-qat-w4a16-ctX✓PASSanswered "[{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag..."interactive-0.0007805671/709full transcript →
gemma-4-31b-it-qat-w4a16-ctX?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007555723/609full transcript →
gemma-4-31b-it-qat-w4a16-ctX✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0009525615/1301full transcript →
gemma-4-31b-it-qat-w4a16-ctX?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007525721/601full transcript →
gemma-4-31b-it-qat-w4a16-ctX?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007455644/602full transcript →