Run camp-ticket-length-gemma4qat-001

40% PASS

What this run revealed, then every individual trial as evidence.

Trials

40%

2/5 scored

over trials

Pass rate

60%

Failures

$0.0035

Run cost

Run camp-ticket-length-gemma4qat-001, 1 arms: clean on none; below 97% on ablate.

Arm ablate: 56% pass rate across 54 trials.

(ablate) 24 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

record 0dd95acc — asked "You may make at most 22 tool calls."
expected: (unknown)
actual: [{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag_0"}, "ok": true, "error_kind": null, "r...
answer did not match the verified ground truth
record 59d124a7 — asked "You may make at most 22 tool calls."
expected: (unknown)
actual: [{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag_0"}, "ok": true, "error_kind": null, "r...
answer did not match the verified ground truth

Trials5 trials

Per-trial table (5 trials -- prompt, reasoning, verdict, answer key)

model	verdict	kind	difficulty	cost ($)	tokens in/out
gemma-4-31b-it-qat-w4a16-ct	✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."	interactive	-	0.000608	5113/323	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "[{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag..."unverified	interactive	-	0.000701	5350/554	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverified	interactive	-	0.000722	5392/610	full transcript →
gemma-4-31b-it-qat-w4a16-ct	✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."	interactive	-	0.000735	5361/662	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverified	interactive	-	0.000719	5389/601	full transcript →