Run camp-code-repair-confirm-gemma4qat-postfix-001/trace

80% PASS

What this run revealed, then every individual trial as evidence.

Trials

80%

4/5 scored

over trials

Pass rate

20%

Failures

$0.0016

Run cost

Run camp-code-repair-confirm-gemma4qat-postfix-001/trace: 80% pass rate on code queries.

Of 5 scored trials (code query kind), the model passed 80% and failed 20%.

1 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Diffuse weakness (below 90% but no outright flip) on: bug_distance (pass rate 50% at the high end).

record 6090200f — asked "After this program runs, what is the value of `rate`? Work through it carefully, then give the final answer on its own last line as `ANSWER: <python literal>`."
expected: (unknown)
actual: ANSWER: 48
answer did not match the verified ground truth

Trials5 trials

Per-trial table (5 trials -- prompt, reasoning, verdict, answer key)

model	verdict	kind	difficulty	cost ($)	tokens in/out
gemma-4-31b-it-qat-w4a16-ct	✓PASSanswered "ANSWER: 104"	code_trace	-	0.000288	322/852	full transcript →
gemma-4-31b-it-qat-w4a16-ct	✓PASSanswered "ANSWER: [9, 0, 14, 23, 9, 1, 1, 4, 3]"	code_trace	-	0.000375	368/1128	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "ANSWER: 48"unverified	code_trace	-	0.000315	353/932	full transcript →
gemma-4-31b-it-qat-w4a16-ct	✓PASSanswered "ANSWER: [8, 9, 20, 22, 3, 70, 2, 5, 3]"	code_trace	-	0.000364	393/1084	full transcript →
gemma-4-31b-it-qat-w4a16-ct	✓PASSanswered "ANSWER: 55"	code_trace	-	0.000303	341/896	full transcript →