Run smoke-v-gemma4qat/trace

50% PASS

What this run revealed, then every individual trial as evidence.

Trials

50%

1/2 scored

Pass rate

50%

Failures

$0.0007

Run cost

Run smoke-v-gemma4qat/trace: 50% pass rate on code queries.

Of 2 scored trials (code query kind), the model passed 50% and failed 50%.

1 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

record d83b219f — asked "What does `solve([7, 3], 4)` return? Work through it carefully, then give the final answer on its own last line as `ANSWER: <python literal>`."
expected: (unknown)
actual: ANSWER: [7, 3, 3, 5, 7, 16, 2, 1]
answer did not match the verified ground truth

Trials2 trials

Per-trial table (2 trials -- prompt, reasoning, verdict, answer key)

model	verdict	kind	difficulty	cost ($)	tokens in/out
gemma-4-31b-it-qat-w4a16-ct	✓PASSanswered "ANSWER: [8, 4, 3, 8, 19, 3, 3, 8, 9]"	code_trace	-	0.000359	352/1078	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "ANSWER: [7, 3, 3, 5, 7, 16, 2, 1]"unverified	code_trace	-	0.000343	356/1024	full transcript →