Data to …… trials

Run camp-code-repair-confirm-gemma4qat-001

35% PASS

What this run revealed, then every individual trial as evidence.

20
3 errors
Trials
35%
6/17 scored
over trials
Pass rate
11
55%
Failures
$0.0076
Run cost

Run camp-code-repair-confirm-gemma4qat-001, 1 arms: clean on none; below 97% on trace.

Arm trace: 65% pass rate across 20 trials.

(trace) 7 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record 31650d9e — asked "What does `solve(8, [8, 4, 8])` return? Work through it carefully, then give the final answer on its own last line as `ANSWER: <python literal>`."
    expected: (unknown)
    actual: ANSWER: 13

    answer did not match the verified ground truth

  • record 29d29945 — asked "What does `solve('red', 1)` return? Work through it carefully, then give the final answer on its own last line as `ANSWER: <python literal>`."
    expected: (unknown)
    actual: ANSWER: [1, 2, 0, 8, 1, -2, 12, 15, 3, 1, 2]

    answer did not match the verified ground truth

Trials20 trials
Per-trial table (20 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```"unverifiedcode_repair-0.000162366/418full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "```"code_repair-0.000359359/1076full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```"unverifiedcode_repair-0.000364458/1060full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```"unverifiedcode_repair-0.000468378/1434full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "```"code_repair-0.000421450/1253full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```"unverifiedcode_repair-0.000424459/1261full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "```"code_repair-0.000371359/1117full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```"unverifiedcode_repair-0.000405386/1220full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```"unverifiedcode_repair-0.000312364/920full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "```"code_repair-0.000464372/1423full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```"unverifiedcode_repair-0.000473350/1460full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```"unverifiedcode_repair-0.000523379/1617full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORstopcode_repair-0.000490475/1474full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORtimeoutcode_repair-0.0000000/0full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "```"code_repair-0.000377385/1127full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "```"code_repair-0.000258454/709full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```"unverifiedcode_repair-0.000382374/1150full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORstopcode_repair-0.000497366/1534full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```"unverifiedcode_repair-0.000289493/800full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```"unverifiedcode_repair-0.000557392/1726full transcript →