Data to …… trials

Run camp-determinism-probe-gemma4qat-001

30% PASS

What this run revealed, then every individual trial as evidence.

30
Trials
30%
9/30 scored
over trials
Pass rate
21
70%
Failures
$0.0188
Run cost

Run camp-determinism-probe-gemma4qat-001, 2 arms: clean on none; below 97% on parallel, serial.

Arm parallel: 27% pass rate across 15 trials.

(parallel) 11 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Arm serial: 33% pass rate across 15 trials.

(serial) 10 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record 5f1d8e71 — asked "Find and fix the bug. The fix will also be checked on other inputs. Return the corrected function, complete (with any helper functions it uses), in one ```python block. Keep the same name and parameters."
    expected: (unknown)
    actual: ```

    answer did not match the verified ground truth

  • record 9f95e142 — asked "Find and fix the bug. The fix will also be checked on other inputs. Return the corrected function, complete (with any helper functions it uses), in one ```python block. Keep the same name and parameters."
    expected: (unknown)
    actual: ```

    answer did not match the verified ground truth

  • record 7b2c42df — asked "Find and fix the bug. The fix will also be checked on other inputs. Return the corrected function, complete (with any helper functions it uses), in one ```python block. Keep the same name and parameters."
    expected: (unknown)
    actual: ```

    answer did not match the verified ground truth

Trials30 trials
Per-trial table (30 trials -- prompt, reasoning, verdict, answer key)
modelstageverdictkinddifficultycost ($)tokens in/out
gemma-4-31b-it-qat-w4a16-ctparallel✓PASSanswered "```"code_repair-0.000361459/1051full transcript →
gemma-4-31b-it-qat-w4a16-ctparallel✓PASSanswered "```"code_repair-0.000361459/1051full transcript →
gemma-4-31b-it-qat-w4a16-ctparallel✓PASSanswered "```"code_repair-0.000361459/1051full transcript →
gemma-4-31b-it-qat-w4a16-ctparallel?FAILanswered "```"unverifiedcode_repair-0.000426459/1268full transcript →
gemma-4-31b-it-qat-w4a16-ctparallel?FAILanswered "```"unverifiedcode_repair-0.000405386/1220full transcript →
gemma-4-31b-it-qat-w4a16-ctparallel?FAILanswered "```"unverifiedcode_repair-0.000405386/1220full transcript →
gemma-4-31b-it-qat-w4a16-ctparallel?FAILanswered "```"unverifiedcode_repair-0.000426459/1268full transcript →
gemma-4-31b-it-qat-w4a16-ctparallel?FAILanswered "```"unverifiedcode_repair-0.000312364/920full transcript →
gemma-4-31b-it-qat-w4a16-ctparallel?FAILanswered "```"unverifiedcode_repair-0.000700386/2206full transcript →
gemma-4-31b-it-qat-w4a16-ctparallel✓PASSanswered "```"code_repair-0.000306364/898full transcript →
gemma-4-31b-it-qat-w4a16-ctparallel?FAILanswered "```"unverifiedcode_repair-0.000312364/920full transcript →
gemma-4-31b-it-qat-w4a16-ctparallel?FAILanswered "```"unverifiedcode_repair-0.000313364/921full transcript →
gemma-4-31b-it-qat-w4a16-ctparallel?FAILanswered "```"unverifiedcode_repair-0.000312364/920full transcript →
gemma-4-31b-it-qat-w4a16-ctparallel?FAILanswered "```"unverifiedcode_repair-0.001892386/6178full transcript →
gemma-4-31b-it-qat-w4a16-ctparallel?FAILanswered "```"unverifiedcode_repair-0.001892386/6178full transcript →
gemma-4-31b-it-qat-w4a16-ctserial?FAILanswered "```"unverifiedcode_repair-0.000561459/1717full transcript →
gemma-4-31b-it-qat-w4a16-ctserial✓PASSanswered "```"code_repair-0.000361459/1051full transcript →
gemma-4-31b-it-qat-w4a16-ctserial✓PASSanswered "```"code_repair-0.000361459/1051full transcript →
gemma-4-31b-it-qat-w4a16-ctserial✓PASSanswered "```"code_repair-0.000361459/1051full transcript →
gemma-4-31b-it-qat-w4a16-ctserial✓PASSanswered "```"code_repair-0.000361459/1051full transcript →
gemma-4-31b-it-qat-w4a16-ctserial?FAILanswered "```"unverifiedcode_repair-0.000327386/962full transcript →
gemma-4-31b-it-qat-w4a16-ctserial?FAILanswered "```"unverifiedcode_repair-0.001436386/4658full transcript →
gemma-4-31b-it-qat-w4a16-ctserial?FAILanswered "```"unverifiedcode_repair-0.001436386/4658full transcript →
gemma-4-31b-it-qat-w4a16-ctserial?FAILanswered "```"unverifiedcode_repair-0.001436386/4658full transcript →
gemma-4-31b-it-qat-w4a16-ctserial?FAILanswered "```"unverifiedcode_repair-0.001436386/4658full transcript →
gemma-4-31b-it-qat-w4a16-ctserial✓PASSanswered "```"code_repair-0.000644364/2025full transcript →
gemma-4-31b-it-qat-w4a16-ctserial?FAILanswered "```"unverifiedcode_repair-0.000312364/920full transcript →
gemma-4-31b-it-qat-w4a16-ctserial?FAILanswered "```"unverifiedcode_repair-0.000312364/920full transcript →
gemma-4-31b-it-qat-w4a16-ctserial?FAILanswered "```"unverifiedcode_repair-0.000312364/920full transcript →
gemma-4-31b-it-qat-w4a16-ctserial?FAILanswered "```"unverifiedcode_repair-0.000312364/920full transcript →