Data to …… trials

Run camp-determinism-probe-gemma4qat-001/serial

33% PASS

What this run revealed, then every individual trial as evidence.

15
Trials
33%
5/15 scored
over trials
Pass rate
10
67%
Failures
$0.0100
Run cost

Run camp-determinism-probe-gemma4qat-001/serial: 33% pass rate on code queries.

Of 15 scored trials (code query kind), the model passed 33% and failed 67%.

10 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record 7b2c42df — asked "Find and fix the bug. The fix will also be checked on other inputs. Return the corrected function, complete (with any helper functions it uses), in one ```python block. Keep the same name and parameters."
    expected: (unknown)
    actual: ```

    answer did not match the verified ground truth

  • record dcc2d5eb — asked "Find and fix the bug. The fix will also be checked on other inputs. Return the corrected function, complete (with any helper functions it uses), in one ```python block. Keep the same name and parameters."
    expected: (unknown)
    actual: ```

    answer did not match the verified ground truth

  • record 2f53b5db — asked "Find and fix the bug. The fix will also be checked on other inputs. Return the corrected function, complete (with any helper functions it uses), in one ```python block. Keep the same name and parameters."
    expected: (unknown)
    actual: ```

    answer did not match the verified ground truth

Trials15 trials
Per-trial table (15 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```"unverifiedcode_repair-0.000561459/1717full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "```"code_repair-0.000361459/1051full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "```"code_repair-0.000361459/1051full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "```"code_repair-0.000361459/1051full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "```"code_repair-0.000361459/1051full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```"unverifiedcode_repair-0.000327386/962full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```"unverifiedcode_repair-0.001436386/4658full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```"unverifiedcode_repair-0.001436386/4658full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```"unverifiedcode_repair-0.001436386/4658full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```"unverifiedcode_repair-0.001436386/4658full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "```"code_repair-0.000644364/2025full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```"unverifiedcode_repair-0.000312364/920full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```"unverifiedcode_repair-0.000312364/920full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```"unverifiedcode_repair-0.000312364/920full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```"unverifiedcode_repair-0.000312364/920full transcript →