Data to …… trials

Run camp-determinism-probe-gemma4qat-001/parallel

27% PASS

What this run revealed, then every individual trial as evidence.

15
Trials
27%
4/15 scored
over trials
Pass rate
11
73%
Failures
$0.0088
Run cost

Run camp-determinism-probe-gemma4qat-001/parallel: 27% pass rate on code queries.

Of 15 scored trials (code query kind), the model passed 27% and failed 73%.

11 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record 5f1d8e71 — asked "Find and fix the bug. The fix will also be checked on other inputs. Return the corrected function, complete (with any helper functions it uses), in one ```python block. Keep the same name and parameters."
    expected: (unknown)
    actual: ```

    answer did not match the verified ground truth

  • record 9f95e142 — asked "Find and fix the bug. The fix will also be checked on other inputs. Return the corrected function, complete (with any helper functions it uses), in one ```python block. Keep the same name and parameters."
    expected: (unknown)
    actual: ```

    answer did not match the verified ground truth

  • record 1b2672e1 — asked "Find and fix the bug. The fix will also be checked on other inputs. Return the corrected function, complete (with any helper functions it uses), in one ```python block. Keep the same name and parameters."
    expected: (unknown)
    actual: ```

    answer did not match the verified ground truth

Trials15 trials
Per-trial table (15 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "```"code_repair-0.000361459/1051full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "```"code_repair-0.000361459/1051full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "```"code_repair-0.000361459/1051full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```"unverifiedcode_repair-0.000426459/1268full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```"unverifiedcode_repair-0.000405386/1220full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```"unverifiedcode_repair-0.000405386/1220full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```"unverifiedcode_repair-0.000426459/1268full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```"unverifiedcode_repair-0.000312364/920full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```"unverifiedcode_repair-0.000700386/2206full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "```"code_repair-0.000306364/898full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```"unverifiedcode_repair-0.000312364/920full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```"unverifiedcode_repair-0.000313364/921full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```"unverifiedcode_repair-0.000312364/920full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```"unverifiedcode_repair-0.001892386/6178full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```"unverifiedcode_repair-0.001892386/6178full transcript →