Data to …… trials
EF · Evidence fidelity

Fixes a bug's cause in another file (paired, B arm)

Gemma 4 31B QAT

Finding:DEGRADES· grade C(research conclusion)

33% correct (3/9)

Pass ratesingle rate, no difficulty dial
33%

3/9 scored trials passed · 95% interval 12%–65%

breaks ≤ 30%holds if interval reaches 95%0%50%100%
Why it slips

The model edits a file other than the one the bug is in (usually the file the failing test names).

in 50% of failures (6 failures)

Example failure
Trial result: FAIL (one model answer)
FAILwrong_site_file
{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "tests/test_audit.py::test_solve_1 FAILED\n assert solve([10, -1], 6) == [10, -1, 16, 1, 7, 1, 18, 6, 2, 10, -1, 16, 1, 7, 1, 18, 6, 2, 16, 3, 7, 1, 10, 12, 4, 10, -1, 16, 1, 7, 1, 18, 6, 2, 16, 3, 7, 1, 10, 12, 4, 21, 14, 10, 19, 7, 6, 7, 103, 1, 8, 3, 9, 15, 20, 8, 5, 8, 10, 4, 6, 0, 9, 21, 8, 3, 4, 10, 6
Open this trial →Open the run it came from →
Evidencewhat this finding rests on
  • Ground truthDeterministic simulation: the right answer is computed, not judged
  • StatisticsWilson 95% interval 12%–65%Grade C: thin evidence
  • Sample size9 scored trials in total: small sample (under 15)1 errored trials excluded
  • Fresh-seed confirmationNot run yet
  • ReplicationNot run yet
Raw evidenceruns and trials

Back to the capability map →