Data to …… trials
VI · Repository

Fixes the same bug as its program is split over more files

Gemma 4 31B QAT

Finding:BREAKS· grade B(research conclusion)

Breaks at 8 files: 4/18 pass (already only 9/22 at 1 files)

Boundary 8 files

Pass rate by level3 levels tested
Pass rate by level for Fixes the same bug as its program is split over more files0%30%50%75%100%Boundary: breaks at 8 files1 files1 files: 9/22 pass (41%), 2 errored9/224 files4 files: 8/24 pass (33%)8/248 files8 files: 4/18 pass (22%), 6 errored4/18

Dashed lines: a level at or below 30% pass counts toward BREAKS, at or below 75% toward DEGRADES. Labels above points: passed/scored.

LevelPassedPass rateNote
1 files9/2241%2 errored (excluded)
4 files8/2433%
8 files4/1822%▲ boundary; 6 errored (excluded)
Why it breaks

The model edits a file other than the one the bug is in (usually the file the failing test names).

in 36% of failures (14 failures at and beyond the boundary)

Example failureat 8 files
Trial result: FAIL (one model answer)
FAILwrong_site_file
{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "tests/test_budgeting.py::test_solve_1 FAILED\n assert solve('ab', [-2, 5, 8, -1, 11, -9]) == 6304\n AssertionError: solve('ab', [-2, 5, 8, -1, 11, -9]) returned 6184\n1 failed, 0 passed"}, {"step": 2, "tool": "read_file", "args": {"path": "budgeting.py"}, "ok": true, "kind": null, "output": " 1 from helper
Open this trial →Open the run it came from →
Evidencewhat this finding rests on
  • Ground truthDeterministic simulation: the right answer is computed, not judged
  • StatisticsWilson 95% interval 9%–45%Grade B: moderate evidence
  • Sample size18 scored trials at the boundary6 errored trials excluded
  • Fresh-seed confirmationNot run yet
  • ReplicationNot run yet
Raw evidenceruns and trials

Back to the capability map →