V · Code repair
Fixes stacked bugs without breaking anything else
Gemma 4 31B QAT
Finding:BREAKS· grade B(research conclusion)
Breaks at 1 bugs: 1/8 pass -- already weak at the easiest level tested
Boundary 1 bugs
Pass rate by level
Dashed lines: a level at or below 30% pass counts toward BREAKS, at or below 75% toward DEGRADES. Labels above points: passed/scored.
| Level | Passed | Pass rate | Note |
|---|---|---|---|
| 1 bugs | 1/8 | 12% | ▲ boundary; 4 errored (excluded) |
Why it breaks
The model edits a different line than the one with the bug.
in 71% of failures (7 failures at and beyond the boundary)
Example failure
Trial result: FAIL (one model answer)
FAILwrong_site
To fix the bug, we need to examine how the `offset` variable is calculated, as it is the only value that differs between the actual output and the expected output for the input `solve(2, [5, 7])`. 1. **Current Calculation for `total = 2`**: * `total` is not 6, so it goes to the `else` block: `total = total if total > total else total + 2`. Since `2 > 2` is False, `total` becomes `2 + 2 = 4`. * `of
Evidence
- Ground truthDeterministic simulation: the right answer is computed, not judged
- StatisticsWilson 95% interval 2%–47%
- Sample size8 scored trials at the boundary: small sample (under 15)
- Fresh-seed confirmationNot run yet
- ReplicationNot run yet
Raw evidence