V · Code repair
Fixes a one-bug function when shown a failing example
Gemma 4 31B QAT
Finding:DEGRADES· grade C(research conclusion)
Degrades at 0 hidden checks: 3/5 pass -- already weak at the easiest level tested
Boundary 0 hidden checks
Pass rate by level
Dashed lines: a level at or below 30% pass counts toward BREAKS, at or below 75% toward DEGRADES. Labels above points: passed/scored.
| Level | Passed | Pass rate | Note |
|---|---|---|---|
| 0 hidden checks | 3/5 | 60% | ▲ boundary |
| 4 hidden checks | 8/22 | 36% | 3 errored (excluded) |
Why it slips
The model gives a wrong value that matches no known misreading.
in 44% of failures (16 failures at and beyond the boundary)
Example failure
Trial result: FAIL (one model answer)
FAILwrong_value
To fix the bug, we need to analyze the calculation of `cost` step-by-step for the input `solve(-2, [-1, 15])`. 1. `cost = -2`, `values = [-1, 15]` 2. `weight` logic: * Initial `weight = 6`. * Iteration 1 (`e = -1`): `weight < cost` (6 < -2) is False $\rightarrow$ `weight = cost + 6 = -2 + 6 = 4`. * Iteration 2 (`e = 15`): `weight < cost` (4 < -2) is False $\rightarrow$ `weight = cost + 6 = -2 + 6
Evidence
- Ground truthDeterministic simulation: the right answer is computed, not judged
- StatisticsWilson 95% interval 23%–88%
- Sample size5 scored trials at the boundary: small sample (under 15)
- Fresh-seed confirmationNot run yet
- ReplicationNot run yet
Raw evidence