IV · Code tracing
Traces a program through semantic traps (aliasing, closures, off-by-one, modulo, defaults)
Gemma 3 27B
Finding:DEGRADES· grade A(research conclusion)
Degrades at 5 of 6 traps active: 26/49 pass (was 41/42 at 0 of 6 traps active)
Boundary 5 of 6 traps active · held through 0 of 6 traps active
Pass rate by level
Dashed lines: a level at or below 30% pass counts toward BREAKS, at or below 75% toward DEGRADES. Labels above points: passed/scored.
| Level | Passed | Pass rate | Note |
|---|---|---|---|
| 0 of 6 traps active | 41/42 | 98% | |
| 5 of 6 traps active | 26/49 | 53% | ▲ boundary |
| 6 of 6 traps active | 18/42 | 43% |
Why it slips
The model treats `b = a` as copying a list, so changes through one name are missed.
in 38% of failures (47 failures at and beyond the boundary)
Example failure
Trial result: FAIL (one model answer)
FAILfoil:value_semantics
Let's trace the execution of the Python code step by step. 1. `limit = 6` 2. `total = 3` 3. `values = [2, 7, 2, 9]` 4. `values.append(limit)`: `values` becomes `[2, 7, 2, 9, 6]` 5. `values.extend([9, 7])`: `values` becomes `[2, 7, 2, 9, 6, 9, 7]` 6. `batch = [5, 0]` 7. `cells = batch`: `cells` is now `[5, 0]` 8. `cells.append(17)`: `cells` becomes `[5, 0, 17]` 9. `limit += batch[-1]`: `limit += 0`
This trial failed a different check from the dominant cause above. It is shown because it is the first failure recorded at the boundary.
Evidence
- Ground truthDeterministic simulation: the right answer is computed, not judged
- StatisticsWilson 95% interval 39%–66%
- Sample size49 scored trials at the boundary
- Fresh-seed confirmationNot run yet
- ReplicationNot run yet
Raw evidence