VI · Repository

Fixes a cross-file bug with a code-graph tool

Gemma 4 31B QAT

Finding:BREAKS· grade C(research conclusion)

28% correct (5/18)

Graph-on arm of the paired VI.graph_value campaign; the paired comparison with the graph-off arm (<run_id>/graph_off) is in the campaign result note.

Pass ratesingle rate, no difficulty dial

28%

5/18 scored trials passed · 95% interval 12%–51%

Why it breaks

The model returns the buggy function unchanged in behaviour -- nothing was fixed.

Example failure

Trial result: FAIL (one model answer)

FAILno_change

{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "tests/test_rules.py::test_solve_1 FAILED\n assert solve(-1, 'xybzzz') == [8, 7, 8, 22, 16, 3, -3, -2, 6, 4, 573, 63]\n AssertionError: solve(-1, 'xybzzz') returned [8, 7, 8, 22, 16, 3, -3, -2, 6, 4, 575, 63]\ntests/test_rules.py::test_solve_2 PASSED\ntests/test_rules.py::test_solve_3 PASSED\n1 failed, 2 pa

Open this trial →Open the run it came from →

Evidencewhat this finding rests on

Ground truthDeterministic simulation: the right answer is computed, not judged
StatisticsWilson 95% interval 12%–51%Grade C: thin evidence
Sample size18 scored trials in total5 errored trials excluded
Fresh-seed confirmationNot run yet
ReplicationNot run yet

Raw evidenceruns and trials

Runs behind this row
- camp-repo-graph-gemma4qat-001/graph_on 8 trials
- camp-repo-graph-gemma4qat-002/graph_on 15 trials
Every matching trial for this model → broader than this row
The example failure, in full →

Back to the capability map →