Data to …… trials
VI · Repository

Finds and fixes a bug whose cause is in another file

Gemma 4 31B QAT

Finding:BREAKS· grade B(research conclusion)

Breaks at cause 1 calls away: 1/11 pass -- already weak at the easiest level tested

Boundary cause 1 calls away

Pass rate by level3 levels tested
Pass rate by level for Finds and fixes a bug whose cause is in another file0%30%50%75%100%Boundary: breaks at cause 1 calls awaycause 1 calls awaycause 1 calls away: 1/11 pass (9%), 1 errored1/11cause 4 calls awaycause 4 calls away: 4/10 pass (40%), 2 errored4/10cause 6 calls awaycause 6 calls away: 2/10 pass (20%), 2 errored2/10

Dashed lines: a level at or below 30% pass counts toward BREAKS, at or below 75% toward DEGRADES. Labels above points: passed/scored.

LevelPassedPass rateNote
cause 1 calls away1/119%▲ boundary; 1 errored (excluded)
cause 4 calls away4/1040%2 errored (excluded)
cause 6 calls away2/1020%2 errored (excluded)
Why it breaks

The model patches the shown example only; the bug is still there on other inputs.

in 46% of failures (24 failures at and beyond the boundary)

Example failureat cause 1 calls away
Trial result: FAIL (one model answer)
FAILvisible_fixed_hidden_broken
{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "tests/test_discounts.py::test_solve_1 FAILED\n assert solve(14, [3, -3, 10, 4, -7]) == [7, 2, 4, 16, -4, 4, 7, 28, 3, 2, 560, 296]\n AssertionError: solve(14, [3, -3, 10, 4, -7]) returned [7, 2, 4, 16, -4, 4, 7, 28, 3, 2, 538, 296]\ntests/test_discounts.py::test_solve_2 PASSED\ntests/test_discounts.py::tes
Open this trial →Open the run it came from →
Evidencewhat this finding rests on
  • Ground truthDeterministic simulation: the right answer is computed, not judged
  • StatisticsWilson 95% interval 2%–38%Grade B: moderate evidence
  • Sample size11 scored trials at the boundary: small sample (under 15)1 errored trials excluded
  • Fresh-seed confirmationNot run yet
  • ReplicationNot run yet
Raw evidenceruns and trials

Back to the capability map →