Data to …… trials
IV · Code tracing

Traces a program when variable names look alike

Gemma 4 31B QAT

Finding:HOLDS· grade A(research conclusion)

Holds through 42-statement programs: 40/40 pass across every tested level

Held through 42-statement programs

Pass rate by level4 levels tested
Pass rate by level for Traces a program when variable names look alike0%30%50%75%100%28-statement programs28-statement programs: 10/10 pass (100%)10/1034-statement programs34-statement programs: 10/10 pass (100%)10/1039-statement programs39-statement programs: 10/10 pass (100%)10/1042-statement programs42-statement programs: 10/10 pass (100%)10/10

Dashed lines: a level at or below 30% pass counts toward BREAKS, at or below 75% toward DEGRADES. Labels above points: passed/scored.

LevelPassedPass rateNote
28-statement programs10/10100%
34-statement programs10/10100%
39-statement programs10/10100%
42-statement programs10/10100%
What the few failures have in common

No failures recorded in the trials behind this row.

Evidencewhat this finding rests on
  • Ground truthDeterministic simulation: the right answer is computed, not judged
  • StatisticsWilson 95% interval 91%–100%Grade A: strong evidence
  • Sample size40 scored trials in total
  • Fresh-seed confirmationNot run yet
  • ReplicationNot run yet
Raw evidenceruns and trials

Back to the capability map →