Data to …… trials
IV · Code tracing

Traces a program when variable names look alike

Gemma 3 27B

Finding:HOLDS· grade B(research conclusion)

Holds through 42-statement programs: 25/28 pass across every tested level

Held through 42-statement programs

Pass rate by level4 levels tested
Pass rate by level for Traces a program when variable names look alike0%30%50%75%100%28-statement programs28-statement programs: 7/7 pass (100%)7/734-statement programs34-statement programs: 7/7 pass (100%)7/739-statement programs39-statement programs: 4/7 pass (57%)4/742-statement programs42-statement programs: 7/7 pass (100%)7/7

Dashed lines: a level at or below 30% pass counts toward BREAKS, at or below 75% toward DEGRADES. Labels above points: passed/scored.

LevelPassedPass rateNote
28-statement programs7/7100%
34-statement programs7/7100%
39-statement programs4/757%
42-statement programs7/7100%
What the few failures have in common

The model gives a wrong value that matches no known misreading.

in 100% of failures (3 failures)

Evidencewhat this finding rests on
  • Ground truthDeterministic simulation: the right answer is computed, not judged
  • StatisticsWilson 95% interval 73%–96%Grade B: moderate evidence
  • Sample size28 scored trials in total
  • Fresh-seed confirmationNot run yet
  • ReplicationNot run yet
Raw evidenceruns and trials

Back to the capability map →