Data to …… trials
EF · Evidence fidelity

Answers from what a person saw, against where things really are (paired, B arm)

Gemma 4 31B QAT

Finding:DEGRADES· grade B(research conclusion)

37% correct (11/30)

Tested up to chain length 8, distractor events 2

Pass ratesingle rate, no difficulty dial
37%

11/30 scored trials passed · 95% interval 22%–55%

breaks ≤ 30%holds if interval reaches 95%0%50%100%
Why it slips

The model answers where the object really is, not where the person believes it is.

in 84% of failures (19 failures)

Example failure
Trial result: FAIL (one model answer)
FAIL
west wing
Open this trial →Open the run it came from →
Evidencewhat this finding rests on
  • Ground truthDeterministic simulation: the right answer is computed, not judged
  • StatisticsWilson 95% interval 22%–55%Grade B: moderate evidence
  • Sample size30 scored trials in total
  • Fresh-seed confirmationNot run yet
  • ReplicationNot run yet
Raw evidenceruns and trials

Back to the capability map →