Data to …… trials
Reliability

Gets the same task right every time (pass^k)

Gemma 4 31B QAT

Finding:DEGRADES· grade B(research conclusion)

58% correct (34/59)

Rate shown is pass^1 over repeats of the same tasks (X.consistency campaign); pass^k needs per-task grouping and is in results/consistency/<run_id>.json. Earlier hand measurement: Nano 63% single-try vs 17% on all 5 tries.

Pass ratesingle rate, no difficulty dial
58%

34/59 scored trials passed · 95% interval 45%–69%

breaks ≤ 30%holds if interval reaches 95%0%50%100%
Why it slips

The model calls a tool that does not exist -- usually a misspelled look-alike of a real tool name.

in 100% of failures (25 failures)

Example failure
Trial result: FAIL (one model answer)
FAILunknown_tool:step=2:set_dep_th_flag_1
[ {"tool": "set_-flag_0", "args": {"flag": "flag_0"}}, {"tool": "set_dep-th_flag_0", "args": {"flag": "depth_flag_0"}}, {"tool": "set_dep_th_flag_1", "args": {"flag": "depth_flag_1"}}, {"tool": "set_dep_th_flag_2", "args": {"flag": "depth_flag_2"}}, {"tool": "set_dep_th_flag_3", "args": {"flag": "depth_flag_3"}}, {"tool": "set_depth_flag_o4", "args": {"flag": "depth_flag_4"}}, {"tool": "set_dep-th
Open this trial →Open the run it came from →
Evidencewhat this finding rests on
  • Ground truthDeterministic simulation: the right answer is computed, not judged
  • StatisticsWilson 95% interval 45%–69%Grade B: moderate evidence
  • Sample size59 scored trials in total1 errored trials excluded
  • Fresh-seed confirmationNot run yet
  • ReplicationNot run yet
Raw evidenceruns and trials

Back to the capability map →