Data to …… trials
EF · Evidence fidelity

Copies exact look-alike tool ids (paired, B arm)

Gemma 4 31B QAT

Finding:DEGRADES· grade C(research conclusion)

80% correct (8/10)

Pass ratesingle rate, no difficulty dial
80%

8/10 scored trials passed · 95% interval 49%–94%

breaks ≤ 30%holds if interval reaches 95%0%50%100%
Why it slips

The model calls a tool that does not exist -- usually a misspelled look-alike of a real tool name.

in 100% of failures (2 failures)

Example failure
Trial result: FAIL (one model answer)
FAILunknown_tool:step=10:set_dep-th_flag_4
[ {"tool": "advance_o0", "args": {"service": "svc_0"}}, {"tool": "adva-nce_1", "args": {"service": "svc_0"}}, {"tool": "adva_nce_2", "args": {"service": "svc_0"}}, {"tool": "build_art_0", "args": {"artifact": "art_0"}}, {"tool": "publis-h_art_0", "args": {"artifact": "art_0"}}, {"tool": "set_flag_0", "args": {"flag": "flag_0"}}, {"tool": "set_dep-th_flag_0", "args": {"flag": "depth_flag_0"}}, {"to
Open this trial →Open the run it came from →
Evidencewhat this finding rests on
  • Ground truthDeterministic simulation: the right answer is computed, not judged
  • StatisticsWilson 95% interval 49%–94%Grade C: thin evidence
  • Sample size10 scored trials in total: small sample (under 15)
  • Fresh-seed confirmationNot run yet
  • ReplicationNot run yet
Raw evidenceruns and trials

Back to the capability map →