Data to …… trials
II · Tool planning

Notices when a task is impossible

Gemma 3 27B

Finding:BREAKS· grade C(research conclusion)

Declines 0 of 16 impossible tasks; writes a plan for the other 16

Pass ratesingle rate, no difficulty dial
0%

0/16 scored trials passed · 95% interval 0%–19%

breaks ≤ 30%holds if interval reaches 95%0%50%100%
Why it breaks

The model writes a plan for an impossible goal instead of declining.

in 100% of failures (16 failures)

Example failure
Trial result: FAIL (one model answer)
FAILfabricated_plan_on_unsolvable:4
```json [ {"tool": "build_art_0", "args": {"artifact": "art_0"}}, {"tool": "publish_art_0", "args": {"artifact": "art_0"}}, {"tool": "set_flag_0", "args": {"flag": "flag_0"}}, {"tool": "advance_2", "args": {"service": "svc_0"}} ] ```
Open this trial →Open the run it came from →
Evidencewhat this finding rests on
  • Ground truthDeterministic simulation: the right answer is computed, not judged
  • StatisticsWilson 95% interval 0%–19%Grade C: thin evidence
  • Sample size16 scored trials in total
  • Fresh-seed confirmationNot run yet
  • ReplicationNot run yet
Raw evidenceruns and trials

Back to the capability map →