Findings
The capability map, derived live from 9,472 recorded trials: what each model does well, where exactly it breaks, and how far you should trust each claim. Every number here is computed from the store when this page loads — replicate a finding and it updates.
| model | query kind | difficulty region (high tercile) | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| state | order | belief | adversarial_belief | tool_plan | interactive | code_trace | code_repair | code_write | entity_state↑ | temporal↑ | coreference↑ | distractors↑ | context_length↑ | witness_gap↑ | |
| Gemma 4 31B QAT | 100%n=125 | 95%n=209 | 60%n=638 | 62%n=260 | 81%n=865 | 57%n=624 | 97%n=825 | 33%n=67 | —n=0 | 66%n=508 | 51%n=266 | 67%n=423 | 48%n=293 | 64%n=572 | 31%n=115 |
| Gemma 3 27B | 57%n=46 | 99%n=80 | 49%n=162 | 43%n=145 | 55%n=398 | 91%n=118 | 75%n=246 | —n=0 | —n=0 | 56%n=102 | 67%n=67 | 55%n=134 | 59%n=92 | 57%n=162 | —n=0 |
| Nemotron 3 Nano 30B | 100%n=90 | 99%n=158 | 92%n=564 | 91%n=306 | 66%n=626 | 78%n=121 | —n=0 | —n=0 | —n=0 | 89%n=342 | 94%n=226 | 93%n=392 | 93%n=227 | 92%n=389 | 88%n=75 |
| Nemotron 3 Ultra 550B | 100%n=30 | 100%n=80 | 99%n=183 | 99%n=128 | 89%n=291 | —n=0 | —n=0 | —n=0 | —n=0 | 99%n=132 | 98%n=115 | 99%n=158 | 99%n=116 | 99%n=166 | 98%n=51 |
| Nemotron 3 Super 120B | 97%n=94 | 100%n=79 | 98%n=290 | 96%n=158 | 86%n=537 | 97%n=137 | —n=0 | —n=0 | —n=0 | 97%n=202 | 95%n=172 | 97%n=219 | 99%n=143 | 98%n=241 | 98%n=52 |
| gpt-oss-120b | 100%n=60 | 100%n=79 | 97%n=159 | 98%n=42 | —n=0 | —n=0 | —n=0 | —n=0 | —n=0 | 99%n=105 | 96%n=56 | 98%n=98 | 98%n=45 | 99%n=125 | 80%n=5 |
Gemma 3 27B
What it does well (1)
google/gemma-3-27b-it handles order queries: 99% pass (79/80 scored).
Where it fails (7)
google/gemma-3-27b-it fails adversarial_belief queries: 43% pass (62/145 scored, 83 real failures) — concentrated at entity_state 6–10, where it drops to 20% (3/15).
Where it fails
- entity_state 6–10 → 20% pass (3/15)
- context_length 8–12 → 36% pass (16/45)
- context_length 13–25 → 37% pass (13/35)
Traps that fired
Example failures
google/gemma-3-27b-it fails belief queries: 49% pass (80/162 scored, 82 real failures) — concentrated at coreference = 2, where it drops to 29% (2/7).
Where it fails
- coreference = 2 → 29% pass (2/7)
- temporal = 0 → 47% pass (63/135)
- entity_state = 2 → 48% pass (62/130)
Traps that fired
Example failures
google/gemma-3-27b-it fails tool_plan queries: 55% pass (219/398 scored, 179 real failures).
Traps that fired
Example failures
google/gemma-3-27b-it fails trials in the high-coreference region: 55% pass (74/134 scored, 60 real failures) — concentrated at context_length 8–12, where it drops to 46% (27/59).
Where it fails
- context_length 8–12 → 46% pass (27/59)
- temporal = 0 → 54% pass (65/121)
- distractors = 2 → 54% pass (62/115)
By query kind
- adversarial_belief → 43% pass (20/46)
- belief → 52% pass (29/56)
- order → 100% pass (18/18)
- state → 50% pass (7/14)
Traps that fired
Example failures
google/gemma-3-27b-it fails trials in the high-entity_state region: 56% pass (57/102 scored, 45 real failures).
Where it fails
- context_length 13–34 → 54% pass (47/87)
- coreference 0–1 → 55% pass (42/76)
- temporal = 0 → 56% pass (53/95)
By query kind
- adversarial_belief → 20% pass (3/15)
- belief → 52% pass (14/27)
- order → 100% pass (20/20)
- state → 50% pass (20/40)
Traps that fired
Example failures
google/gemma-3-27b-it fails state queries: 57% pass (26/46 scored, 20 real failures) — concentrated at entity_state 9–30, where it drops to 50% (20/40).
Where it fails
- entity_state 9–30 → 50% pass (20/40)
- context_length 13–34 → 50% pass (20/40)
- coreference = 3 → 50% pass (7/14)
Traps that fired
Example failures
google/gemma-3-27b-it fails code_trace queries: 75% pass (184/246 scored, 62 real failures).
Traps that fired
Example failures
| capability | strongest | rate | weakest | rate | spread |
|---|---|---|---|---|---|
| state | gpt-oss-120b | 100% | Gemma 3 27B | 57% | 43 pts |
| order | gpt-oss-120b | 100% | Gemma 4 31B QAT | 95% | 5 pts |
| belief | Nemotron 3 Ultra 550B | 99% | Gemma 3 27B | 49% | 50 pts |
| adversarial_belief | Nemotron 3 Ultra 550B | 99% | Gemma 3 27B | 43% | 56 pts |
| tool_plan | Nemotron 3 Ultra 550B | 89% | Gemma 3 27B | 55% | 34 pts |
| interactive | Nemotron 3 Super 120B | 97% | Gemma 4 31B QAT | 57% | 40 pts |
| code_trace | Gemma 4 31B QAT | 97% | Gemma 3 27B | 75% | 22 pts |
| entity_state (high) | Nemotron 3 Ultra 550B | 99% | Gemma 3 27B | 56% | 43 pts |
| temporal (high) | Nemotron 3 Ultra 550B | 98% | Gemma 4 31B QAT | 51% | 48 pts |
| coreference (high) | Nemotron 3 Ultra 550B | 99% | Gemma 3 27B | 55% | 44 pts |
| distractors (high) | Nemotron 3 Ultra 550B | 99% | Gemma 4 31B QAT | 48% | 51 pts |
| context_length (high) | Nemotron 3 Ultra 550B | 99% | Gemma 3 27B | 57% | 42 pts |
| witness_gap (high) | Nemotron 3 Super 120B | 98% | Gemma 4 31B QAT | 31% | 67 pts |
Once non-actor witnesses are actually rendered into prose, Nemotron Super shows NO belief/reality collapse: it tracks false beliefs correctly across chain_length, scramble, coref_pressure, and distractor_events.
Nemotron Nano assumes an entity keeps observing an object's moves even after the story stops placing that entity at the scene -- it over-extends 'last witnessed' into 'still watching', producing a false-belief error the true reasoning chain doesn't support.
Sweeping witness_cutoff (how early an entity's last witnessed exposure to an object is cut off from the object's later moves) reproduces the same over-generalization systematically: at witness_cutoff=0.7, belief/truth divergence hits 94%.
Ultra misresolves pronouns toward the story's protagonist specifically -- an ablation that swaps pronoun/name surface form while holding the answer key fixed separates this cleanly from ordinary coreference noise in 6/6 ablated cases.
Under adversarial (foil-selected) belief queries at high trap density, a model can abstain/answer 'unknown' rather than commit to a wrong-but-plausible foil answer, on questions with a fully determinable ground-truth belief. First observed on Ultra (2026-09-17, 2 unreplicated cases); a 2026-09-20 replication attempt on a fresh Ultra candidate carrying the same trap signature evaporated under proper token budget (28000 vs the 16000 'ultra'-class ceiling: 5/5 completed cleanly, 0/5 abstained) -- the original Ultra 'unknown' answers were very plausibly reasoning-chain truncation misread as abstention, not genuine behavior. The SAME replication pass found the real thing on Super instead: 7 adversarial_belief FAILs under the identical trap combination, all clean completions (finish_reason=stop, 500-1900 tokens, no truncation risk on Super's 4000-token budget). Resample-classified per this project's standard noise gate: 3 of 7 were noise (majority/unanimous PASS on resample -- filtered out), 4 survive stable, 3 of those answering 'unknown' on all 4 independent draws (original + 3 resamples).
Escalating every difficulty dim to its generator ceiling (chain_length 25, scramble 1.0, coref_pressure 8, distractor_events 40, filler_events 60, witness_cutoff 0.9 simultaneously) never broke Ultra. Adversarial seed search at a much shorter chain_length=2 -- picking the single world among many candidates where a plausible-but-wrong foil answer most directly competes with the truth -- did break it.
Retractions (1)
First read of camp-belief-nemotron-super-001 (baseline only 3/5) looked like a real ~60%% capability ceiling on false belief tracking, independent of surface difficulty.