Findings
The capability map, derived live from 9,472 recorded trials: what each model does well, where exactly it breaks, and how far you should trust each claim. Every number here is computed from the store when this page loads — replicate a finding and it updates.
| model | query kind | difficulty region (high tercile) | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| state | order | belief | adversarial_belief | tool_plan | interactive | code_trace | code_repair | code_write | entity_state↑ | temporal↑ | coreference↑ | distractors↑ | context_length↑ | witness_gap↑ | |
| Gemma 4 31B QAT | 100%n=125 | 95%n=209 | 60%n=638 | 62%n=260 | 81%n=865 | 57%n=624 | 97%n=825 | 33%n=67 | —n=0 | 66%n=508 | 51%n=266 | 67%n=423 | 48%n=293 | 64%n=572 | 31%n=115 |
| Gemma 3 27B | 57%n=46 | 99%n=80 | 49%n=162 | 43%n=145 | 55%n=398 | 91%n=118 | 75%n=246 | —n=0 | —n=0 | 56%n=102 | 67%n=67 | 55%n=134 | 59%n=92 | 57%n=162 | —n=0 |
| Nemotron 3 Nano 30B | 100%n=90 | 99%n=158 | 92%n=564 | 91%n=306 | 66%n=626 | 78%n=121 | —n=0 | —n=0 | —n=0 | 89%n=342 | 94%n=226 | 93%n=392 | 93%n=227 | 92%n=389 | 88%n=75 |
| Nemotron 3 Ultra 550B | 100%n=30 | 100%n=80 | 99%n=183 | 99%n=128 | 89%n=291 | —n=0 | —n=0 | —n=0 | —n=0 | 99%n=132 | 98%n=115 | 99%n=158 | 99%n=116 | 99%n=166 | 98%n=51 |
| Nemotron 3 Super 120B | 97%n=94 | 100%n=79 | 98%n=290 | 96%n=158 | 86%n=537 | 97%n=137 | —n=0 | —n=0 | —n=0 | 97%n=202 | 95%n=172 | 97%n=219 | 99%n=143 | 98%n=241 | 98%n=52 |
| gpt-oss-120b | 100%n=60 | 100%n=79 | 97%n=159 | 98%n=42 | —n=0 | —n=0 | —n=0 | —n=0 | —n=0 | 99%n=105 | 96%n=56 | 98%n=98 | 98%n=45 | 99%n=125 | 80%n=5 |
Gemma 4 31B QAT
What it does well (3)
gemma-4-31b-it-qat-w4a16-ct passes state queries cleanly: 125/125, no failures at all.
gemma-4-31b-it-qat-w4a16-ct handles code_trace queries: 97% pass (797/825 scored).
gemma-4-31b-it-qat-w4a16-ct handles order queries: 95% pass (199/209 scored).
Where it fails (7)
gemma-4-31b-it-qat-w4a16-ct fails trials in the high-witness_gap region: 31% pass (36/115 scored, 79 real failures) — concentrated at distractors = 2, where it drops to 17% (1/6).
Where it fails
- distractors = 2 → 17% pass (1/6)
- entity_state 5–12 → 28% pass (26/94)
- context_length 21–66 → 28% pass (26/94)
By query kind
- belief → 31% pass (36/115)
Traps that fired
Example failures
gemma-4-31b-it-qat-w4a16-ct fails code_repair queries: 33% pass (22/67 scored, 45 real failures).
Traps that fired
Example failures
gemma-4-31b-it-qat-w4a16-ct fails trials in the high-distractors region: 48% pass (142/293 scored, 151 real failures) — concentrated at temporal 0.15–0.58, where it drops to 29% (35/119).
Where it fails
- temporal 0.15–0.58 → 29% pass (35/119)
- witness_gap 2–8 → 32% pass (35/109)
- coreference = 2 → 36% pass (4/11)
By query kind
- adversarial_belief → 35% pass (14/40)
- belief → 42% pass (87/208)
- order → 91% pass (41/45)
Traps that fired
Example failures
gemma-4-31b-it-qat-w4a16-ct fails interactive queries: 57% pass (354/624 scored, 270 real failures).
Traps that fired
Example failures
gemma-4-31b-it-qat-w4a16-ct fails belief queries: 60% pass (384/638 scored, 254 real failures) — concentrated at witness_gap 2–8, where it drops to 31% (36/115).
Where it fails
- witness_gap 2–8 → 31% pass (36/115)
- temporal 0.15–0.71 → 36% pass (57/160)
- distractors 4–20 → 42% pass (87/208)
Traps that fired
Example failures
gemma-4-31b-it-qat-w4a16-ct fails adversarial_belief queries: 62% pass (160/260 scored, 100 real failures) — concentrated at entity_state 6–10, where it drops to 13% (4/30).
Where it fails
- entity_state 6–10 → 13% pass (4/30)
- entity_state = 4 → 20% pass (2/10)
- context_length 13–25 → 24% pass (12/50)
Traps that fired
Example failures
gemma-4-31b-it-qat-w4a16-ct fails tool_plan queries: 81% pass (700/865 scored, 165 real failures).
Traps that fired
Example failures
| capability | strongest | rate | weakest | rate | spread |
|---|---|---|---|---|---|
| state | gpt-oss-120b | 100% | Gemma 3 27B | 57% | 43 pts |
| order | gpt-oss-120b | 100% | Gemma 4 31B QAT | 95% | 5 pts |
| belief | Nemotron 3 Ultra 550B | 99% | Gemma 3 27B | 49% | 50 pts |
| adversarial_belief | Nemotron 3 Ultra 550B | 99% | Gemma 3 27B | 43% | 56 pts |
| tool_plan | Nemotron 3 Ultra 550B | 89% | Gemma 3 27B | 55% | 34 pts |
| interactive | Nemotron 3 Super 120B | 97% | Gemma 4 31B QAT | 57% | 40 pts |
| code_trace | Gemma 4 31B QAT | 97% | Gemma 3 27B | 75% | 22 pts |
| entity_state (high) | Nemotron 3 Ultra 550B | 99% | Gemma 3 27B | 56% | 43 pts |
| temporal (high) | Nemotron 3 Ultra 550B | 98% | Gemma 4 31B QAT | 51% | 48 pts |
| coreference (high) | Nemotron 3 Ultra 550B | 99% | Gemma 3 27B | 55% | 44 pts |
| distractors (high) | Nemotron 3 Ultra 550B | 99% | Gemma 4 31B QAT | 48% | 51 pts |
| context_length (high) | Nemotron 3 Ultra 550B | 99% | Gemma 3 27B | 57% | 42 pts |
| witness_gap (high) | Nemotron 3 Super 120B | 98% | Gemma 4 31B QAT | 31% | 67 pts |
Once non-actor witnesses are actually rendered into prose, Nemotron Super shows NO belief/reality collapse: it tracks false beliefs correctly across chain_length, scramble, coref_pressure, and distractor_events.
Nemotron Nano assumes an entity keeps observing an object's moves even after the story stops placing that entity at the scene -- it over-extends 'last witnessed' into 'still watching', producing a false-belief error the true reasoning chain doesn't support.
Sweeping witness_cutoff (how early an entity's last witnessed exposure to an object is cut off from the object's later moves) reproduces the same over-generalization systematically: at witness_cutoff=0.7, belief/truth divergence hits 94%.
Ultra misresolves pronouns toward the story's protagonist specifically -- an ablation that swaps pronoun/name surface form while holding the answer key fixed separates this cleanly from ordinary coreference noise in 6/6 ablated cases.
Under adversarial (foil-selected) belief queries at high trap density, a model can abstain/answer 'unknown' rather than commit to a wrong-but-plausible foil answer, on questions with a fully determinable ground-truth belief. First observed on Ultra (2026-09-17, 2 unreplicated cases); a 2026-09-20 replication attempt on a fresh Ultra candidate carrying the same trap signature evaporated under proper token budget (28000 vs the 16000 'ultra'-class ceiling: 5/5 completed cleanly, 0/5 abstained) -- the original Ultra 'unknown' answers were very plausibly reasoning-chain truncation misread as abstention, not genuine behavior. The SAME replication pass found the real thing on Super instead: 7 adversarial_belief FAILs under the identical trap combination, all clean completions (finish_reason=stop, 500-1900 tokens, no truncation risk on Super's 4000-token budget). Resample-classified per this project's standard noise gate: 3 of 7 were noise (majority/unanimous PASS on resample -- filtered out), 4 survive stable, 3 of those answering 'unknown' on all 4 independent draws (original + 3 resamples).
Escalating every difficulty dim to its generator ceiling (chain_length 25, scramble 1.0, coref_pressure 8, distractor_events 40, filler_events 60, witness_cutoff 0.9 simultaneously) never broke Ultra. Adversarial seed search at a much shorter chain_length=2 -- picking the single world among many candidates where a plausible-but-wrong foil answer most directly competes with the truth -- did break it.
Retractions (1)
First read of camp-belief-nemotron-super-001 (baseline only 3/5) looked like a real ~60%% capability ceiling on false belief tracking, independent of surface difficulty.