Findings
The capability map, derived live from 9,472 recorded trials: what each model does well, where exactly it breaks, and how far you should trust each claim. Every number here is computed from the store when this page loads — replicate a finding and it updates.
| model | query kind | difficulty region (high tercile) | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| state | order | belief | adversarial_belief | tool_plan | interactive | code_trace | code_repair | code_write | entity_state↑ | temporal↑ | coreference↑ | distractors↑ | context_length↑ | witness_gap↑ | |
| Gemma 4 31B QAT | 100%n=125 | 95%n=209 | 60%n=638 | 62%n=260 | 81%n=865 | 57%n=624 | 97%n=825 | 33%n=67 | —n=0 | 66%n=508 | 51%n=266 | 67%n=423 | 48%n=293 | 64%n=572 | 31%n=115 |
| Gemma 3 27B | 57%n=46 | 99%n=80 | 49%n=162 | 43%n=145 | 55%n=398 | 91%n=118 | 75%n=246 | —n=0 | —n=0 | 56%n=102 | 67%n=67 | 55%n=134 | 59%n=92 | 57%n=162 | —n=0 |
| Nemotron 3 Nano 30B | 100%n=90 | 99%n=158 | 92%n=564 | 91%n=306 | 66%n=626 | 78%n=121 | —n=0 | —n=0 | —n=0 | 89%n=342 | 94%n=226 | 93%n=392 | 93%n=227 | 92%n=389 | 88%n=75 |
| Nemotron 3 Ultra 550B | 100%n=30 | 100%n=80 | 99%n=183 | 99%n=128 | 89%n=291 | —n=0 | —n=0 | —n=0 | —n=0 | 99%n=132 | 98%n=115 | 99%n=158 | 99%n=116 | 99%n=166 | 98%n=51 |
| Nemotron 3 Super 120B | 97%n=94 | 100%n=79 | 98%n=290 | 96%n=158 | 86%n=537 | 97%n=137 | —n=0 | —n=0 | —n=0 | 97%n=202 | 95%n=172 | 97%n=219 | 99%n=143 | 98%n=241 | 98%n=52 |
| gpt-oss-120b | 100%n=60 | 100%n=79 | 97%n=159 | 98%n=42 | —n=0 | —n=0 | —n=0 | —n=0 | —n=0 | 99%n=105 | 96%n=56 | 98%n=98 | 98%n=45 | 99%n=125 | 80%n=5 |
Nemotron 3 Nano 30B
What it does well (2)
NVIDIA-Nemotron-3-Nano-30B-A3B passes state queries cleanly: 90/90, no failures at all.
NVIDIA-Nemotron-3-Nano-30B-A3B handles order queries: 99% pass (156/158 scored).
Where it fails (2)
NVIDIA-Nemotron-3-Nano-30B-A3B fails tool_plan queries: 66% pass (413/626 scored, 213 real failures).
Traps that fired
Example failures
NVIDIA-Nemotron-3-Nano-30B-A3B fails interactive queries: 78% pass (94/121 scored, 27 real failures).
Traps that fired
Example failures
| capability | strongest | rate | weakest | rate | spread |
|---|---|---|---|---|---|
| state | gpt-oss-120b | 100% | Gemma 3 27B | 57% | 43 pts |
| order | gpt-oss-120b | 100% | Gemma 4 31B QAT | 95% | 5 pts |
| belief | Nemotron 3 Ultra 550B | 99% | Gemma 3 27B | 49% | 50 pts |
| adversarial_belief | Nemotron 3 Ultra 550B | 99% | Gemma 3 27B | 43% | 56 pts |
| tool_plan | Nemotron 3 Ultra 550B | 89% | Gemma 3 27B | 55% | 34 pts |
| interactive | Nemotron 3 Super 120B | 97% | Gemma 4 31B QAT | 57% | 40 pts |
| code_trace | Gemma 4 31B QAT | 97% | Gemma 3 27B | 75% | 22 pts |
| entity_state (high) | Nemotron 3 Ultra 550B | 99% | Gemma 3 27B | 56% | 43 pts |
| temporal (high) | Nemotron 3 Ultra 550B | 98% | Gemma 4 31B QAT | 51% | 48 pts |
| coreference (high) | Nemotron 3 Ultra 550B | 99% | Gemma 3 27B | 55% | 44 pts |
| distractors (high) | Nemotron 3 Ultra 550B | 99% | Gemma 4 31B QAT | 48% | 51 pts |
| context_length (high) | Nemotron 3 Ultra 550B | 99% | Gemma 3 27B | 57% | 42 pts |
| witness_gap (high) | Nemotron 3 Super 120B | 98% | Gemma 4 31B QAT | 31% | 67 pts |
Once non-actor witnesses are actually rendered into prose, Nemotron Super shows NO belief/reality collapse: it tracks false beliefs correctly across chain_length, scramble, coref_pressure, and distractor_events.
Nemotron Nano assumes an entity keeps observing an object's moves even after the story stops placing that entity at the scene -- it over-extends 'last witnessed' into 'still watching', producing a false-belief error the true reasoning chain doesn't support.
Sweeping witness_cutoff (how early an entity's last witnessed exposure to an object is cut off from the object's later moves) reproduces the same over-generalization systematically: at witness_cutoff=0.7, belief/truth divergence hits 94%.
Ultra misresolves pronouns toward the story's protagonist specifically -- an ablation that swaps pronoun/name surface form while holding the answer key fixed separates this cleanly from ordinary coreference noise in 6/6 ablated cases.
Under adversarial (foil-selected) belief queries at high trap density, a model can abstain/answer 'unknown' rather than commit to a wrong-but-plausible foil answer, on questions with a fully determinable ground-truth belief. First observed on Ultra (2026-09-17, 2 unreplicated cases); a 2026-09-20 replication attempt on a fresh Ultra candidate carrying the same trap signature evaporated under proper token budget (28000 vs the 16000 'ultra'-class ceiling: 5/5 completed cleanly, 0/5 abstained) -- the original Ultra 'unknown' answers were very plausibly reasoning-chain truncation misread as abstention, not genuine behavior. The SAME replication pass found the real thing on Super instead: 7 adversarial_belief FAILs under the identical trap combination, all clean completions (finish_reason=stop, 500-1900 tokens, no truncation risk on Super's 4000-token budget). Resample-classified per this project's standard noise gate: 3 of 7 were noise (majority/unanimous PASS on resample -- filtered out), 4 survive stable, 3 of those answering 'unknown' on all 4 independent draws (original + 3 resamples).
Escalating every difficulty dim to its generator ceiling (chain_length 25, scramble 1.0, coref_pressure 8, distractor_events 40, filler_events 60, witness_cutoff 0.9 simultaneously) never broke Ultra. Adversarial seed search at a much shorter chain_length=2 -- picking the single world among many candidates where a plausible-but-wrong foil answer most directly competes with the truth -- did break it.
Retractions (1)
First read of camp-belief-nemotron-super-001 (baseline only 3/5) looked like a real ~60%% capability ceiling on false belief tracking, independent of surface difficulty.