Data to …… trials

Findings

The capability map, derived live from 9,472 recorded trials: what each model does well, where exactly it breaks, and how far you should trust each claim. Every number here is computed from the store when this page loads — replicate a finding and it updates.

Map a model →
6
Models mapped
36
Derived strengths
18
Derived weaknesses
6
6 confirmed · 0 open
Documented findings
Capability map 6 models × 15 capabilities = 90 cells

Pass rate over SCORED trials (PASS+FAIL; infra ERRORs and resample-noise failures excluded, same denominator as every run's verdict panel). Every cell is a link into the trials behind it. Dimension columns report the HIGH region only -- the top tercile of values observed across this whole store, so a column means the same thing on every row.

under 50%50-75%75-90%90-97%97%+no data
model query kind difficulty region (high tercile)
stateorderbeliefadversarial_belieftool_planinteractivecode_tracecode_repaircode_write entity_state↑temporal↑coreference↑distractors↑context_length↑witness_gap↑
Gemma 4 31B QAT73% overall · 3,919 trials100%n=12595%n=20960%n=63862%n=26081%n=86557%n=62497%n=82533%n=67—n=066%n=50851%n=26667%n=42348%n=29364%n=57231%n=115
Gemma 3 27B63% overall · 1,198 trials57%n=4699%n=8049%n=16243%n=14555%n=39891%n=11875%n=246—n=0—n=056%n=10267%n=6755%n=13459%n=9257%n=162—n=0
Nemotron 3 Nano 30B83% overall · 1,961 trials100%n=9099%n=15892%n=56491%n=30666%n=62678%n=121—n=0—n=0—n=089%n=34294%n=22693%n=39293%n=22792%n=38988%n=75
Nemotron 3 Ultra 550B95% overall · 727 trials100%n=30100%n=8099%n=18399%n=12889%n=291—n=0—n=0—n=0—n=099%n=13298%n=11599%n=15899%n=11699%n=16698%n=51
Nemotron 3 Super 120B93% overall · 1,325 trials97%n=94100%n=7998%n=29096%n=15886%n=53797%n=137—n=0—n=0—n=097%n=20295%n=17297%n=21999%n=14398%n=24198%n=52
gpt-oss-120b98% overall · 342 trials100%n=60100%n=7997%n=15998%n=42—n=0—n=0—n=0—n=0—n=099%n=10596%n=5698%n=9898%n=4599%n=12580%n=5
All modelsGemma 4 31B QATGemma 3 27BNemotron 3 Nano 30BNemotron 3 Ultra 550BNemotron 3 Super 120Bgpt-oss-120b

Gemma 4 31B QAT · 3,919 trials · 73% overall · $3.6876 spent

What it does well (3)

state — gemma-4-31b-it-qat-w4a16-ct no failures

gemma-4-31b-it-qat-w4a16-ct passes state queries cleanly: 125/125, no failures at all.

no failures recorded here -- nothing to re-sample

Hardest slice inside this strength: temporal = 0 at 100% (125/125).

see all 125 trials →

code_trace — gemma-4-31b-it-qat-w4a16-ct unverified

gemma-4-31b-it-qat-w4a16-ct handles code_trace queries: 97% pass (797/825 scored).

28 failure(s): 28 not yet resample-checked

see all 825 trials →

order — gemma-4-31b-it-qat-w4a16-ct unverified

gemma-4-31b-it-qat-w4a16-ct handles order queries: 95% pass (199/209 scored).

10 failure(s): 1 survived resampling, 9 not yet resample-checked

Hardest slice inside this strength: temporal 0.23–0.73 at 85% (56/66).

see all 209 trials →

Where it fails (7)

witness_gap (high) — gemma-4-31b-it-qat-w4a16-ct unverified

gemma-4-31b-it-qat-w4a16-ct fails trials in the high-witness_gap region: 31% pass (36/115 scored, 79 real failures) — concentrated at distractors = 2, where it drops to 17% (1/6).

Where it fails

  • distractors = 2 → 17% pass (1/6)
  • entity_state 5–12 → 28% pass (26/94)
  • context_length 21–66 → 28% pass (26/94)

By query kind

  • belief → 31% pass (36/115)

Traps that fired

None recorded -- these trials were not adversarial (no traps= in their verifier detail), so there is no trap attribution to report.

Example failures

  • e67dad0b86e8 entity_state=12, temporal=0.36, coreference=9, distractors=19, context_length=66, witness_gap=5 — answered “wine cellar”
  • 318073c8449c entity_state=12, temporal=0.37, coreference=11, distractors=19, context_length=66, witness_gap=5 — answered “reception area”
  • 8863c10b7222 entity_state=12, temporal=0.37, coreference=11, distractors=19, context_length=66, witness_gap=5 — answered “wine cellar”

79 failure(s): 79 not yet resample-checked

replication: not replicated no fresh-seed trials for this finding yet.

see all 115 trials →

code_repair — gemma-4-31b-it-qat-w4a16-ct unverified

gemma-4-31b-it-qat-w4a16-ct fails code_repair queries: 33% pass (22/67 scored, 45 real failures).

Traps that fired

None recorded -- these trials were not adversarial (no traps= in their verifier detail), so there is no trap attribution to report.

Example failures

  • 7497d9b59f73 no realized dims recorded — answered “The bug in the provided code is related to how the `marks` list is ...”
  • 8c35bddbbd6a no realized dims recorded — answered “To fix the bug, we need to examine how the `offset` variable is cal...”
  • 1c08bc71522d no realized dims recorded — answered “To fix the bug, we need to analyze the logic flow and identify wher...”

45 failure(s): 45 not yet resample-checked

replication: not replicated no fresh-seed trials for this finding yet.

see all 67 trials →

distractors (high) — gemma-4-31b-it-qat-w4a16-ct unverified

gemma-4-31b-it-qat-w4a16-ct fails trials in the high-distractors region: 48% pass (142/293 scored, 151 real failures) — concentrated at temporal 0.15–0.58, where it drops to 29% (35/119).

Where it fails

  • temporal 0.15–0.58 → 29% pass (35/119)
  • witness_gap 2–8 → 32% pass (35/109)
  • coreference = 2 → 36% pass (4/11)

By query kind

  • adversarial_belief → 35% pass (14/40)
  • belief → 42% pass (87/208)
  • order → 91% pass (41/45)

Traps that fired

final_true_state ×26, last_mentioned_location ×26, last_witnessed_by_anyone ×26

Example failures

  • e67dad0b86e8 entity_state=12, temporal=0.36, coreference=9, distractors=19, context_length=66, witness_gap=5 — answered “wine cellar”
  • 318073c8449c entity_state=12, temporal=0.37, coreference=11, distractors=19, context_length=66, witness_gap=5 — answered “reception area”
  • 8863c10b7222 entity_state=12, temporal=0.37, coreference=11, distractors=19, context_length=66, witness_gap=5 — answered “wine cellar”

151 failure(s): 28 survived resampling, 123 not yet resample-checked

replication: not replicated no fresh-seed trials for this finding yet.

see all 293 trials →

interactive — gemma-4-31b-it-qat-w4a16-ct unverified

gemma-4-31b-it-qat-w4a16-ct fails interactive queries: 57% pass (354/624 scored, 270 real failures).

Traps that fired

None recorded -- these trials were not adversarial (no traps= in their verifier detail), so there is no trap attribution to report.

Example failures

  • 4a88d5498495 no realized dims recorded — answered “[{"step": 1, "tool": "advance_0", "args": {"service": "svc_0"}, "ok...”
  • 62f8e22bf14d no realized dims recorded — answered “[{"step": 1, "tool": "advance_0", "args": {"service": "svc_0"}, "ok...”
  • 2a704eee35da no realized dims recorded — answered “[{"step": 1, "tool": "advance_0", "args": {"service": "svc_0"}, "ok...”

270 failure(s): 270 not yet resample-checked

replication: not replicated no fresh-seed trials for this finding yet.

see all 624 trials →

belief — gemma-4-31b-it-qat-w4a16-ct unverified

gemma-4-31b-it-qat-w4a16-ct fails belief queries: 60% pass (384/638 scored, 254 real failures) — concentrated at witness_gap 2–8, where it drops to 31% (36/115).

Where it fails

  • witness_gap 2–8 → 31% pass (36/115)
  • temporal 0.15–0.71 → 36% pass (57/160)
  • distractors 4–20 → 42% pass (87/208)

Traps that fired

None recorded -- these trials were not adversarial (no traps= in their verifier detail), so there is no trap attribution to report.

Example failures

  • 3a2038d79ba0 entity_state=4, temporal=0, coreference=1, distractors=2, context_length=9, witness_gap=1 — answered “front office”
  • 4c673e5dbbdc entity_state=4, temporal=0, coreference=3, distractors=2, context_length=9, witness_gap=1 — answered “print room”
  • a3fdbfd4c8e1 entity_state=4, temporal=0, coreference=1, distractors=2, context_length=9, witness_gap=1 — answered “north stairwell”

254 failure(s): 30 survived resampling, 224 not yet resample-checked

replication: not replicated no fresh-seed trials for this finding yet.

see all 638 trials →

adversarial_belief — gemma-4-31b-it-qat-w4a16-ct unverified

gemma-4-31b-it-qat-w4a16-ct fails adversarial_belief queries: 62% pass (160/260 scored, 100 real failures) — concentrated at entity_state 6–10, where it drops to 13% (4/30).

Where it fails

  • entity_state 6–10 → 13% pass (4/30)
  • entity_state = 4 → 20% pass (2/10)
  • context_length 13–25 → 24% pass (12/50)

Traps that fired

final_true_state ×100, last_mentioned_location ×100, last_witnessed_by_anyone ×100, most_frequent_location ×34, query_agent_own_last_move ×34

Example failures

  • 8288454bdd36 entity_state=10, temporal=0, coreference=1, distractors=2, context_length=15, witness_gap=1 — answered “library”
  • b42b5eb1a664 entity_state=10, temporal=0, coreference=3, distractors=2, context_length=15, witness_gap=1 — answered “workshop”
  • a140f419b38b entity_state=10, temporal=0, coreference=1, distractors=2, context_length=15, witness_gap=1 — answered “break room”

100 failure(s): 30 survived resampling, 70 not yet resample-checked

replication: not replicated no fresh-seed trials for this finding yet.

see all 260 trials →

tool_plan — gemma-4-31b-it-qat-w4a16-ct unverified

gemma-4-31b-it-qat-w4a16-ct fails tool_plan queries: 81% pass (700/865 scored, 165 real failures).

Traps that fired

id_collisions_on=1 ×88, poison_traps=1 ×67, poison_traps=0 ×6

Example failures

  • 108a2a077583 no realized dims recorded — answered “[ {"tool": "set_-flag_0", "args": {"flag": "flag_0"}}, {"tool":...”
  • d5834920b193 no realized dims recorded — answered “[ {"tool": "set_-flag_0", "args": {"flag": "flag_0"}}, {"tool":...”
  • 88bf14010198 no realized dims recorded — answered “[ {"tool": "set_-flag_0", "args": {"flag": "flag_0"}}, {"tool":...”

165 failure(s): 165 not yet resample-checked

replication: not replicated no fresh-seed trials for this finding yet.

see all 865 trials →

Gemma 3 27B · 1,198 trials · 63% overall · $1.0673 spent

What it does well (1)

order — google/gemma-3-27b-it verified stable

google/gemma-3-27b-it handles order queries: 99% pass (79/80 scored).

1 failure(s): 1 survived resampling

Hardest slice inside this strength: temporal 0.27–0.73 at 95% (19/20).

see all 80 trials →

Where it fails (7)

adversarial_belief — google/gemma-3-27b-it unverified

google/gemma-3-27b-it fails adversarial_belief queries: 43% pass (62/145 scored, 83 real failures) — concentrated at entity_state 6–10, where it drops to 20% (3/15).

Where it fails

  • entity_state 6–10 → 20% pass (3/15)
  • context_length 8–12 → 36% pass (16/45)
  • context_length 13–25 → 37% pass (13/35)

Traps that fired

final_true_state ×83, last_mentioned_location ×83, last_witnessed_by_anyone ×83, most_frequent_location ×15, query_agent_own_last_move ×15

Example failures

  • 5adc61efbf93 entity_state=8, temporal=0, coreference=1, distractors=2, context_length=13, witness_gap=1 — answered “break room”
  • 0b4b1b017d01 entity_state=8, temporal=0, coreference=1, distractors=2, context_length=13, witness_gap=1 — answered “basement”
  • f724b36c3e7c entity_state=8, temporal=0, coreference=1, distractors=2, context_length=13, witness_gap=1 — answered “greenhouse”

83 failure(s): 30 survived resampling, 53 not yet resample-checked

replication: not replicated no fresh-seed trials for this finding yet.

see all 145 trials →

belief — google/gemma-3-27b-it unverified

google/gemma-3-27b-it fails belief queries: 49% pass (80/162 scored, 82 real failures) — concentrated at coreference = 2, where it drops to 29% (2/7).

Where it fails

  • coreference = 2 → 29% pass (2/7)
  • temporal = 0 → 47% pass (63/135)
  • entity_state = 2 → 48% pass (62/130)

Traps that fired

None recorded -- these trials were not adversarial (no traps= in their verifier detail), so there is no trap attribution to report.

Example failures

  • ba9990b5c1bf entity_state=2, temporal=0.71, coreference=1, distractors=2, context_length=7, witness_gap=1 — answered “security office”
  • 79c0905384fd entity_state=2, temporal=0.71, coreference=1, distractors=2, context_length=7, witness_gap=1 — answered “archive room”
  • d8508660ad3a entity_state=2, temporal=0.71, coreference=1, distractors=2, context_length=7, witness_gap=1 — answered “south stairwell”

82 failure(s): 30 survived resampling, 52 not yet resample-checked

replication: not replicated no fresh-seed trials for this finding yet.

see all 162 trials →

tool_plan — google/gemma-3-27b-it unverified

google/gemma-3-27b-it fails tool_plan queries: 55% pass (219/398 scored, 179 real failures).

Traps that fired

poison_traps=1 ×62, id_collisions_on=1 ×60, poison_traps=0 ×20, dishonest_decoys_on=1 ×10, id_collisions_on=0 ×8

Example failures

  • 6911fdb1c51a no realized dims recorded — answered “```json [ {"tool": "build_art_0", "args": {"artifact": "art_0"}},...”
  • ac4020202f94 no realized dims recorded — answered “```json [ {"tool": "build_art_0", "args": {"artifact": "art_0"}},...”
  • d13046e4df68 no realized dims recorded — answered “```json [ {"tool": "advance_0", "args": {"service": "svc_0"}}, ...”

179 failure(s): 179 not yet resample-checked

replication: not replicated no fresh-seed trials for this finding yet.

see all 398 trials →

coreference (high) — google/gemma-3-27b-it unverified

google/gemma-3-27b-it fails trials in the high-coreference region: 55% pass (74/134 scored, 60 real failures) — concentrated at context_length 8–12, where it drops to 46% (27/59).

Where it fails

  • context_length 8–12 → 46% pass (27/59)
  • temporal = 0 → 54% pass (65/121)
  • distractors = 2 → 54% pass (62/115)

By query kind

  • adversarial_belief → 43% pass (20/46)
  • belief → 52% pass (29/56)
  • order → 100% pass (18/18)
  • state → 50% pass (7/14)

Traps that fired

final_true_state ×26, last_mentioned_location ×26, last_witnessed_by_anyone ×26, most_frequent_location ×4, query_agent_own_last_move ×4

Example failures

  • faa856e24521 entity_state=2, temporal=0, coreference=10, distractors=2, context_length=11, witness_gap=1 — answered “security office”
  • 501631fa6e39 entity_state=2, temporal=0, coreference=10, distractors=2, context_length=11, witness_gap=1 — answered “supply closet”
  • d9c1dbd38d0f entity_state=2, temporal=0, coreference=4, distractors=2, context_length=11, witness_gap=1 — answered “east wing”

60 failure(s): 26 survived resampling, 34 not yet resample-checked

replication: not replicated no fresh-seed trials for this finding yet.

see all 134 trials →

entity_state (high) — google/gemma-3-27b-it unverified

google/gemma-3-27b-it fails trials in the high-entity_state region: 56% pass (57/102 scored, 45 real failures).

Where it fails

  • context_length 13–34 → 54% pass (47/87)
  • coreference 0–1 → 55% pass (42/76)
  • temporal = 0 → 56% pass (53/95)

By query kind

  • adversarial_belief → 20% pass (3/15)
  • belief → 52% pass (14/27)
  • order → 100% pass (20/20)
  • state → 50% pass (20/40)

Traps that fired

final_true_state ×12, last_mentioned_location ×12, last_witnessed_by_anyone ×12, most_frequent_location ×12, query_agent_own_last_move ×12

Example failures

  • 706866355190 entity_state=27, temporal=0, coreference=3, distractors=2, context_length=31, witness_gap=0 — answered “Storage room”
  • fe5a255840ad entity_state=27, temporal=0, coreference=1, distractors=2, context_length=31, witness_gap=0 — answered “library”
  • d4c03de120a7 entity_state=27, temporal=0, coreference=1, distractors=2, context_length=31, witness_gap=0 — answered “server room”

45 failure(s): 24 survived resampling, 21 not yet resample-checked

replication: not replicated no fresh-seed trials for this finding yet.

see all 102 trials →

state — google/gemma-3-27b-it verified stable

google/gemma-3-27b-it fails state queries: 57% pass (26/46 scored, 20 real failures) — concentrated at entity_state 9–30, where it drops to 50% (20/40).

Where it fails

  • entity_state 9–30 → 50% pass (20/40)
  • context_length 13–34 → 50% pass (20/40)
  • coreference = 3 → 50% pass (7/14)

Traps that fired

None recorded -- these trials were not adversarial (no traps= in their verifier detail), so there is no trap attribution to report.

Example failures

  • 706866355190 entity_state=27, temporal=0, coreference=3, distractors=2, context_length=31, witness_gap=0 — answered “Storage room”
  • fe5a255840ad entity_state=27, temporal=0, coreference=1, distractors=2, context_length=31, witness_gap=0 — answered “library”
  • d4c03de120a7 entity_state=27, temporal=0, coreference=1, distractors=2, context_length=31, witness_gap=0 — answered “server room”

20 failure(s): 20 survived resampling

replication: not replicated no fresh-seed trials for this finding yet.

see all 46 trials →

code_trace — google/gemma-3-27b-it unverified

google/gemma-3-27b-it fails code_trace queries: 75% pass (184/246 scored, 62 real failures).

Traps that fired

aliasing_on=1 ×45, aliasing_on=0 ×17

Example failures

  • d94ae6de3be6 no realized dims recorded — answered “Let's trace the execution of the Python code step by step: 1. `ste...”
  • 88696c254612 no realized dims recorded — answered “Let's trace the execution of the Python code step by step to determ...”
  • e77d633f7a81 no realized dims recorded — answered “Let's trace the execution of the Python code step by step to determ...”

62 failure(s): 62 not yet resample-checked

replication: not replicated no fresh-seed trials for this finding yet.

see all 246 trials →

Nemotron 3 Nano 30B · 1,961 trials · 83% overall · $1.3691 spent

What it does well (2)

state — NVIDIA-Nemotron-3-Nano-30B-A3B no failures

NVIDIA-Nemotron-3-Nano-30B-A3B passes state queries cleanly: 90/90, no failures at all.

no failures recorded here -- nothing to re-sample

Hardest slice inside this strength: temporal = 0 at 100% (90/90).

see all 90 trials →

order — NVIDIA-Nemotron-3-Nano-30B-A3B verified stable

NVIDIA-Nemotron-3-Nano-30B-A3B handles order queries: 99% pass (156/158 scored).

2 failure(s): 2 survived resampling

Hardest slice inside this strength: distractors 9–30 at 95% (37/39).

see all 158 trials →

Where it fails (2)

tool_plan — NVIDIA-Nemotron-3-Nano-30B-A3B unverified

NVIDIA-Nemotron-3-Nano-30B-A3B fails tool_plan queries: 66% pass (413/626 scored, 213 real failures).

Traps that fired

poison_traps=1 ×101, id_collisions_on=1 ×79, poison_traps=0 ×16, id_collisions_on=0 ×8

Example failures

  • 1a00508cbda6 no realized dims recorded — answered “[ {"tool":"adva_nce_0","args":{"service":"svc_0"}}, {"tool":"ad...”
  • 8bebb59cab7e no realized dims recorded — answered “[ {"tool":"set_flag_o0","args":{"flag":"flag_0"}}, {"tool":"set...”
  • 254efb17d1de no realized dims recorded — answered “[ {"tool":"set_flag_o0","args":{"flag":"flag_0"}}, {"tool":"set...”

213 failure(s): 213 not yet resample-checked

replication: not replicated no fresh-seed trials for this finding yet.

see all 626 trials →

interactive — NVIDIA-Nemotron-3-Nano-30B-A3B unverified

NVIDIA-Nemotron-3-Nano-30B-A3B fails interactive queries: 78% pass (94/121 scored, 27 real failures).

Traps that fired

None recorded -- these trials were not adversarial (no traps= in their verifier detail), so there is no trap attribution to report.

Example failures

  • 67f49d919030 no realized dims recorded — answered “[{"step": 1, "tool": "set_flag_o0", "args": {"flag": "flag_0"}, "ok...”
  • 773223433c91 no realized dims recorded — answered “[{"step": 1, "tool": "set_flag_o0", "args": {"flag": "flag_0"}, "ok...”
  • df93b25b8c60 no realized dims recorded — answered “[{"step": 1, "tool": "build_art_o0", "args": {"artifact": "art_0"},...”

27 failure(s): 27 not yet resample-checked

replication: not replicated no fresh-seed trials for this finding yet.

see all 121 trials →

Nemotron 3 Ultra 550B · 727 trials · 95% overall · $2.4716 spent

What it does well (10)

order — Nemotron-3-Ultra-550b-a55b no failures

Nemotron-3-Ultra-550b-a55b passes order queries cleanly: 80/80, no failures at all.

no failures recorded here -- nothing to re-sample

Hardest slice inside this strength: witness_gap = 0 at 100% (80/80).

see all 80 trials →

state — Nemotron-3-Ultra-550b-a55b no failures

Nemotron-3-Ultra-550b-a55b passes state queries cleanly: 30/30, no failures at all.

no failures recorded here -- nothing to re-sample

Hardest slice inside this strength: temporal = 0 at 100% (30/30).

see all 30 trials →

context_length (high) — Nemotron-3-Ultra-550b-a55b unverified

Nemotron-3-Ultra-550b-a55b handles the high-context_length region: 99% pass (165/166 scored).

1 failure(s): 1 not yet resample-checked

Hardest slice inside this strength: witness_gap 3–22 at 98% (50/51).

see all 166 trials →

entity_state (high) — Nemotron-3-Ultra-550b-a55b unverified

Nemotron-3-Ultra-550b-a55b handles the high-entity_state region: 99% pass (131/132 scored).

1 failure(s): 1 not yet resample-checked

Hardest slice inside this strength: witness_gap 3–22 at 98% (50/51).

see all 132 trials →

adversarial_belief — Nemotron-3-Ultra-550b-a55b verified stable

Nemotron-3-Ultra-550b-a55b handles adversarial_belief queries: 99% pass (127/128 scored).

1 failure(s): 1 survived resampling

Hardest slice inside this strength: temporal 0.24–0.71 at 96% (22/23).

see all 128 trials →

distractors (high) — Nemotron-3-Ultra-550b-a55b unverified

Nemotron-3-Ultra-550b-a55b handles the high-distractors region: 99% pass (115/116 scored).

1 failure(s): 1 not yet resample-checked

Hardest slice inside this strength: witness_gap 3–22 at 98% (50/51).

see all 116 trials →

belief — Nemotron-3-Ultra-550b-a55b unverified

Nemotron-3-Ultra-550b-a55b handles belief queries: 99% pass (181/183 scored).

2 failure(s): 1 survived resampling, 1 not yet resample-checked

Hardest slice inside this strength: entity_state = 4 at 80% (4/5).

see all 183 trials →

coreference (high) — Nemotron-3-Ultra-550b-a55b unverified

Nemotron-3-Ultra-550b-a55b handles the high-coreference region: 99% pass (156/158 scored).

2 failure(s): 1 survived resampling, 1 not yet resample-checked

Hardest slice inside this strength: context_length 8–12 at 98% (41/42).

see all 158 trials →

temporal (high) — Nemotron-3-Ultra-550b-a55b unverified

Nemotron-3-Ultra-550b-a55b handles the high-temporal region: 98% pass (113/115 scored).

2 failure(s): 1 survived resampling, 1 not yet resample-checked

Hardest slice inside this strength: coreference 0–1 at 98% (45/46).

see all 115 trials →

witness_gap (high) — Nemotron-3-Ultra-550b-a55b unverified

Nemotron-3-Ultra-550b-a55b handles the high-witness_gap region: 98% pass (50/51 scored).

1 failure(s): 1 not yet resample-checked

Hardest slice inside this strength: entity_state 9–25 at 98% (50/51).

see all 51 trials →

Where it fails (1)

tool_plan — Nemotron-3-Ultra-550b-a55b unverified

Nemotron-3-Ultra-550b-a55b fails tool_plan queries: 89% pass (260/291 scored, 31 real failures).

Traps that fired

id_collisions_on=1 ×23, poison_traps=1 ×5

Example failures

  • 23c8b119cfc9 no realized dims recorded — answered “[ {"tool": "set_flag_o0", "args": {"flag": "flag_0"}}, {"tool":...”
  • 0b0ccbb48389 no realized dims recorded — answered “[ {"tool": "set_-flag_0", "args": {"flag": "flag_0"}}, {"tool":...”
  • af48ecdf4be3 no realized dims recorded — answered “[ {"tool": "set_-flag_0", "args": {"flag": "flag_0"}}, {"tool":...”

31 failure(s): 31 not yet resample-checked

replication: not replicated no fresh-seed trials for this finding yet.

see all 291 trials →

Nemotron 3 Super 120B · 1,325 trials · 93% overall · $1.8047 spent

What it does well (11)

order — nemotron-3-super-120b-a12b contains noise

nemotron-3-super-120b-a12b passes order queries cleanly: 79/79, no failures at all.

1 failure(s): 1 were sampling noise and are excluded

Hardest slice inside this strength: witness_gap = 0 at 100% (79/79).

see all 79 trials →

distractors (high) — nemotron-3-super-120b-a12b contains noise

nemotron-3-super-120b-a12b handles the high-distractors region: 99% pass (141/143 scored).

3 failure(s): 1 were sampling noise and are excluded, 2 not yet resample-checked

Hardest slice inside this strength: coreference = 2 at 80% (4/5).

see all 143 trials →

witness_gap (high) — nemotron-3-super-120b-a12b unverified

nemotron-3-super-120b-a12b handles the high-witness_gap region: 98% pass (51/52 scored).

1 failure(s): 1 not yet resample-checked

Hardest slice inside this strength: entity_state 9–25 at 98% (51/52).

see all 52 trials →

context_length (high) — nemotron-3-super-120b-a12b contains noise

nemotron-3-super-120b-a12b handles the high-context_length region: 98% pass (236/241 scored).

8 failure(s): 1 survived resampling, 3 were sampling noise and are excluded, 4 not yet resample-checked

Hardest slice inside this strength: coreference = 2 at 80% (4/5).

see all 241 trials →

belief — nemotron-3-super-120b-a12b contains noise

nemotron-3-super-120b-a12b handles belief queries: 98% pass (283/290 scored).

8 failure(s): 1 survived resampling, 1 were sampling noise and are excluded, 6 not yet resample-checked

Hardest slice inside this strength: entity_state = 4 at 90% (9/10).

see all 290 trials →

coreference (high) — nemotron-3-super-120b-a12b contains noise

nemotron-3-super-120b-a12b handles the high-coreference region: 97% pass (213/219 scored).

7 failure(s): 2 survived resampling, 1 were sampling noise and are excluded, 4 not yet resample-checked

Hardest slice inside this strength: context_length 8–12 at 94% (58/62).

see all 219 trials →

interactive — nemotron-3-super-120b-a12b unverified

nemotron-3-super-120b-a12b handles interactive queries: 97% pass (133/137 scored).

4 failure(s): 4 not yet resample-checked

see all 137 trials →

state — nemotron-3-super-120b-a12b contains noise

nemotron-3-super-120b-a12b handles state queries: 97% pass (91/94 scored).

4 failure(s): 1 survived resampling, 1 were sampling noise and are excluded, 2 not yet resample-checked

Hardest slice inside this strength: coreference = 1 at 96% (70/73).

see all 94 trials →

entity_state (high) — nemotron-3-super-120b-a12b contains noise

nemotron-3-super-120b-a12b handles the high-entity_state region: 97% pass (195/202 scored).

9 failure(s): 1 survived resampling, 2 were sampling noise and are excluded, 6 not yet resample-checked

Hardest slice inside this strength: coreference = 2 at 80% (4/5).

see all 202 trials →

adversarial_belief — nemotron-3-super-120b-a12b contains noise

nemotron-3-super-120b-a12b handles adversarial_belief queries: 96% pass (152/158 scored).

8 failure(s): 6 survived resampling, 2 were sampling noise and are excluded

Hardest slice inside this strength: temporal 0.24–0.71 at 89% (47/53).

see all 158 trials →

temporal (high) — nemotron-3-super-120b-a12b contains noise

nemotron-3-super-120b-a12b handles the high-temporal region: 95% pass (164/172 scored).

10 failure(s): 6 survived resampling, 2 were sampling noise and are excluded, 2 not yet resample-checked

Hardest slice inside this strength: coreference = 2 at 80% (4/5).

see all 172 trials →

Where it fails (1)

tool_plan — nemotron-3-super-120b-a12b unverified

nemotron-3-super-120b-a12b fails tool_plan queries: 86% pass (460/537 scored, 77 real failures).

Traps that fired

id_collisions_on=1 ×74, poison_traps=0 ×1

Example failures

  • acd37ba9ae30 no realized dims recorded — answered “[ {"tool": "set_flag_0", "args": {"flag": "flag_0"}}, {"tool": ...”
  • 3671d94b41dd no realized dims recorded — answered “[ {"tool":"set_flag_0","args":{"flag":"flag_0"}}, {"tool":"set_...”
  • aeb0a404ccf8 no realized dims recorded — answered “[ {"tool":"set_flag_0","args":{"flag":"flag_0"}}, {"tool":"set_...”

77 failure(s): 77 not yet resample-checked

replication: not replicated no fresh-seed trials for this finding yet.

see all 537 trials →

gpt-oss-120b · 342 trials · 98% overall · $0.0867 spent

What it does well (9)

order — gpt-oss-120b contains noise

gpt-oss-120b passes order queries cleanly: 79/79, no failures at all.

1 failure(s): 1 were sampling noise and are excluded

Hardest slice inside this strength: witness_gap = 0 at 100% (79/79).

see all 79 trials →

state — gpt-oss-120b no failures

gpt-oss-120b passes state queries cleanly: 60/60, no failures at all.

no failures recorded here -- nothing to re-sample

Hardest slice inside this strength: temporal = 0 at 100% (60/60).

see all 60 trials →

context_length (high) — gpt-oss-120b verified stable

gpt-oss-120b handles the high-context_length region: 99% pass (124/125 scored).

1 failure(s): 1 survived resampling

Hardest slice inside this strength: temporal 0.52–0.55 at 80% (4/5).

see all 125 trials →

entity_state (high) — gpt-oss-120b verified stable

gpt-oss-120b handles the high-entity_state region: 99% pass (104/105 scored).

1 failure(s): 1 survived resampling

Hardest slice inside this strength: temporal 0.52–0.55 at 80% (4/5).

see all 105 trials →

coreference (high) — gpt-oss-120b verified stable

gpt-oss-120b handles the high-coreference region: 98% pass (96/98 scored).

2 failure(s): 2 survived resampling

Hardest slice inside this strength: witness_gap = 8 at 80% (4/5).

see all 98 trials →

distractors (high) — gpt-oss-120b verified stable

gpt-oss-120b handles the high-distractors region: 98% pass (44/45 scored).

1 failure(s): 1 survived resampling

Hardest slice inside this strength: entity_state = 10 at 80% (4/5).

see all 45 trials →

adversarial_belief — gpt-oss-120b unverified

gpt-oss-120b handles adversarial_belief queries: 98% pass (41/42 scored).

1 failure(s): 1 not yet resample-checked

Hardest slice inside this strength: entity_state = 4 at 80% (4/5).

see all 42 trials →

belief — gpt-oss-120b verified stable

gpt-oss-120b handles belief queries: 97% pass (154/159 scored).

5 failure(s): 5 survived resampling

Hardest slice inside this strength: entity_state = 4 at 80% (4/5).

see all 159 trials →

temporal (high) — gpt-oss-120b contains noise

gpt-oss-120b handles the high-temporal region: 96% pass (54/56 scored).

3 failure(s): 2 survived resampling, 1 were sampling noise and are excluded

Hardest slice inside this strength: entity_state = 10 at 80% (4/5).

see all 56 trials →

Where it fails (0)

No capability fell below 90% over at least 20 scored trials for this model -- nothing here is broken at the difficulty range probed so far.

Model vs model13 contested capabilities
capabilitystrongestrateweakestratespread
stategpt-oss-120b100% n=60Gemma 3 27B57% n=4643 pts
ordergpt-oss-120b100% n=79Gemma 4 31B QAT95% n=2095 pts
beliefNemotron 3 Ultra 550B99% n=183Gemma 3 27B49% n=16250 pts
adversarial_beliefNemotron 3 Ultra 550B99% n=128Gemma 3 27B43% n=14556 pts
tool_planNemotron 3 Ultra 550B89% n=291Gemma 3 27B55% n=39834 pts
interactiveNemotron 3 Super 120B97% n=137Gemma 4 31B QAT57% n=62440 pts
code_traceGemma 4 31B QAT97% n=825Gemma 3 27B75% n=24622 pts
entity_state (high)Nemotron 3 Ultra 550B99% n=132Gemma 3 27B56% n=10243 pts
temporal (high)Nemotron 3 Ultra 550B98% n=115Gemma 4 31B QAT51% n=26648 pts
coreference (high)Nemotron 3 Ultra 550B99% n=158Gemma 3 27B55% n=13444 pts
distractors (high)Nemotron 3 Ultra 550B99% n=116Gemma 4 31B QAT48% n=29351 pts
context_length (high)Nemotron 3 Ultra 550B99% n=166Gemma 3 27B57% n=16242 pts
witness_gap (high)Nemotron 3 Super 120B98% n=52Gemma 4 31B QAT31% n=11567 pts
Documented findings curated · not auto-derived

These are causal results from docs/lab-log.md that no aggregate over pass/fail columns can reproduce — a pronoun ablation, a renderer-bug retraction, a strategic comparison between two campaign designs. They are written by hand, dated, and tagged with the run_ids they came from, and they do NOT update when the store does.

Belief-to-reality collapse (witness-rendering bug, corrected) confirmed

Once non-actor witnesses are actually rendered into prose, Nemotron Super shows NO belief/reality collapse: it tracks false beliefs correctly across chain_length, scramble, coref_pressure, and distractor_events.

evidence: camp-belief-nemotron-super-002: 105 calls, baseline 3/3, no boundary found on any of the four swept dims.

date: 2026-09-17 · run_id(s): camp-belief-nemotron-super-002

Nano continuity-of-observation over-generalization confirmed

Nemotron Nano assumes an entity keeps observing an object's moves even after the story stops placing that entity at the scene -- it over-extends 'last witnessed' into 'still watching', producing a false-belief error the true reasoning chain doesn't support.

evidence: camp-belief-nemotron-nano-001: 105 calls, 93% pass, 7 real failures verified against regenerated ground truth -- the first confirmed tier discrimination between Super and Nano.

date: 2026-09-17 · run_id(s): camp-belief-nemotron-nano-001

witness_cutoff generalizes the Nano failure mode confirmed

Sweeping witness_cutoff (how early an entity's last witnessed exposure to an object is cut off from the object's later moves) reproduces the same over-generalization systematically: at witness_cutoff=0.7, belief/truth divergence hits 94%.

evidence: docs/lab-log.md 2026-09-17 entry, commit 313e3b2.

date: 2026-09-17 · run_id(s): -

Ultra protagonist-continuity pronoun bias confirmed

Ultra misresolves pronouns toward the story's protagonist specifically -- an ablation that swaps pronoun/name surface form while holding the answer key fixed separates this cleanly from ordinary coreference noise in 6/6 ablated cases.

evidence: 5 Super belief failures verified real against regenerated ground truth: 4 were pronoun misresolution, 1 was a continuity over-generalization (first recorded Super failures) -- ablation catalog run gave 6/6 separation.

date: 2026-09-17 · run_id(s): -

Trap-pressure abstention (confirmed on Super, not Ultra) confirmed

Under adversarial (foil-selected) belief queries at high trap density, a model can abstain/answer 'unknown' rather than commit to a wrong-but-plausible foil answer, on questions with a fully determinable ground-truth belief. First observed on Ultra (2026-09-17, 2 unreplicated cases); a 2026-09-20 replication attempt on a fresh Ultra candidate carrying the same trap signature evaporated under proper token budget (28000 vs the 16000 'ultra'-class ceiling: 5/5 completed cleanly, 0/5 abstained) -- the original Ultra 'unknown' answers were very plausibly reasoning-chain truncation misread as abstention, not genuine behavior. The SAME replication pass found the real thing on Super instead: 7 adversarial_belief FAILs under the identical trap combination, all clean completions (finish_reason=stop, 500-1900 tokens, no truncation risk on Super's 4000-token budget). Resample-classified per this project's standard noise gate: 3 of 7 were noise (majority/unanimous PASS on resample -- filtered out), 4 survive stable, 3 of those answering 'unknown' on all 4 independent draws (original + 3 resamples).

evidence: camp-full_map-super-002/adversarial_belief: 4 stable cases (pass rate <=0.33 across original+3 resamples), verified against regenerated ground truth on one case (prompt reproduces byte-identical, answer_key='the workshop' -- a normal determinable belief). camp-full_map-ultra-002: the debunking resample (5x at max_tokens=28000 on the fresh Ultra candidate) is not itself a stored campaign run -- see docs/lab-log.md 2026-09-20 entries for both the Ultra debunking and the Super confirmation, including the full per-record resample table.

date: 2026-09-20 · run_id(s): camp-full_map-super-002/adversarial_belief

Trap-selection breaks Ultra where magnitude escalation didn't confirmed

Escalating every difficulty dim to its generator ceiling (chain_length 25, scramble 1.0, coref_pressure 8, distractor_events 40, filler_events 60, witness_cutoff 0.9 simultaneously) never broke Ultra. Adversarial seed search at a much shorter chain_length=2 -- picking the single world among many candidates where a plausible-but-wrong foil answer most directly competes with the truth -- did break it.

evidence: run_breaking_point_campaign (escalation ray, t up to 1.0): no breaking point found. run_adversarial_belief_campaign at chain_length=2: confirmed failures. This is the strategic headline result: difficulty magnitude and trap salience are different axes, and only the second one moved Ultra.

date: 2026-09-17 · run_id(s): -

Retractions 1 withdrawn
Retractions (1)

Kept deliberately, not deleted -- self-skepticism is a feature of this instrument, not a blemish. A retracted finding means the audit trail caught its own mistake before it shipped as a public claim.

[RETRACTED] Nemotron Super has an intrinsic false-belief floor retracted

First read of camp-belief-nemotron-super-001 (baseline only 3/5) looked like a real ~60%% capability ceiling on false belief tracking, independent of surface difficulty.

evidence: Root cause was a renderer bug, not a model limitation: cartographer/world/simulate.py assigned beliefs to every witness, but cartographer/render/templates.py only ever interpolated the acting entity into prose -- non-actor witnesses were silently never mentioned in the story the model actually read. The model was correctly saying 'unknown' about entities it was never told were present. Fixed in commit 5bd297f; corrected re-run (camp-belief-nemotron-super-002) reversed the result completely -- see the corrected finding above.

date: 2026-09-17 · run_id(s): camp-belief-nemotron-super-001