Trials · state
Every trial behind one square of the capability map: Nemotron 3 Super 120B on state.
95
0 errors
Trials in this cell
97%
91/94 scored
Pass rate
3
1 noise excluded
Real failures
$0.0334
Cell cost
Trials
| kind | verdict | realized dims | seeded traps | run | |
|---|---|---|---|---|---|
| state | ✓PASSanswered "storage room" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 | camp-state-nemotron-super-001 | transcript → | |
| state | ✓PASSanswered "courtyard" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 | camp-state-nemotron-super-001 | transcript → | |
| state | ✓PASSanswered "courtyard" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 | camp-state-nemotron-super-001 | transcript → | |
| state | ✓PASSanswered "photo studio" | entity_state=2 temporal=0 coreference=3 distractors=2 context_length=6 | camp-state-nemotron-super-001 | transcript → | |
| state | ✓PASSanswered "utility closet" | entity_state=2 temporal=0 coreference=3 distractors=2 context_length=6 | camp-state-nemotron-super-001 | transcript → | |
| state | ✓PASSanswered "boiler room" | entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 | camp-state-nemotron-super-001 | transcript → | |
| state | ✓PASSanswered "cafeteria" | entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 | camp-state-nemotron-super-001 | transcript → | |
| state | ✓PASSanswered "storage room" | entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 | camp-state-nemotron-super-001 | transcript → | |
| state | ✓PASSanswered "observation deck" | entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 | camp-state-nemotron-super-001 | transcript → | |
| state | ✓PASSanswered "boiler room" | entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 | camp-state-nemotron-super-001 | transcript → | |
| state | ✓PASSanswered "loading dock" | entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 | camp-state-nemotron-super-001 | transcript → | |
| state | ✓PASSanswered "east wing" | entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 | camp-state-nemotron-super-001 | transcript → | |
| state | ✓PASSanswered "cafeteria" | entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 | camp-state-nemotron-super-001 | transcript → | |
| state | ✓PASSanswered "mail room" | entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 | camp-state-nemotron-super-001 | transcript → | |
| state | ✓PASSanswered "south stairwell" | entity_state=16 temporal=0 coreference=3 distractors=2 context_length=20 | camp-state-nemotron-super-001 | transcript → | |
| state | ✓PASSanswered "archive room" | entity_state=23 temporal=0 coreference=3 distractors=2 context_length=27 | camp-state-nemotron-super-001 | transcript → | |
| state | ✓PASSanswered "west wing" | entity_state=23 temporal=0 coreference=1 distractors=2 context_length=27 | camp-state-nemotron-super-001 | transcript → | |
| state | ?FAILanswered "wine cellar"unverified | entity_state=23 temporal=0 coreference=1 distractors=2 context_length=27 | camp-state-nemotron-super-001 | transcript → | |
| state | ✓PASSanswered "mail room" | entity_state=23 temporal=0 coreference=1 distractors=2 context_length=27 | camp-state-nemotron-super-001 | transcript → | |
| state | ✓PASSanswered "print room" | entity_state=23 temporal=0 coreference=1 distractors=2 context_length=27 | camp-state-nemotron-super-001 | transcript → | |
| state | ✓PASSanswered "storage room" | entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 | camp-state-nemotron-super-001 | transcript → | |
| state | ✓PASSanswered "rooftop terrace" | entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 | camp-state-nemotron-super-001 | transcript → | |
| state | ✓PASSanswered "photo studio" | entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 | camp-state-nemotron-super-001 | transcript → | |
| state | ✓PASSanswered "photo studio" | entity_state=30 temporal=0 coreference=3 distractors=2 context_length=34 | camp-state-nemotron-super-001 | transcript → | |
| state | ✓PASSanswered "east wing" | entity_state=30 temporal=0 coreference=3 distractors=2 context_length=34 | camp-state-nemotron-super-001 | transcript → | |
| state | ✓PASSanswered "cafeteria" | entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 | camp-state-nemotron-super-001 | transcript → | |
| state | ✓PASSanswered "photo studio" | entity_state=30 temporal=0 coreference=3 distractors=2 context_length=34 | camp-state-nemotron-super-001 | transcript → | |
| state | ✓PASSanswered "print room" | entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 | camp-state-nemotron-super-001 | transcript → | |
| state | ✓PASSanswered "supply closet" | entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 | camp-state-nemotron-super-001 | transcript → | |
| state | ✓PASSanswered "south stairwell" | entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 | camp-state-nemotron-super-001 | transcript → | |
| state | ✓PASSanswered "courtyard" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-super-002/state | transcript → | |
| state | ✓PASSanswered "utility closet" | entity_state=2 temporal=0 coreference=3 distractors=2 context_length=6 witness_gap=0 | camp-full_map-super-002/state | transcript → | |
| state | ✓PASSanswered "photo studio" | entity_state=2 temporal=0 coreference=3 distractors=2 context_length=6 witness_gap=0 | camp-full_map-super-002/state | transcript → | |
| state | ✓PASSanswered "courtyard" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-super-002/state | transcript → | |
| state | ✓PASSanswered "boiler room" | entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0 | camp-full_map-super-002/state | transcript → | |
| state | ✓PASSanswered "storage room" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-super-002/state | transcript → | |
| state | ✓PASSanswered "cafeteria" | entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0 | camp-full_map-super-002/state | transcript → | |
| state | ✓PASSanswered "boiler room" | entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0 | camp-full_map-super-002/state | transcript → | |
| state | ✓PASSanswered "observation deck" | entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0 | camp-full_map-super-002/state | transcript → | |
| state | ✓PASSanswered "storage room" | entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0 | camp-full_map-super-002/state | transcript → | |
| state | ✓PASSanswered "loading dock" | entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0 | camp-full_map-super-002/state | transcript → | |
| state | ✓PASSanswered "east wing" | entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0 | camp-full_map-super-002/state | transcript → | |
| state | ✓PASSanswered "cafeteria" | entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0 | camp-full_map-super-002/state | transcript → | |
| state | ✕FAILanswered "photo studio"noise | entity_state=16 temporal=0 coreference=3 distractors=2 context_length=20 witness_gap=0 | camp-full_map-super-002/state | transcript → | |
| state | ✓PASSanswered "mail room" | entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0 | camp-full_map-super-002/state | transcript → | |
| state | ✓PASSanswered "Archive Room" | entity_state=23 temporal=0 coreference=3 distractors=2 context_length=27 witness_gap=0 | camp-full_map-super-002/state | transcript → | |
| state | ✓PASSanswered "west wing" | entity_state=23 temporal=0 coreference=1 distractors=2 context_length=27 witness_gap=0 | camp-full_map-super-002/state | transcript → | |
| state | ✕FAILanswered "wine cellar"verified-stable | entity_state=23 temporal=0 coreference=1 distractors=2 context_length=27 witness_gap=0 | camp-full_map-super-002/state | transcript → | |
| state | ✓PASSanswered "print room" | entity_state=23 temporal=0 coreference=1 distractors=2 context_length=27 witness_gap=0 | camp-full_map-super-002/state | transcript → | |
| state | ✓PASSanswered "mail room" | entity_state=23 temporal=0 coreference=1 distractors=2 context_length=27 witness_gap=0 | camp-full_map-super-002/state | transcript → | |
| state | ✓PASSanswered "rooftop terrace" | entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0 | camp-full_map-super-002/state | transcript → | |
| state | ✓PASSanswered "photo studio" | entity_state=30 temporal=0 coreference=3 distractors=2 context_length=34 witness_gap=0 | camp-full_map-super-002/state | transcript → | |
| state | ✓PASSanswered "east wing" | entity_state=30 temporal=0 coreference=3 distractors=2 context_length=34 witness_gap=0 | camp-full_map-super-002/state | transcript → | |
| state | ✓PASSanswered "storage room" | entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0 | camp-full_map-super-002/state | transcript → | |
| state | ✓PASSanswered "photo studio" | entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0 | camp-full_map-super-002/state | transcript → | |
| state | ✓PASSanswered "print room" | entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0 | camp-full_map-super-002/state | transcript → | |
| state | ✓PASSanswered "photo studio" | entity_state=30 temporal=0 coreference=3 distractors=2 context_length=34 witness_gap=0 | camp-full_map-super-002/state | transcript → | |
| state | ✓PASSanswered "supply closet" | entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0 | camp-full_map-super-002/state | transcript → | |
| state | ✓PASSanswered "south stairwell" | entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0 | camp-full_map-super-002/state | transcript → | |
| state | ✓PASSanswered "cafeteria" | entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0 | camp-full_map-super-002/state | transcript → | |
| state | ✓PASSanswered "utility closet" | entity_state=2 temporal=0 coreference=3 distractors=2 context_length=6 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ✓PASSanswered "courtyard" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ✓PASSanswered "photo studio" | entity_state=2 temporal=0 coreference=3 distractors=2 context_length=6 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ✓PASSanswered "storage room" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ✓PASSanswered "courtyard" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ✓PASSanswered "courtyard" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ✓PASSanswered "cafeteria" | entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ✓PASSanswered "utility closet" | entity_state=2 temporal=0 coreference=3 distractors=2 context_length=6 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ✓PASSanswered "storage room" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ✓PASSanswered "photo studio" | entity_state=2 temporal=0 coreference=3 distractors=2 context_length=6 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ✓PASSanswered "courtyard" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ✓PASSanswered "boiler room" | entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ✓PASSanswered "boiler room" | entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ✓PASSanswered "cafeteria" | entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ✓PASSanswered "boiler room" | entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ✓PASSanswered "observation deck" | entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ✓PASSanswered "east wing" | entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ✓PASSanswered "storage room" | entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ✓PASSanswered "observation deck" | entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ✓PASSanswered "south stairwell" | entity_state=16 temporal=0 coreference=3 distractors=2 context_length=20 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ✓PASSanswered "east wing" | entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ✓PASSanswered "mail room" | entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ✓PASSanswered "loading dock" | entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ✓PASSanswered "archive room" | entity_state=23 temporal=0 coreference=3 distractors=2 context_length=27 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ✓PASSanswered "atrium" | entity_state=23 temporal=0 coreference=1 distractors=2 context_length=27 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ✓PASSanswered "archive room" | entity_state=23 temporal=0 coreference=3 distractors=2 context_length=27 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ?FAILanswered "loading dock"unverified | entity_state=23 temporal=0 coreference=1 distractors=2 context_length=27 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ✓PASSanswered "west wing" | entity_state=23 temporal=0 coreference=1 distractors=2 context_length=27 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ✓PASSanswered "atrium" | entity_state=23 temporal=0 coreference=1 distractors=2 context_length=27 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ✓PASSanswered "photo studio" | entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ✓PASSanswered "print room" | entity_state=23 temporal=0 coreference=1 distractors=2 context_length=27 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ✓PASSanswered "storage room" | entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ✓PASSanswered "rooftop terrace" | entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ✓PASSanswered "mail room" | entity_state=23 temporal=0 coreference=1 distractors=2 context_length=27 witness_gap=0 | camp-nemrender-showcase-001 | transcript → | |
| state | ✓PASSanswered "photo studio" | entity_state=30 temporal=0 coreference=3 distractors=2 context_length=34 witness_gap=0 | camp-nemrender-showcase-001 | transcript → |