Trials · state
Every trial behind one square of the capability map: Gemma 3 27B on state.
46
0 errors
Trials in this cell
57%
26/46 scored
Pass rate
20
0 noise excluded
Real failures
$0.0022
Cell cost
Trials
| kind | verdict | realized dims | seeded traps | run | |
|---|---|---|---|---|---|
| state | ✓PASSanswered "Kitchen" | entity_state=4 temporal=0 coreference=3 distractors=2 context_length=8 witness_gap=0 | smoke-gemma-state | transcript → | |
| state | ✓PASSanswered "Utility closet" | entity_state=2 temporal=0 coreference=3 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "Courtyard" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "West wing"verified-stable | entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "Photo studio" | entity_state=2 temporal=0 coreference=3 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "Storage room" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "Courtyard" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "Cafeteria" | entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "Storage room" | entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "Observation deck" | entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "loading dock" | entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "Boiler room" | entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "mail room"verified-stable | entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "Cafeteria" | entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "mail room" | entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "Server room"verified-stable | entity_state=16 temporal=0 coreference=3 distractors=2 context_length=20 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "West wing" | entity_state=23 temporal=0 coreference=1 distractors=2 context_length=27 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "The archive room." | entity_state=23 temporal=0 coreference=3 distractors=2 context_length=27 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "Wine cellar"verified-stable | entity_state=23 temporal=0 coreference=1 distractors=2 context_length=27 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "Security office"verified-stable | entity_state=23 temporal=0 coreference=1 distractors=2 context_length=27 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "Print room" | entity_state=23 temporal=0 coreference=1 distractors=2 context_length=27 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "Storage room" | entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "atrium"verified-stable | entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "rooftop terrace" | entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "wine cellar"verified-stable | entity_state=30 temporal=0 coreference=3 distractors=2 context_length=34 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "Archive room"verified-stable | entity_state=30 temporal=0 coreference=3 distractors=2 context_length=34 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "Photo studio" | entity_state=26 temporal=0 coreference=1 distractors=2 context_length=30 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "north stairwell"verified-stable | entity_state=26 temporal=0 coreference=1 distractors=2 context_length=30 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "print room" | entity_state=26 temporal=0 coreference=1 distractors=2 context_length=30 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "library" | entity_state=26 temporal=0 coreference=3 distractors=2 context_length=30 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "Courtyard"verified-stable | entity_state=26 temporal=0 coreference=3 distractors=2 context_length=30 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "Garage"verified-stable | entity_state=28 temporal=0 coreference=3 distractors=2 context_length=32 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "photo studio"verified-stable | entity_state=28 temporal=0 coreference=1 distractors=2 context_length=32 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "Kitchen"verified-stable | entity_state=28 temporal=0 coreference=3 distractors=2 context_length=32 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "West wing" | entity_state=28 temporal=0 coreference=1 distractors=2 context_length=32 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "mail room" | entity_state=28 temporal=0 coreference=3 distractors=2 context_length=32 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "Workshop" | entity_state=27 temporal=0 coreference=1 distractors=2 context_length=31 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "Storage room"verified-stable | entity_state=27 temporal=0 coreference=3 distractors=2 context_length=31 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "library"verified-stable | entity_state=27 temporal=0 coreference=1 distractors=2 context_length=31 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "server room"verified-stable | entity_state=27 temporal=0 coreference=1 distractors=2 context_length=31 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "break room"verified-stable | entity_state=27 temporal=0 coreference=1 distractors=2 context_length=31 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "Library"verified-stable | entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "workshop" | entity_state=30 temporal=0 coreference=3 distractors=2 context_length=34 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "Storage room" | entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "boiler room"verified-stable | entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "Gym"verified-stable | entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0 | camp-full_map-gemma-001/state | transcript → |