Data to …… trials

Trials · state

Every trial behind one square of the capability map: Gemma 4 31B QAT on state.

Gemma 4 31B QAT state no-failures

125
0 errors
Trials in this cell
100%
125/125 scored
Pass rate
0
0 noise excluded
Real failures
$0.0048
Cell cost
Trials125/125100%
kindverdictrealized dimsseeded trapsrun
state✓PASSanswered "kitchen"entity_state=4 temporal=0 coreference=3 distractors=2 context_length=8 witness_gap=0smoke-gemma4qat-statetranscript →
state✓PASSanswered "utility closet"entity_state=2 temporal=0 coreference=3 distractors=2 context_length=6 witness_gap=0camp-full_map-gemma4qat-001/statetranscript →
state✓PASSanswered "Courtyard"entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0camp-full_map-gemma4qat-001/statetranscript →
state✓PASSanswered "storage room"entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0camp-full_map-gemma4qat-001/statetranscript →
state✓PASSanswered "Boiler room"entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0camp-full_map-gemma4qat-001/statetranscript →
state✓PASSanswered "Courtyard"entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0camp-full_map-gemma4qat-001/statetranscript →
state✓PASSanswered "photo studio"entity_state=2 temporal=0 coreference=3 distractors=2 context_length=6 witness_gap=0camp-full_map-gemma4qat-001/statetranscript →
state✓PASSanswered "Storage room"entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0camp-full_map-gemma4qat-001/statetranscript →
state✓PASSanswered "observation deck"entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0camp-full_map-gemma4qat-001/statetranscript →
state✓PASSanswered "Cafeteria"entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0camp-full_map-gemma4qat-001/statetranscript →
state✓PASSanswered "Boiler room"entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0camp-full_map-gemma4qat-001/statetranscript →
state✓PASSanswered "loading dock"entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0camp-full_map-gemma4qat-001/statetranscript →
state✓PASSanswered "east wing"entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0camp-full_map-gemma4qat-001/statetranscript →
state✓PASSanswered "mail room"entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0camp-full_map-gemma4qat-001/statetranscript →
state✓PASSanswered "Cafeteria"entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0camp-full_map-gemma4qat-001/statetranscript →
state✓PASSanswered "south stairwell"entity_state=16 temporal=0 coreference=3 distractors=2 context_length=20 witness_gap=0camp-full_map-gemma4qat-001/statetranscript →
state✓PASSanswered "archive room"entity_state=23 temporal=0 coreference=3 distractors=2 context_length=27 witness_gap=0camp-full_map-gemma4qat-001/statetranscript →
state✓PASSanswered "West wing"entity_state=23 temporal=0 coreference=1 distractors=2 context_length=27 witness_gap=0camp-full_map-gemma4qat-001/statetranscript →
state✓PASSanswered "atrium"entity_state=23 temporal=0 coreference=1 distractors=2 context_length=27 witness_gap=0camp-full_map-gemma4qat-001/statetranscript →
state✓PASSanswered "mail room"entity_state=23 temporal=0 coreference=1 distractors=2 context_length=27 witness_gap=0camp-full_map-gemma4qat-001/statetranscript →
state✓PASSanswered "print room"entity_state=23 temporal=0 coreference=1 distractors=2 context_length=27 witness_gap=0camp-full_map-gemma4qat-001/statetranscript →
state✓PASSanswered "storage room"entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0camp-full_map-gemma4qat-001/statetranscript →
state✓PASSanswered "rooftop terrace"entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0camp-full_map-gemma4qat-001/statetranscript →
state✓PASSanswered "photo studio"entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0camp-full_map-gemma4qat-001/statetranscript →
state✓PASSanswered "Photo studio"entity_state=30 temporal=0 coreference=3 distractors=2 context_length=34 witness_gap=0camp-full_map-gemma4qat-001/statetranscript →
state✓PASSanswered "east wing"entity_state=30 temporal=0 coreference=3 distractors=2 context_length=34 witness_gap=0camp-full_map-gemma4qat-001/statetranscript →
state✓PASSanswered "supply closet"entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0camp-full_map-gemma4qat-001/statetranscript →
state✓PASSanswered "cafeteria"entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0camp-full_map-gemma4qat-001/statetranscript →
state✓PASSanswered "print room"entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0camp-full_map-gemma4qat-001/statetranscript →
state✓PASSanswered "photo studio"entity_state=30 temporal=0 coreference=3 distractors=2 context_length=34 witness_gap=0camp-full_map-gemma4qat-001/statetranscript →
state✓PASSanswered "south stairwell"entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0camp-full_map-gemma4qat-001/statetranscript →
state✓PASSanswered "storage room"entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0camp-state-gemma4qat-001transcript →
state✓PASSanswered "Courtyard"entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0camp-state-gemma4qat-001transcript →
state✓PASSanswered "Courtyard"entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0camp-state-gemma4qat-001transcript →
state✓PASSanswered "photo studio"entity_state=2 temporal=0 coreference=3 distractors=2 context_length=6 witness_gap=0camp-state-gemma4qat-001transcript →
state✓PASSanswered "Garage"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=12 witness_gap=0camp-noop-gemma4qat-001transcript →
state✓PASSanswered "Boiler room"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=12 witness_gap=0camp-noop-gemma4qat-001transcript →
state✓PASSanswered "print room"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=12 witness_gap=0camp-noop-gemma4qat-001transcript →
state✓PASSanswered "utility closet"entity_state=2 temporal=0 coreference=3 distractors=2 context_length=6 witness_gap=0camp-state-gemma4qat-001transcript →
state✓PASSanswered "Boiler room"entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0camp-state-gemma4qat-001transcript →
state✓PASSanswered "Cafeteria"entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0camp-state-gemma4qat-001transcript →
state✓PASSanswered "Storage room"entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0camp-state-gemma4qat-001transcript →
state✓PASSanswered "Print room"entity_state=8 temporal=0 coreference=3 distractors=2 context_length=13 witness_gap=0camp-noop-gemma4qat-001transcript →
state✓PASSanswered "Basement"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0camp-noop-gemma4qat-001transcript →
state✓PASSanswered "Break room"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0camp-noop-gemma4qat-001transcript →
state✓PASSanswered "observation deck"entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0camp-state-gemma4qat-001transcript →
state✓PASSanswered "loading dock"entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0camp-state-gemma4qat-001transcript →
state✓PASSanswered "east wing"entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0camp-state-gemma4qat-001transcript →
state✓PASSanswered "Boiler room"entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0camp-state-gemma4qat-001transcript →
state✓PASSanswered "Lobby"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=14 witness_gap=0camp-noop-gemma4qat-001transcript →
state✓PASSanswered "Security office"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=14 witness_gap=0camp-noop-gemma4qat-001transcript →
state✓PASSanswered "Basement"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=14 witness_gap=0camp-noop-gemma4qat-001transcript →
state✓PASSanswered "Cafeteria"entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0camp-state-gemma4qat-001transcript →
state✓PASSanswered "mail room"entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0camp-state-gemma4qat-001transcript →
state✓PASSanswered "archive room"entity_state=23 temporal=0 coreference=3 distractors=2 context_length=27 witness_gap=0camp-state-gemma4qat-001transcript →
state✓PASSanswered "south stairwell"entity_state=16 temporal=0 coreference=3 distractors=2 context_length=20 witness_gap=0camp-state-gemma4qat-001transcript →
state✓PASSanswered "Conference room"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=16 witness_gap=0camp-noop-gemma4qat-001transcript →
state✓PASSanswered "server room"entity_state=8 temporal=0 coreference=3 distractors=2 context_length=16 witness_gap=0camp-noop-gemma4qat-001transcript →
state✓PASSanswered "Rooftop terrace"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=16 witness_gap=0camp-noop-gemma4qat-001transcript →
state✓PASSanswered "West wing"entity_state=23 temporal=0 coreference=1 distractors=2 context_length=27 witness_gap=0camp-state-gemma4qat-001transcript →
state✓PASSanswered "Conference room"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=17 witness_gap=0camp-noop-gemma4qat-001transcript →
state✓PASSanswered "Wine cellar"entity_state=8 temporal=0 coreference=3 distractors=2 context_length=17 witness_gap=0camp-noop-gemma4qat-001transcript →
state✓PASSanswered "Lobby"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=17 witness_gap=0camp-noop-gemma4qat-001transcript →
state✓PASSanswered "Wine cellar"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=18 witness_gap=0camp-noop-gemma4qat-001transcript →
state✓PASSanswered "Front office"entity_state=8 temporal=0 coreference=3 distractors=2 context_length=18 witness_gap=0camp-noop-gemma4qat-001transcript →
state✓PASSanswered "supply closet"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=18 witness_gap=0camp-noop-gemma4qat-001transcript →
state✓PASSanswered "storage room"entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0camp-state-gemma4qat-001transcript →
state✓PASSanswered "mail room"entity_state=23 temporal=0 coreference=1 distractors=2 context_length=27 witness_gap=0camp-state-gemma4qat-001transcript →
state✓PASSanswered "print room"entity_state=23 temporal=0 coreference=1 distractors=2 context_length=27 witness_gap=0camp-state-gemma4qat-001transcript →
state✓PASSanswered "atrium"entity_state=23 temporal=0 coreference=1 distractors=2 context_length=27 witness_gap=0camp-state-gemma4qat-001transcript →
state✓PASSanswered "Observation deck"entity_state=8 temporal=0 coreference=3 distractors=2 context_length=19 witness_gap=0camp-noop-gemma4qat-001transcript →
state✓PASSanswered "Wine cellar"entity_state=8 temporal=0 coreference=3 distractors=2 context_length=19 witness_gap=0camp-noop-gemma4qat-001transcript →
state✓PASSanswered "east wing"entity_state=8 temporal=0 coreference=3 distractors=2 context_length=19 witness_gap=0camp-noop-gemma4qat-001transcript →
state✓PASSanswered "rooftop terrace"entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0camp-state-gemma4qat-001transcript →
state✓PASSanswered "photo studio"entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0camp-state-gemma4qat-001transcript →
state✓PASSanswered "Photo studio"entity_state=30 temporal=0 coreference=3 distractors=2 context_length=34 witness_gap=0camp-state-gemma4qat-001transcript →
state✓PASSanswered "east wing"entity_state=30 temporal=0 coreference=3 distractors=2 context_length=34 witness_gap=0camp-state-gemma4qat-001transcript →
state✓PASSanswered "gym"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0camp-noop-gemma4qat-001transcript →
state✓PASSanswered "photo studio"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0camp-noop-gemma4qat-001transcript →
state✓PASSanswered "south stairwell"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0camp-noop-gemma4qat-001transcript →
state✓PASSanswered "supply closet"entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0camp-state-gemma4qat-001transcript →
state✓PASSanswered "cafeteria"entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0camp-state-gemma4qat-001transcript →
state✓PASSanswered "print room"entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0camp-state-gemma4qat-001transcript →
state✓PASSanswered "photo studio"entity_state=30 temporal=0 coreference=3 distractors=2 context_length=34 witness_gap=0camp-state-gemma4qat-001transcript →
state✓PASSanswered "south stairwell"entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0camp-state-gemma4qat-001transcript →
state✓PASSanswered "greenhouse"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=12 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "Library"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=12 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "Garage"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=12 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "print room"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=12 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "Boiler room"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=12 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "Break room"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "Server room"entity_state=8 temporal=0 coreference=3 distractors=2 context_length=13 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "Lobby"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "Storage room"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "utility closet"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "Greenhouse"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=14 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "Conference room"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=14 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "Cafeteria"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=14 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "Print room"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=14 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "meditation room"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=14 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "Kitchen"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=16 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "Break room"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=16 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "Photo studio"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=16 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "rooftop terrace"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=16 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "print room"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=16 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "Security office"entity_state=8 temporal=0 coreference=3 distractors=2 context_length=17 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "Wine cellar"entity_state=8 temporal=0 coreference=3 distractors=2 context_length=17 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "North stairwell"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=17 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "Photo studio"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=17 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "Atrium"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=17 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "Break room"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=18 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "Observation deck"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=18 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "reception area"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=18 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "North stairwell"entity_state=8 temporal=0 coreference=3 distractors=2 context_length=18 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "Cafeteria"entity_state=8 temporal=0 coreference=3 distractors=2 context_length=18 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "Security office"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=19 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "Garage"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=19 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "storage room"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=19 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "Break room"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=19 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "cafeteria"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=19 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "Reception area"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "Conference room"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "West wing"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "Boiler room"entity_state=8 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0camp-noop-gemma4qat-002transcript →
state✓PASSanswered "Meditation room"entity_state=8 temporal=0 coreference=3 distractors=2 context_length=20 witness_gap=0camp-noop-gemma4qat-002transcript →