Data to …… trials

Trials · witness_gap (high)

Every trial behind one square of the capability map: Gemma 4 31B QAT on witness_gap (high).

Gemma 4 31B QAT witness_gap (high) unverified

115
0 errors
Trials in this cell
31%
36/115 scored
Pass rate
79
0 noise excluded
Real failures
$0.0071
Cell cost
Trials36/11531%
kindverdictrealized dimsseeded trapsrun
belief✓PASSanswered "cafeteria"entity_state=3 temporal=0.58 coreference=3 distractors=4 context_length=11 witness_gap=2camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief✓PASSanswered "supply closet"entity_state=3 temporal=0.51 coreference=3 distractors=4 context_length=11 witness_gap=2camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief✓PASSanswered "lobby"entity_state=3 temporal=0.58 coreference=3 distractors=4 context_length=11 witness_gap=2camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief✓PASSanswered "rooftop terrace"entity_state=3 temporal=0.51 coreference=1 distractors=4 context_length=11 witness_gap=2camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "reception area"unverifiedentity_state=3 temporal=0.58 coreference=3 distractors=4 context_length=11 witness_gap=2camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief✓PASSanswered "photo studio"entity_state=3 temporal=0.55 coreference=1 distractors=4 context_length=11 witness_gap=2camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "west wing"unverifiedentity_state=3 temporal=0.58 coreference=1 distractors=4 context_length=11 witness_gap=2camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "front office"unverifiedentity_state=6 temporal=0.51 coreference=3 distractors=9 context_length=30 witness_gap=5camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "boiler room"unverifiedentity_state=6 temporal=0.47 coreference=2 distractors=9 context_length=30 witness_gap=5camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief✓PASSanswered "courtyard"entity_state=6 temporal=0.51 coreference=2 distractors=9 context_length=30 witness_gap=5camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "unknown"unverifiedentity_state=6 temporal=0.47 coreference=2 distractors=9 context_length=30 witness_gap=5camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief✓PASSanswered "security office"entity_state=6 temporal=0.51 coreference=3 distractors=9 context_length=30 witness_gap=5camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "unknown"unverifiedentity_state=6 temporal=0.51 coreference=3 distractors=9 context_length=30 witness_gap=5camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "observation deck"unverifiedentity_state=6 temporal=0.49 coreference=3 distractors=9 context_length=30 witness_gap=5camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief✓PASSanswered "library"entity_state=5 temporal=0.51 coreference=3 distractors=6 context_length=21 witness_gap=4camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief✓PASSanswered "boiler room"entity_state=5 temporal=0.48 coreference=2 distractors=6 context_length=21 witness_gap=4camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief✓PASSanswered "workshop"entity_state=5 temporal=0.50 coreference=3 distractors=6 context_length=21 witness_gap=4camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "cafeteria"unverifiedentity_state=5 temporal=0.51 coreference=3 distractors=6 context_length=21 witness_gap=4camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "greenhouse"unverifiedentity_state=5 temporal=0.48 coreference=3 distractors=6 context_length=21 witness_gap=4camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "server room"unverifiedentity_state=5 temporal=0.51 coreference=3 distractors=6 context_length=21 witness_gap=4camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief✓PASSanswered "workshop"entity_state=5 temporal=0.51 coreference=3 distractors=6 context_length=21 witness_gap=4camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "east wing"unverifiedentity_state=5 temporal=0.50 coreference=3 distractors=8 context_length=26 witness_gap=4camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief✓PASSanswered "print room"entity_state=5 temporal=0.50 coreference=2 distractors=8 context_length=26 witness_gap=4camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "conference room"unverifiedentity_state=5 temporal=0.50 coreference=3 distractors=8 context_length=26 witness_gap=4camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "security office"unverifiedentity_state=5 temporal=0.50 coreference=6 distractors=8 context_length=26 witness_gap=4camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "photo studio"unverifiedentity_state=5 temporal=0.50 coreference=6 distractors=8 context_length=26 witness_gap=4camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "gym"unverifiedentity_state=5 temporal=0.50 coreference=2 distractors=8 context_length=26 witness_gap=4camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "unknown"unverifiedentity_state=5 temporal=0.50 coreference=3 distractors=8 context_length=26 witness_gap=4camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "loading dock"unverifiedentity_state=5 temporal=0.51 coreference=2 distractors=7 context_length=23 witness_gap=4camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "unknown"unverifiedentity_state=5 temporal=0.49 coreference=3 distractors=7 context_length=23 witness_gap=4camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief✓PASSanswered "storage room"entity_state=5 temporal=0.51 coreference=2 distractors=7 context_length=23 witness_gap=4camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "utility closet"unverifiedentity_state=5 temporal=0.49 coreference=2 distractors=7 context_length=23 witness_gap=4camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "kitchen"unverifiedentity_state=5 temporal=0.49 coreference=3 distractors=7 context_length=23 witness_gap=4camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "loading dock"unverifiedentity_state=5 temporal=0.49 coreference=3 distractors=7 context_length=23 witness_gap=4camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "print room"unverifiedentity_state=5 temporal=0.49 coreference=3 distractors=7 context_length=23 witness_gap=4camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "unknown"unverifiedentity_state=5 temporal=0.51 coreference=3 distractors=7 context_length=23 witness_gap=4camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "unknown"unverifiedentity_state=5 temporal=0.49 coreference=3 distractors=7 context_length=23 witness_gap=4camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "boiler room"unverifiedentity_state=5 temporal=0.51 coreference=3 distractors=7 context_length=23 witness_gap=4camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "unknown"unverifiedentity_state=5 temporal=0.49 coreference=3 distractors=7 context_length=23 witness_gap=4camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "unknown"unverifiedentity_state=5 temporal=0.51 coreference=6 distractors=7 context_length=23 witness_gap=4camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "courtyard"unverifiedentity_state=5 temporal=0.49 coreference=3 distractors=7 context_length=23 witness_gap=4camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "greenhouse"unverifiedentity_state=5 temporal=0.51 coreference=2 distractors=7 context_length=23 witness_gap=4camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief✓PASSanswered "storage room"entity_state=3 temporal=0.58 coreference=1 distractors=4 context_length=11 witness_gap=2camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief✓PASSanswered "north stairwell"entity_state=3 temporal=0.55 coreference=1 distractors=4 context_length=11 witness_gap=2camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief✓PASSanswered "break room"entity_state=3 temporal=0.51 coreference=3 distractors=4 context_length=11 witness_gap=2camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "storage room"unverifiedentity_state=3 temporal=0.51 coreference=1 distractors=4 context_length=11 witness_gap=2camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief✓PASSanswered "gym"entity_state=3 temporal=0.58 coreference=1 distractors=4 context_length=11 witness_gap=2camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "server room"unverifiedentity_state=3 temporal=0.58 coreference=1 distractors=4 context_length=11 witness_gap=2camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "unknown"unverifiedentity_state=3 temporal=0.58 coreference=1 distractors=4 context_length=11 witness_gap=2camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "photo studio"unverifiedentity_state=3 temporal=0.58 coreference=1 distractors=4 context_length=11 witness_gap=2camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "supply closet"unverifiedentity_state=3 temporal=0.58 coreference=3 distractors=4 context_length=11 witness_gap=2camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief✓PASSanswered "reception area"entity_state=3 temporal=0.58 coreference=1 distractors=4 context_length=11 witness_gap=2camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "reception area"unverifiedentity_state=3 temporal=0.51 coreference=1 distractors=4 context_length=11 witness_gap=2camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "unknown"unverifiedentity_state=3 temporal=0.58 coreference=1 distractors=4 context_length=11 witness_gap=2camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "unknown"unverifiedentity_state=3 temporal=0.51 coreference=1 distractors=4 context_length=11 witness_gap=2camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief?FAILanswered "server room"unverifiedentity_state=3 temporal=0.58 coreference=1 distractors=4 context_length=11 witness_gap=2camp-full_map-gemma4qat-001/breaking_pointtranscript →
belief✓PASSanswered "break room"entity_state=9 temporal=0.27 coreference=4 distractors=13 context_length=46 witness_gap=3camp-breaking_point-gemma4qat-001transcript →
belief✓PASSanswered "cafeteria"entity_state=9 temporal=0.27 coreference=4 distractors=13 context_length=46 witness_gap=3camp-breaking_point-gemma4qat-001transcript →
belief✓PASSanswered "break room"entity_state=9 temporal=0.27 coreference=6 distractors=13 context_length=46 witness_gap=3camp-breaking_point-gemma4qat-001transcript →
belief?FAILanswered "wine cellar"unverifiedentity_state=12 temporal=0.36 coreference=9 distractors=19 context_length=66 witness_gap=5camp-breaking_point-gemma4qat-001transcript →
belief?FAILanswered "reception area"unverifiedentity_state=12 temporal=0.37 coreference=11 distractors=19 context_length=66 witness_gap=5camp-breaking_point-gemma4qat-001transcript →
belief?FAILanswered "wine cellar"unverifiedentity_state=12 temporal=0.37 coreference=11 distractors=19 context_length=66 witness_gap=5camp-breaking_point-gemma4qat-001transcript →
belief?FAILanswered "garage"unverifiedentity_state=11 temporal=0.32 coreference=15 distractors=16 context_length=57 witness_gap=4camp-breaking_point-gemma4qat-001transcript →
belief?FAILanswered "photo studio"unverifiedentity_state=11 temporal=0.32 coreference=7 distractors=16 context_length=57 witness_gap=4camp-breaking_point-gemma4qat-001transcript →
belief?FAILanswered "garage"unverifiedentity_state=11 temporal=0.32 coreference=7 distractors=16 context_length=57 witness_gap=4camp-breaking_point-gemma4qat-001transcript →
belief✓PASSanswered "photo studio"entity_state=10 temporal=0.29 coreference=6 distractors=15 context_length=53 witness_gap=3camp-breaking_point-gemma4qat-001transcript →
belief?FAILanswered "boiler room"unverifiedentity_state=10 temporal=0.29 coreference=10 distractors=15 context_length=53 witness_gap=3camp-breaking_point-gemma4qat-001transcript →
belief?FAILanswered "wine cellar"unverifiedentity_state=10 temporal=0.30 coreference=7 distractors=15 context_length=53 witness_gap=3camp-breaking_point-gemma4qat-001transcript →
belief✓PASSanswered "the gym"entity_state=9 temporal=0.28 coreference=7 distractors=14 context_length=49 witness_gap=3camp-breaking_point-gemma4qat-001transcript →
belief?FAILanswered "atrium"unverifiedentity_state=9 temporal=0.28 coreference=7 distractors=14 context_length=49 witness_gap=3camp-breaking_point-gemma4qat-001transcript →
belief✓PASSanswered "mail room"entity_state=9 temporal=0.28 coreference=7 distractors=14 context_length=49 witness_gap=3camp-breaking_point-gemma4qat-001transcript →
belief?FAILanswered "front office"unverifiedentity_state=10 temporal=0.30 coreference=15 distractors=15 context_length=53 witness_gap=3camp-breaking_point-gemma4qat-001transcript →
belief?FAILanswered "utility closet"unverifiedentity_state=10 temporal=0.30 coreference=10 distractors=15 context_length=53 witness_gap=3camp-breaking_point-gemma4qat-001transcript →
belief?FAILanswered "garage"unverifiedentity_state=10 temporal=0.30 coreference=6 distractors=15 context_length=53 witness_gap=3camp-breaking_point-gemma4qat-001transcript →
belief✓PASSanswered "conference room"entity_state=10 temporal=0.30 coreference=3 distractors=15 context_length=50 witness_gap=3camp-breaking_point-gemma4qat-001transcript →
belief✓PASSanswered "wine cellar"entity_state=10 temporal=0.30 coreference=3 distractors=15 context_length=50 witness_gap=3camp-breaking_point-gemma4qat-001transcript →
belief?FAILanswered "lobby"unverifiedentity_state=10 temporal=0.30 coreference=3 distractors=15 context_length=50 witness_gap=3camp-breaking_point-gemma4qat-001transcript →
belief?FAILanswered "basement"unverifiedentity_state=10 temporal=0.29 coreference=1 distractors=15 context_length=50 witness_gap=3camp-breaking_point-gemma4qat-001transcript →
belief?FAILanswered "lobby"unverifiedentity_state=10 temporal=0.29 coreference=1 distractors=15 context_length=50 witness_gap=3camp-breaking_point-gemma4qat-001transcript →
belief?FAILanswered "cafeteria"unverifiedentity_state=10 temporal=0.29 coreference=1 distractors=15 context_length=50 witness_gap=3camp-breaking_point-gemma4qat-001transcript →
belief?FAILanswered "boiler room"unverifiedentity_state=10 temporal=0.30 coreference=10 distractors=2 context_length=40 witness_gap=3camp-breaking_point-gemma4qat-001transcript →
belief?FAILanswered "north stairwell"unverifiedentity_state=10 temporal=0.55 coreference=13 distractors=20 context_length=40 witness_gap=8camp-belief-gemma4qat-001transcript →
belief?FAILanswered "lobby"unverifiedentity_state=10 temporal=0.51 coreference=12 distractors=20 context_length=40 witness_gap=8camp-belief-gemma4qat-001transcript →
belief?FAILanswered "north stairwell"unverifiedentity_state=10 temporal=0.30 coreference=6 distractors=2 context_length=40 witness_gap=3camp-breaking_point-gemma4qat-001transcript →
belief?FAILanswered "unknown"unverifiedentity_state=10 temporal=0.56 coreference=12 distractors=20 context_length=40 witness_gap=8camp-belief-gemma4qat-001transcript →
belief✓PASSanswered "storage room"entity_state=10 temporal=0.30 coreference=6 distractors=2 context_length=40 witness_gap=3camp-breaking_point-gemma4qat-001transcript →
belief?FAILanswered "east wing"unverifiedentity_state=10 temporal=0.53 coreference=13 distractors=20 context_length=40 witness_gap=8camp-belief-gemma4qat-001transcript →
belief?FAILanswered "observation deck"unverifiedentity_state=10 temporal=0.31 coreference=6 distractors=2 context_length=40 witness_gap=3camp-breaking_point-gemma4qat-001transcript →
belief?FAILanswered "cafeteria"unverifiedentity_state=10 temporal=0.51 coreference=12 distractors=20 context_length=40 witness_gap=8camp-belief-gemma4qat-001transcript →
belief✓PASSanswered "boiler room"entity_state=10 temporal=0.52 coreference=13 distractors=20 context_length=40 witness_gap=8camp-belief-gemma4qat-001transcript →
belief?FAILanswered "conference room"unverifiedentity_state=10 temporal=0.30 coreference=7 distractors=2 context_length=40 witness_gap=3camp-breaking_point-gemma4qat-001transcript →
belief?FAILanswered "unknown"unverifiedentity_state=10 temporal=0.52 coreference=12 distractors=20 context_length=40 witness_gap=8camp-belief-gemma4qat-001transcript →
belief?FAILanswered "boiler room"unverifiedentity_state=10 temporal=0.30 coreference=7 distractors=2 context_length=40 witness_gap=3camp-breaking_point-gemma4qat-001transcript →
belief?FAILanswered "break room"unverifiedentity_state=10 temporal=0.54 coreference=13 distractors=20 context_length=40 witness_gap=8camp-belief-gemma4qat-001transcript →
belief?FAILanswered "utility closet"unverifiedentity_state=10 temporal=0.30 coreference=4 distractors=15 context_length=33 witness_gap=3camp-breaking_point-gemma4qat-001transcript →
belief?FAILanswered "unknown"unverifiedentity_state=10 temporal=0.53 coreference=13 distractors=20 context_length=40 witness_gap=8camp-belief-gemma4qat-001transcript →
belief?FAILanswered "rooftop terrace"unverifiedentity_state=10 temporal=0.30 coreference=7 distractors=15 context_length=33 witness_gap=3camp-breaking_point-gemma4qat-001transcript →
belief?FAILanswered "reception area"unverifiedentity_state=10 temporal=0.53 coreference=13 distractors=20 context_length=40 witness_gap=8camp-belief-gemma4qat-001transcript →
belief?FAILanswered "break room"unverifiedentity_state=10 temporal=0.30 coreference=10 distractors=15 context_length=33 witness_gap=3camp-breaking_point-gemma4qat-001transcript →
belief✓PASSanswered "archive room"entity_state=10 temporal=0 coreference=7 distractors=15 context_length=53 witness_gap=3camp-breaking_point-gemma4qat-001transcript →
belief?FAILanswered "reception area"unverifiedentity_state=10 temporal=0.53 coreference=12 distractors=20 context_length=40 witness_gap=8camp-belief-gemma4qat-001transcript →
belief✓PASSanswered "atrium"entity_state=10 temporal=0 coreference=10 distractors=15 context_length=53 witness_gap=3camp-breaking_point-gemma4qat-001transcript →
belief?FAILanswered "north stairwell"unverifiedentity_state=10 temporal=0.54 coreference=13 distractors=20 context_length=40 witness_gap=8camp-belief-gemma4qat-001transcript →
belief?FAILanswered "gym"unverifiedentity_state=10 temporal=0 coreference=7 distractors=15 context_length=53 witness_gap=3camp-breaking_point-gemma4qat-001transcript →
belief?FAILanswered "workshop"unverifiedentity_state=10 temporal=0.54 coreference=12 distractors=20 context_length=40 witness_gap=8camp-belief-gemma4qat-001transcript →
belief✓PASSanswered "gym"entity_state=10 temporal=0 coreference=10 distractors=15 context_length=53 witness_gap=3camp-breaking_point-gemma4qat-001transcript →
belief?FAILanswered "conference room"unverifiedentity_state=10 temporal=0.55 coreference=13 distractors=20 context_length=40 witness_gap=8camp-belief-gemma4qat-001transcript →
belief✓PASSanswered "photo studio"entity_state=10 temporal=0 coreference=6 distractors=15 context_length=53 witness_gap=3camp-breaking_point-gemma4qat-001transcript →
belief?FAILanswered "unknown"unverifiedentity_state=10 temporal=0.55 coreference=13 distractors=20 context_length=40 witness_gap=8camp-belief-gemma4qat-001transcript →
belief?FAILanswered "boiler room"unverifiedentity_state=10 temporal=0 coreference=7 distractors=15 context_length=53 witness_gap=3camp-breaking_point-gemma4qat-001transcript →
belief✓PASSanswered "rooftop terrace"entity_state=10 temporal=0.53 coreference=13 distractors=20 context_length=40 witness_gap=8camp-belief-gemma4qat-001transcript →
belief✓PASSanswered "cafeteria"entity_state=10 temporal=0.52 coreference=13 distractors=20 context_length=40 witness_gap=8camp-belief-gemma4qat-001transcript →
belief✓PASSanswered "loading dock"entity_state=10 temporal=0.55 coreference=13 distractors=20 context_length=40 witness_gap=8camp-belief-gemma4qat-001transcript →
belief?FAILanswered "unknown"unverifiedentity_state=10 temporal=0.54 coreference=12 distractors=20 context_length=40 witness_gap=8camp-belief-gemma4qat-001transcript →
belief✓PASSanswered "courtyard"entity_state=10 temporal=0.55 coreference=13 distractors=20 context_length=40 witness_gap=8camp-belief-gemma4qat-001transcript →