Trials · entity_state (high)
Every trial behind one square of the capability map: Gemma 3 27B on entity_state (high).
102
0 errors
Trials in this cell
56%
57/102 scored
Pass rate
45
0 noise excluded
Real failures
$0.0049
Cell cost
Trials
| kind | verdict | realized dims | seeded traps | run | |
|---|---|---|---|---|---|
| state | ✕FAILanswered "West wing"verified-stable | entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "Cafeteria" | entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "Storage room" | entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "Observation deck" | entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "loading dock" | entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "Boiler room" | entity_state=9 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "mail room"verified-stable | entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "Cafeteria" | entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "mail room" | entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "Server room"verified-stable | entity_state=16 temporal=0 coreference=3 distractors=2 context_length=20 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "West wing" | entity_state=23 temporal=0 coreference=1 distractors=2 context_length=27 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "The archive room." | entity_state=23 temporal=0 coreference=3 distractors=2 context_length=27 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "Wine cellar"verified-stable | entity_state=23 temporal=0 coreference=1 distractors=2 context_length=27 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "Security office"verified-stable | entity_state=23 temporal=0 coreference=1 distractors=2 context_length=27 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "Print room" | entity_state=23 temporal=0 coreference=1 distractors=2 context_length=27 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "Storage room" | entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "atrium"verified-stable | entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "rooftop terrace" | entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "wine cellar"verified-stable | entity_state=30 temporal=0 coreference=3 distractors=2 context_length=34 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "Archive room"verified-stable | entity_state=30 temporal=0 coreference=3 distractors=2 context_length=34 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "Photo studio" | entity_state=26 temporal=0 coreference=1 distractors=2 context_length=30 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "north stairwell"verified-stable | entity_state=26 temporal=0 coreference=1 distractors=2 context_length=30 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "print room" | entity_state=26 temporal=0 coreference=1 distractors=2 context_length=30 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "library" | entity_state=26 temporal=0 coreference=3 distractors=2 context_length=30 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "Courtyard"verified-stable | entity_state=26 temporal=0 coreference=3 distractors=2 context_length=30 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "Garage"verified-stable | entity_state=28 temporal=0 coreference=3 distractors=2 context_length=32 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "photo studio"verified-stable | entity_state=28 temporal=0 coreference=1 distractors=2 context_length=32 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "Kitchen"verified-stable | entity_state=28 temporal=0 coreference=3 distractors=2 context_length=32 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "West wing" | entity_state=28 temporal=0 coreference=1 distractors=2 context_length=32 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "mail room" | entity_state=28 temporal=0 coreference=3 distractors=2 context_length=32 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "Workshop" | entity_state=27 temporal=0 coreference=1 distractors=2 context_length=31 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "Storage room"verified-stable | entity_state=27 temporal=0 coreference=3 distractors=2 context_length=31 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "library"verified-stable | entity_state=27 temporal=0 coreference=1 distractors=2 context_length=31 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "server room"verified-stable | entity_state=27 temporal=0 coreference=1 distractors=2 context_length=31 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "break room"verified-stable | entity_state=27 temporal=0 coreference=1 distractors=2 context_length=31 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "Library"verified-stable | entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "workshop" | entity_state=30 temporal=0 coreference=3 distractors=2 context_length=34 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✓PASSanswered "Storage room" | entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "boiler room"verified-stable | entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| state | ✕FAILanswered "Gym"verified-stable | entity_state=30 temporal=0 coreference=1 distractors=2 context_length=34 witness_gap=0 | camp-full_map-gemma-001/state | transcript → | |
| order | ✓PASSanswered "Answer: **first**" | entity_state=6 temporal=0 coreference=3 distractors=2 context_length=10 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Answer: **first**" | entity_state=6 temporal=0 coreference=3 distractors=2 context_length=10 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=6 temporal=0 coreference=1 distractors=2 context_length=10 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "8:00 is before 8:08, so Noor's action happened first." | entity_state=6 temporal=0 coreference=3 distractors=2 context_length=10 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=11 temporal=0 coreference=3 distractors=2 context_length=15 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "The answer is **first**." | entity_state=11 temporal=0 coreference=1 distractors=2 context_length=15 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=11 temporal=0 coreference=1 distractors=2 context_length=15 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "The answer is **first**." | entity_state=11 temporal=0 coreference=3 distractors=2 context_length=15 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=11 temporal=0 coreference=1 distractors=2 context_length=15 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=16 temporal=0 coreference=3 distractors=2 context_length=20 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Answer: **first**" | entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=6 temporal=0 coreference=1 distractors=2 context_length=10 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**first**" | entity_state=20 temporal=0 coreference=1 distractors=2 context_length=24 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=20 temporal=0 coreference=1 distractors=2 context_length=24 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Therefore, Kira moving the box happened first." | entity_state=20 temporal=0 coreference=1 distractors=2 context_length=24 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Therefore, the answer is **first**." | entity_state=16 temporal=0 coreference=3 distractors=2 context_length=20 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=20 temporal=0 coreference=1 distractors=2 context_length=24 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Answer: **first**" | entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=20 temporal=0 coreference=1 distractors=2 context_length=24 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Therefore, the answer is **first**." | entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| belief | ✓PASSanswered "break room" | entity_state=6 temporal=0 coreference=1 distractors=2 context_length=11 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "reception area" | entity_state=6 temporal=0 coreference=1 distractors=2 context_length=11 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "break room" | entity_state=6 temporal=0 coreference=3 distractors=2 context_length=11 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ?FAILanswered "atrium"unverified | entity_state=6 temporal=0 coreference=1 distractors=2 context_length=11 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "boiler room" | entity_state=6 temporal=0 coreference=1 distractors=2 context_length=11 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "library" | entity_state=8 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ?FAILanswered "rooftop terrace"unverified | entity_state=8 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ?FAILanswered "kitchen"unverified | entity_state=8 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ?FAILanswered "garage"unverified | entity_state=10 temporal=0 coreference=1 distractors=2 context_length=15 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "cafeteria" | entity_state=10 temporal=0 coreference=3 distractors=2 context_length=15 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "reception area" | entity_state=8 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "rooftop terrace" | entity_state=8 temporal=0 coreference=3 distractors=2 context_length=13 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ?FAILanswered "courtyard"unverified | entity_state=10 temporal=0 coreference=1 distractors=2 context_length=15 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ?FAILanswered "mail room"unverified | entity_state=10 temporal=0 coreference=1 distractors=2 context_length=15 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "server room" | entity_state=10 temporal=0 coreference=1 distractors=2 context_length=15 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✕FAILanswered "archive room"verified-stable | entity_state=9 temporal=0 coreference=1 distractors=2 context_length=14 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "loading dock" | entity_state=9 temporal=0 coreference=3 distractors=2 context_length=14 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✕FAILanswered "server room"verified-stable | entity_state=9 temporal=0 coreference=1 distractors=2 context_length=14 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✕FAILanswered "west wing"verified-stable | entity_state=9 temporal=0 coreference=1 distractors=2 context_length=14 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✕FAILanswered "break room"verified-stable | entity_state=9 temporal=0 coreference=1 distractors=2 context_length=14 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ?FAILanswered "south stairwell"unverified | entity_state=6 temporal=0 coreference=0 distractors=2 context_length=11 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyonemost_frequent_locationquery_agent_own_last_move | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ?FAILanswered "observation deck"unverified | entity_state=6 temporal=0 coreference=1 distractors=2 context_length=11 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyonemost_frequent_locationquery_agent_own_last_move | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✓PASSanswered "archive room" | entity_state=6 temporal=0 coreference=1 distractors=2 context_length=11 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyonemost_frequent_locationquery_agent_own_last_move | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ?FAILanswered "break room"unverified | entity_state=8 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyonemost_frequent_locationquery_agent_own_last_move | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ?FAILanswered "south stairwell"unverified | entity_state=6 temporal=0 coreference=3 distractors=2 context_length=11 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyonemost_frequent_locationquery_agent_own_last_move | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ?FAILanswered "conference room"unverified | entity_state=6 temporal=0 coreference=3 distractors=2 context_length=11 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyonemost_frequent_locationquery_agent_own_last_move | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ?FAILanswered "basement"unverified | entity_state=8 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyonemost_frequent_locationquery_agent_own_last_move | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ?FAILanswered "greenhouse"unverified | entity_state=8 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyonemost_frequent_locationquery_agent_own_last_move | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ?FAILanswered "meditation room"unverified | entity_state=8 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyonemost_frequent_locationquery_agent_own_last_move | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ?FAILanswered "reception area"unverified | entity_state=8 temporal=0 coreference=1 distractors=2 context_length=13 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyonemost_frequent_locationquery_agent_own_last_move | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ?FAILanswered "mail room"unverified | entity_state=10 temporal=0 coreference=3 distractors=2 context_length=15 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyonemost_frequent_locationquery_agent_own_last_move | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✓PASSanswered "wine cellar" | entity_state=10 temporal=0 coreference=1 distractors=2 context_length=15 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyonemost_frequent_locationquery_agent_own_last_move | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✓PASSanswered "lobby" | entity_state=10 temporal=0 coreference=1 distractors=2 context_length=15 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyonemost_frequent_locationquery_agent_own_last_move | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ?FAILanswered "workshop"unverified | entity_state=10 temporal=0 coreference=3 distractors=2 context_length=15 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyonemost_frequent_locationquery_agent_own_last_move | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ?FAILanswered "library"unverified | entity_state=10 temporal=0 coreference=1 distractors=2 context_length=15 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyonemost_frequent_locationquery_agent_own_last_move | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✓PASSanswered "west wing" | entity_state=7 temporal=0.55 coreference=1 distractors=12 context_length=22 witness_gap=1 | camp-full_map-gemma-001/breaking_point | transcript → | |
| belief | ✓PASSanswered "mail room" | entity_state=7 temporal=0.56 coreference=1 distractors=12 context_length=22 witness_gap=1 | camp-full_map-gemma-001/breaking_point | transcript → | |
| belief | ?FAILanswered "security office"unverified | entity_state=7 temporal=0.55 coreference=1 distractors=12 context_length=22 witness_gap=1 | camp-full_map-gemma-001/breaking_point | transcript → | |
| belief | ✓PASSanswered "conference room" | entity_state=7 temporal=0.55 coreference=1 distractors=12 context_length=22 witness_gap=1 | camp-full_map-gemma-001/breaking_point | transcript → | |
| belief | ?FAILanswered "rooftop terrace"unverified | entity_state=7 temporal=0.56 coreference=1 distractors=12 context_length=22 witness_gap=1 | camp-full_map-gemma-001/breaking_point | transcript → | |
| belief | ✓PASSanswered "break room" | entity_state=7 temporal=0.56 coreference=1 distractors=12 context_length=22 witness_gap=1 | camp-full_map-gemma-001/breaking_point | transcript → | |
| belief | ?FAILanswered "south stairwell"unverified | entity_state=7 temporal=0.55 coreference=1 distractors=12 context_length=22 witness_gap=1 | camp-full_map-gemma-001/breaking_point | transcript → |