Trials · distractors (high)
Every trial behind one square of the capability map: Gemma 3 27B on distractors (high).
92
0 errors
Trials in this cell
59%
54/92 scored
Pass rate
38
0 noise excluded
Real failures
$0.0043
Cell cost
Trials
| kind | verdict | realized dims | seeded traps | run | |
|---|---|---|---|---|---|
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0 coreference=1 distractors=9 context_length=13 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Therefore, the answer is **first**." | entity_state=2 temporal=0 coreference=3 distractors=9 context_length=13 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Therefore, the answer is **first**." | entity_state=2 temporal=0 coreference=1 distractors=9 context_length=13 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0 coreference=1 distractors=9 context_length=13 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0 coreference=1 distractors=9 context_length=13 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0 coreference=3 distractors=16 context_length=20 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Answer: **first**" | entity_state=2 temporal=0 coreference=1 distractors=16 context_length=20 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0 coreference=1 distractors=16 context_length=20 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Answer: first" | entity_state=2 temporal=0 coreference=1 distractors=16 context_length=20 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0 coreference=1 distractors=23 context_length=27 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Answer: **first**" | entity_state=2 temporal=0 coreference=1 distractors=23 context_length=27 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0 coreference=1 distractors=23 context_length=27 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0 coreference=3 distractors=23 context_length=27 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "8:00 is before 8:32, so Willa moving the box happened..." | entity_state=2 temporal=0 coreference=1 distractors=30 context_length=34 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Answer: **first**" | entity_state=2 temporal=0 coreference=1 distractors=30 context_length=34 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Answer: **first**" | entity_state=2 temporal=0 coreference=1 distractors=23 context_length=27 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Answer: **first**" | entity_state=2 temporal=0 coreference=3 distractors=30 context_length=34 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Therefore, the answer is **first**." | entity_state=2 temporal=0 coreference=1 distractors=30 context_length=34 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0 coreference=1 distractors=16 context_length=20 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Answer: first" | entity_state=2 temporal=0 coreference=1 distractors=30 context_length=34 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| belief | ?FAILanswered "lobby"unverified | entity_state=2 temporal=0 coreference=1 distractors=6 context_length=11 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ?FAILanswered "unknown"unverified | entity_state=2 temporal=0 coreference=1 distractors=6 context_length=11 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "archive room" | entity_state=2 temporal=0 coreference=3 distractors=6 context_length=11 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "security office" | entity_state=2 temporal=0 coreference=3 distractors=6 context_length=11 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ?FAILanswered "east wing"unverified | entity_state=2 temporal=0 coreference=3 distractors=11 context_length=16 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ?FAILanswered "library"unverified | entity_state=2 temporal=0 coreference=1 distractors=11 context_length=16 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "photo studio" | entity_state=2 temporal=0 coreference=1 distractors=6 context_length=11 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ?FAILanswered "unknown"unverified | entity_state=2 temporal=0 coreference=1 distractors=16 context_length=21 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ?FAILanswered "loading dock"unverified | entity_state=2 temporal=0 coreference=3 distractors=16 context_length=21 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ?FAILanswered "unknown"unverified | entity_state=2 temporal=0 coreference=1 distractors=11 context_length=16 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ?FAILanswered "kitchen"unverified | entity_state=2 temporal=0 coreference=3 distractors=16 context_length=21 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "north stairwell" | entity_state=2 temporal=0 coreference=1 distractors=11 context_length=16 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "basement" | entity_state=2 temporal=0 coreference=3 distractors=11 context_length=16 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "meditation room" | entity_state=2 temporal=0 coreference=1 distractors=20 context_length=25 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "wine cellar" | entity_state=2 temporal=0 coreference=1 distractors=20 context_length=25 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "south stairwell" | entity_state=2 temporal=0 coreference=1 distractors=16 context_length=21 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ?FAILanswered "unknown"unverified | entity_state=2 temporal=0 coreference=1 distractors=20 context_length=25 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "storage room" | entity_state=2 temporal=0 coreference=3 distractors=16 context_length=21 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "reception area" | entity_state=2 temporal=0 coreference=1 distractors=20 context_length=25 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ?FAILanswered "unknown"unverified | entity_state=2 temporal=0 coreference=1 distractors=20 context_length=25 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "east wing" | entity_state=2 temporal=0 coreference=1 distractors=11 context_length=16 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "north stairwell" | entity_state=2 temporal=0 coreference=1 distractors=11 context_length=16 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "south stairwell" | entity_state=2 temporal=0 coreference=1 distractors=11 context_length=16 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✕FAILanswered "storage room"verified-stable | entity_state=2 temporal=0 coreference=1 distractors=11 context_length=16 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✕FAILanswered "unknown"verified-stable | entity_state=2 temporal=0 coreference=1 distractors=11 context_length=16 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "kitchen" | entity_state=2 temporal=0 coreference=3 distractors=14 context_length=19 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✕FAILanswered "unknown"verified-stable | entity_state=2 temporal=0 coreference=1 distractors=14 context_length=19 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "greenhouse" | entity_state=2 temporal=0 coreference=1 distractors=14 context_length=19 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "archive room" | entity_state=2 temporal=0 coreference=3 distractors=14 context_length=19 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "security office" | entity_state=2 temporal=0 coreference=3 distractors=14 context_length=19 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✕FAILanswered "unknown"verified-stable | entity_state=2 temporal=0 coreference=1 distractors=15 context_length=20 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✕FAILanswered "photo studio"verified-stable | entity_state=2 temporal=0 coreference=1 distractors=15 context_length=20 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "greenhouse" | entity_state=2 temporal=0 coreference=1 distractors=15 context_length=20 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✕FAILanswered "archive room"verified-stable | entity_state=2 temporal=0 coreference=1 distractors=15 context_length=20 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✕FAILanswered "storage room"verified-stable | entity_state=2 temporal=0 coreference=3 distractors=15 context_length=20 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ?FAILanswered "unknown"unverified | entity_state=2 temporal=0 coreference=1 distractors=6 context_length=11 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✓PASSanswered "mail room" | entity_state=2 temporal=0 coreference=3 distractors=6 context_length=11 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ?FAILanswered "archive room"unverified | entity_state=2 temporal=0 coreference=1 distractors=6 context_length=11 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ?FAILanswered "unknown"unverified | entity_state=2 temporal=0 coreference=1 distractors=11 context_length=16 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ?FAILanswered "unknown"unverified | entity_state=2 temporal=0 coreference=1 distractors=6 context_length=11 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ?FAILanswered "library"unverified | entity_state=2 temporal=0 coreference=1 distractors=11 context_length=16 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✕FAILanswered "north stairwell"verified-stable | entity_state=2 temporal=0 coreference=1 distractors=6 context_length=11 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✓PASSanswered "garage" | entity_state=2 temporal=0 coreference=1 distractors=16 context_length=21 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✓PASSanswered "archive room" | entity_state=2 temporal=0 coreference=1 distractors=11 context_length=16 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✕FAILanswered "unknown"verified-stable | entity_state=2 temporal=0 coreference=3 distractors=11 context_length=16 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✓PASSanswered "server room" | entity_state=2 temporal=0 coreference=1 distractors=16 context_length=21 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✓PASSanswered "security office" | entity_state=2 temporal=0 coreference=1 distractors=11 context_length=16 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✕FAILanswered "basement"verified-stable | entity_state=2 temporal=0 coreference=1 distractors=16 context_length=21 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✕FAILanswered "unknown"verified-stable | entity_state=2 temporal=0 coreference=1 distractors=20 context_length=25 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✓PASSanswered "west wing" | entity_state=2 temporal=0 coreference=1 distractors=16 context_length=21 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✓PASSanswered "gym" | entity_state=2 temporal=0 coreference=1 distractors=16 context_length=21 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✓PASSanswered "east wing" | entity_state=2 temporal=0 coreference=1 distractors=20 context_length=25 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✕FAILanswered "storage room"verified-stable | entity_state=2 temporal=0 coreference=3 distractors=20 context_length=25 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✕FAILanswered "cafeteria"verified-stable | entity_state=2 temporal=0 coreference=1 distractors=20 context_length=25 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✕FAILanswered "unknown"verified-stable | entity_state=2 temporal=0 coreference=1 distractors=20 context_length=25 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✕FAILanswered "basement"verified-stable | entity_state=2 temporal=0 coreference=1 distractors=18 context_length=23 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✓PASSanswered "loading dock" | entity_state=2 temporal=0 coreference=1 distractors=18 context_length=23 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✓PASSanswered "conference room" | entity_state=2 temporal=0 coreference=1 distractors=18 context_length=23 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✓PASSanswered "wine cellar" | entity_state=2 temporal=0 coreference=1 distractors=18 context_length=23 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✕FAILanswered "unknown"verified-stable | entity_state=2 temporal=0 coreference=3 distractors=18 context_length=23 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✕FAILanswered "north stairwell"verified-stable | entity_state=2 temporal=0 coreference=1 distractors=19 context_length=24 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✕FAILanswered "unknown"verified-stable | entity_state=2 temporal=0 coreference=1 distractors=19 context_length=24 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✓PASSanswered "basement" | entity_state=2 temporal=0 coreference=1 distractors=19 context_length=24 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✕FAILanswered "server room"verified-stable | entity_state=2 temporal=0 coreference=1 distractors=19 context_length=24 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✕FAILanswered "print room"verified-stable | entity_state=2 temporal=0 coreference=1 distractors=19 context_length=24 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✓PASSanswered "west wing" | entity_state=7 temporal=0.55 coreference=1 distractors=12 context_length=22 witness_gap=1 | camp-full_map-gemma-001/breaking_point | transcript → | |
| belief | ✓PASSanswered "mail room" | entity_state=7 temporal=0.56 coreference=1 distractors=12 context_length=22 witness_gap=1 | camp-full_map-gemma-001/breaking_point | transcript → | |
| belief | ?FAILanswered "security office"unverified | entity_state=7 temporal=0.55 coreference=1 distractors=12 context_length=22 witness_gap=1 | camp-full_map-gemma-001/breaking_point | transcript → | |
| belief | ✓PASSanswered "conference room" | entity_state=7 temporal=0.55 coreference=1 distractors=12 context_length=22 witness_gap=1 | camp-full_map-gemma-001/breaking_point | transcript → | |
| belief | ?FAILanswered "rooftop terrace"unverified | entity_state=7 temporal=0.56 coreference=1 distractors=12 context_length=22 witness_gap=1 | camp-full_map-gemma-001/breaking_point | transcript → | |
| belief | ✓PASSanswered "break room" | entity_state=7 temporal=0.56 coreference=1 distractors=12 context_length=22 witness_gap=1 | camp-full_map-gemma-001/breaking_point | transcript → | |
| belief | ?FAILanswered "south stairwell"unverified | entity_state=7 temporal=0.55 coreference=1 distractors=12 context_length=22 witness_gap=1 | camp-full_map-gemma-001/breaking_point | transcript → |