Trials · temporal (high)
Every trial behind one square of the capability map: Gemma 3 27B on temporal (high).
67
0 errors
Trials in this cell
67%
45/67 scored
Pass rate
22
0 noise excluded
Real failures
$0.0022
Cell cost
Trials
| kind | verdict | realized dims | seeded traps | run | |
|---|---|---|---|---|---|
| order | ✓PASSanswered "Dana's action happened at 8:00, which is before Wyatt..." | entity_state=2 temporal=0.27 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Therefore, the answer is **first**." | entity_state=2 temporal=0.27 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0.53 coreference=0 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0.27 coreference=3 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0.53 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0.27 coreference=3 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0.53 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "The answer is **first**." | entity_state=2 temporal=0.27 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Therefore, the answer is **first**." | entity_state=2 temporal=0.73 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0.60 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0.53 coreference=3 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0.60 coreference=0 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**first**" | entity_state=2 temporal=0.73 coreference=3 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**first**" | entity_state=2 temporal=0.73 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Therefore, the answer is **first**." | entity_state=2 temporal=0.73 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Since 8:00 is earlier than 8:05, Sana moving the fold..." | entity_state=2 temporal=0.60 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0.53 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0.73 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0.73 coreference=0 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✕FAILanswered "**second**"verified-stable | entity_state=2 temporal=0.73 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| belief | ✓PASSanswered "north stairwell" | entity_state=2 temporal=0.24 coreference=1 distractors=2 context_length=7 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "library" | entity_state=2 temporal=0.24 coreference=3 distractors=2 context_length=7 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "loading dock" | entity_state=2 temporal=0.24 coreference=1 distractors=2 context_length=7 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "rooftop terrace" | entity_state=2 temporal=0.24 coreference=3 distractors=2 context_length=7 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "east wing" | entity_state=2 temporal=0.24 coreference=1 distractors=2 context_length=7 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ?FAILanswered "rooftop terrace"unverified | entity_state=2 temporal=0.48 coreference=3 distractors=2 context_length=7 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "rooftop terrace" | entity_state=2 temporal=0.48 coreference=1 distractors=2 context_length=7 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "security office" | entity_state=2 temporal=0.48 coreference=1 distractors=2 context_length=7 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "greenhouse" | entity_state=2 temporal=0.67 coreference=3 distractors=2 context_length=7 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "storage room" | entity_state=2 temporal=0.67 coreference=1 distractors=2 context_length=7 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ?FAILanswered "reception area"unverified | entity_state=2 temporal=0.67 coreference=1 distractors=2 context_length=7 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ?FAILanswered "wine cellar"unverified | entity_state=2 temporal=0.67 coreference=3 distractors=2 context_length=7 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "workshop" | entity_state=2 temporal=0.48 coreference=3 distractors=2 context_length=7 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "front office" | entity_state=2 temporal=0.48 coreference=1 distractors=2 context_length=7 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ?FAILanswered "security office"unverified | entity_state=2 temporal=0.71 coreference=1 distractors=2 context_length=7 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "observation deck" | entity_state=2 temporal=0.67 coreference=1 distractors=2 context_length=7 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "utility closet" | entity_state=2 temporal=0.71 coreference=1 distractors=2 context_length=7 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ?FAILanswered "archive room"unverified | entity_state=2 temporal=0.71 coreference=1 distractors=2 context_length=7 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ?FAILanswered "south stairwell"unverified | entity_state=2 temporal=0.71 coreference=1 distractors=2 context_length=7 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ?FAILanswered "photo studio"unverified | entity_state=2 temporal=0.71 coreference=3 distractors=2 context_length=7 witness_gap=1 | camp-full_map-gemma-001/belief | transcript → | |
| belief | ✓PASSanswered "front office" | entity_state=2 temporal=0.24 coreference=1 distractors=2 context_length=7 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ?FAILanswered "archive room"unverified | entity_state=2 temporal=0.24 coreference=1 distractors=2 context_length=7 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ?FAILanswered "storage room"unverified | entity_state=2 temporal=0.24 coreference=1 distractors=2 context_length=7 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ?FAILanswered "print room"unverified | entity_state=2 temporal=0.24 coreference=1 distractors=2 context_length=7 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✓PASSanswered "print room" | entity_state=2 temporal=0.48 coreference=1 distractors=2 context_length=7 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ?FAILanswered "south stairwell"unverified | entity_state=2 temporal=0.24 coreference=0 distractors=2 context_length=7 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ?FAILanswered "break room"unverified | entity_state=2 temporal=0.48 coreference=1 distractors=2 context_length=7 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✓PASSanswered "supply closet" | entity_state=2 temporal=0.48 coreference=1 distractors=2 context_length=7 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ?FAILanswered "workshop"unverified | entity_state=2 temporal=0.48 coreference=1 distractors=2 context_length=7 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✓PASSanswered "the garage" | entity_state=2 temporal=0.67 coreference=1 distractors=2 context_length=7 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ?FAILanswered "photo studio"unverified | entity_state=2 temporal=0.48 coreference=1 distractors=2 context_length=7 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ?FAILanswered "archive room"unverified | entity_state=2 temporal=0.71 coreference=0 distractors=2 context_length=7 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✓PASSanswered "library" | entity_state=2 temporal=0.67 coreference=1 distractors=2 context_length=7 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ?FAILanswered "boiler room"unverified | entity_state=2 temporal=0.67 coreference=1 distractors=2 context_length=7 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✓PASSanswered "archive room" | entity_state=2 temporal=0.67 coreference=3 distractors=2 context_length=7 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ?FAILanswered "photo studio"unverified | entity_state=2 temporal=0.67 coreference=3 distractors=2 context_length=7 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✓PASSanswered "break room" | entity_state=2 temporal=0.71 coreference=1 distractors=2 context_length=7 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✓PASSanswered "gym" | entity_state=2 temporal=0.71 coreference=0 distractors=2 context_length=7 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ?FAILanswered "archive room"unverified | entity_state=2 temporal=0.71 coreference=1 distractors=2 context_length=7 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✓PASSanswered "rooftop terrace" | entity_state=2 temporal=0.71 coreference=1 distractors=2 context_length=7 witness_gap=1 | final_true_statelast_mentioned_locationlast_witnessed_by_anyone | camp-full_map-gemma-001/adversarial_belief | transcript → |
| belief | ✓PASSanswered "west wing" | entity_state=7 temporal=0.55 coreference=1 distractors=12 context_length=22 witness_gap=1 | camp-full_map-gemma-001/breaking_point | transcript → | |
| belief | ✓PASSanswered "mail room" | entity_state=7 temporal=0.56 coreference=1 distractors=12 context_length=22 witness_gap=1 | camp-full_map-gemma-001/breaking_point | transcript → | |
| belief | ?FAILanswered "security office"unverified | entity_state=7 temporal=0.55 coreference=1 distractors=12 context_length=22 witness_gap=1 | camp-full_map-gemma-001/breaking_point | transcript → | |
| belief | ✓PASSanswered "conference room" | entity_state=7 temporal=0.55 coreference=1 distractors=12 context_length=22 witness_gap=1 | camp-full_map-gemma-001/breaking_point | transcript → | |
| belief | ?FAILanswered "rooftop terrace"unverified | entity_state=7 temporal=0.56 coreference=1 distractors=12 context_length=22 witness_gap=1 | camp-full_map-gemma-001/breaking_point | transcript → | |
| belief | ✓PASSanswered "break room" | entity_state=7 temporal=0.56 coreference=1 distractors=12 context_length=22 witness_gap=1 | camp-full_map-gemma-001/breaking_point | transcript → | |
| belief | ?FAILanswered "south stairwell"unverified | entity_state=7 temporal=0.55 coreference=1 distractors=12 context_length=22 witness_gap=1 | camp-full_map-gemma-001/breaking_point | transcript → |