Trials · order
Every trial behind one square of the capability map: Gemma 3 27B on order.
80
0 errors
Trials in this cell
99%
79/80 scored
Pass rate
1
0 noise excluded
Real failures
$0.0047
Cell cost
Trials
| kind | verdict | realized dims | seeded traps | run | |
|---|---|---|---|---|---|
| order | ✓PASSanswered "Therefore, Bruno moving the box happened first." | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "8:00 is before 8:04, so Oscar's action happened first." | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Therefore, the answer is **first**." | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Answer: **first**" | entity_state=6 temporal=0 coreference=3 distractors=2 context_length=10 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Therefore, Maya's action happened first." | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Answer: **first**" | entity_state=6 temporal=0 coreference=3 distractors=2 context_length=10 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "8:00 is before 8:04, so Sana moving the tablet to the..." | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=6 temporal=0 coreference=1 distractors=2 context_length=10 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "8:00 is before 8:08, so Noor's action happened first." | entity_state=6 temporal=0 coreference=3 distractors=2 context_length=10 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=11 temporal=0 coreference=3 distractors=2 context_length=15 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "The answer is **first**." | entity_state=11 temporal=0 coreference=1 distractors=2 context_length=15 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=11 temporal=0 coreference=1 distractors=2 context_length=15 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "The answer is **first**." | entity_state=11 temporal=0 coreference=3 distractors=2 context_length=15 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=11 temporal=0 coreference=1 distractors=2 context_length=15 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=16 temporal=0 coreference=3 distractors=2 context_length=20 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Answer: **first**" | entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=6 temporal=0 coreference=1 distractors=2 context_length=10 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**first**" | entity_state=20 temporal=0 coreference=1 distractors=2 context_length=24 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=20 temporal=0 coreference=1 distractors=2 context_length=24 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Therefore, Kira moving the box happened first." | entity_state=20 temporal=0 coreference=1 distractors=2 context_length=24 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Therefore, the answer is **first**." | entity_state=16 temporal=0 coreference=3 distractors=2 context_length=20 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=20 temporal=0 coreference=1 distractors=2 context_length=24 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Answer: **first**" | entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=20 temporal=0 coreference=1 distractors=2 context_length=24 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Therefore, the answer is **first**." | entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**first**" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Answer: **first**" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "The answer is **first**." | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "The answer is **first**." | entity_state=2 temporal=0 coreference=3 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "8:00 is before 8:05, so moving the box happened first." | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Dana's action happened at 8:00, which is before Wyatt..." | entity_state=2 temporal=0.27 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Therefore, the answer is **first**." | entity_state=2 temporal=0.27 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0.53 coreference=0 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0.27 coreference=3 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0.53 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0.27 coreference=3 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0.53 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "The answer is **first**." | entity_state=2 temporal=0.27 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Therefore, the answer is **first**." | entity_state=2 temporal=0.73 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0.60 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0.53 coreference=3 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0.60 coreference=0 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**first**" | entity_state=2 temporal=0.73 coreference=3 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**first**" | entity_state=2 temporal=0.73 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Therefore, the answer is **first**." | entity_state=2 temporal=0.73 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Since 8:00 is earlier than 8:05, Sana moving the fold..." | entity_state=2 temporal=0.60 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0.53 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0.73 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0.73 coreference=0 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✕FAILanswered "**second**"verified-stable | entity_state=2 temporal=0.73 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "8:00 is before 8:04, so moving the folder happened fi..." | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "The answer is **first**." | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0 coreference=1 distractors=9 context_length=13 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Therefore, the answer is **first**." | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Therefore, the answer is **first**." | entity_state=2 temporal=0 coreference=3 distractors=9 context_length=13 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Therefore, the answer is **first**." | entity_state=2 temporal=0 coreference=1 distractors=9 context_length=13 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0 coreference=1 distractors=9 context_length=13 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "The answer is **first**." | entity_state=2 temporal=0 coreference=3 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0 coreference=1 distractors=9 context_length=13 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**first**" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0 coreference=3 distractors=16 context_length=20 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Answer: **first**" | entity_state=2 temporal=0 coreference=1 distractors=16 context_length=20 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0 coreference=1 distractors=16 context_length=20 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Answer: first" | entity_state=2 temporal=0 coreference=1 distractors=16 context_length=20 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0 coreference=1 distractors=23 context_length=27 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Answer: **first**" | entity_state=2 temporal=0 coreference=1 distractors=23 context_length=27 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0 coreference=1 distractors=23 context_length=27 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0 coreference=3 distractors=23 context_length=27 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "8:00 is before 8:32, so Willa moving the box happened..." | entity_state=2 temporal=0 coreference=1 distractors=30 context_length=34 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Answer: **first**" | entity_state=2 temporal=0 coreference=1 distractors=30 context_length=34 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Answer: **first**" | entity_state=2 temporal=0 coreference=1 distractors=23 context_length=27 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Answer: **first**" | entity_state=2 temporal=0 coreference=3 distractors=30 context_length=34 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Therefore, the answer is **first**." | entity_state=2 temporal=0 coreference=1 distractors=30 context_length=34 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0 coreference=1 distractors=16 context_length=20 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Answer: first" | entity_state=2 temporal=0 coreference=1 distractors=30 context_length=34 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=3 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "8:00 is before 8:05, so Finn moving the folder to the..." | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "**Answer: first**" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → | |
| order | ✓PASSanswered "Therefore, the answer is **first**." | entity_state=2 temporal=0 coreference=0 distractors=2 context_length=6 witness_gap=0 | camp-full_map-gemma-001/order | transcript → |