Trials · order
Every trial behind one square of the capability map: Nemotron 3 Ultra 550B on order.
80
0 errors
Trials in this cell
100%
80/80 scored
Pass rate
0
0 noise excluded
Real failures
$0.0856
Cell cost
Trials
| kind | verdict | realized dims | seeded traps | run | |
|---|---|---|---|---|---|
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=6 temporal=0 coreference=1 distractors=2 context_length=10 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=6 temporal=0 coreference=1 distractors=2 context_length=10 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=6 temporal=0 coreference=3 distractors=2 context_length=10 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=6 temporal=0 coreference=3 distractors=2 context_length=10 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=11 temporal=0 coreference=3 distractors=2 context_length=15 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=6 temporal=0 coreference=3 distractors=2 context_length=10 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=11 temporal=0 coreference=1 distractors=2 context_length=15 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=11 temporal=0 coreference=3 distractors=2 context_length=15 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=11 temporal=0 coreference=1 distractors=2 context_length=15 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=11 temporal=0 coreference=1 distractors=2 context_length=15 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=16 temporal=0 coreference=1 distractors=2 context_length=20 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=16 temporal=0 coreference=3 distractors=2 context_length=20 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=16 temporal=0 coreference=3 distractors=2 context_length=20 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=20 temporal=0 coreference=1 distractors=2 context_length=24 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=20 temporal=0 coreference=1 distractors=2 context_length=24 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=20 temporal=0 coreference=1 distractors=2 context_length=24 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=20 temporal=0 coreference=1 distractors=2 context_length=24 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=20 temporal=0 coreference=1 distractors=2 context_length=24 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=3 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0.27 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0.27 coreference=3 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0.27 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0.27 coreference=3 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0.53 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0.27 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0.53 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0.53 coreference=3 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0.73 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0.53 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0.53 coreference=0 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0.60 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0.60 coreference=0 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0.73 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0.73 coreference=3 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0.73 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0.73 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0.60 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0.73 coreference=0 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0.73 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=3 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=3 distractors=9 context_length=13 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=3 distractors=16 context_length=20 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=1 distractors=9 context_length=13 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=1 distractors=9 context_length=13 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=1 distractors=9 context_length=13 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=1 distractors=9 context_length=13 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=1 distractors=16 context_length=20 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=1 distractors=16 context_length=20 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=1 distractors=16 context_length=20 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=1 distractors=16 context_length=20 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=1 distractors=23 context_length=27 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=1 distractors=23 context_length=27 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=1 distractors=23 context_length=27 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=1 distractors=23 context_length=27 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=3 distractors=23 context_length=27 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=1 distractors=30 context_length=34 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=1 distractors=30 context_length=34 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=1 distractors=30 context_length=34 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=3 distractors=30 context_length=34 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=1 distractors=30 context_length=34 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=3 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=1 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → | |
| order | ✓PASSanswered "first" | entity_state=2 temporal=0 coreference=0 distractors=2 context_length=6 witness_gap=0 | camp-full_map-ultra-002/order | transcript → |