Data to …… trials

Run camp-full_map-gemma-001/order

99% PASS

What this run revealed, then every individual trial as evidence.

80
Trials
99%
79/80 scored
over trials
Pass rate
1
1 stable / 0 noise
1%
Failures
$0.0047
Run cost

Run camp-full_map-gemma-001/order: passed essentially everything (99%) -- this test did not challenge the model.

Of 80 scored trials (order query kind), the model passed 99%. That's saturation: the difficulty range probed here isn't finding this model's limits. Consider a harder preset (breaking_point or adversarial_belief) or a wider dimension range.

1 failures recorded; all 1 survived verification.

Notable failures

  • record d73cbb84 — asked "Which happened first: Quinn moving the faded basket to the meditation room or Wren's aside? Answer 'first' if the former, 'second' if the latter."
    expected: first
    actual: **second**

    the model recovered narration order, not the true event timestamps

Trials80 trials
Per-trial table (80 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
google/gemma-3-27b-it✓PASSanswered "Therefore, Bruno moving the box happened first."order-0.000033149/59full transcript →
google/gemma-3-27b-it✓PASSanswered "8:00 is before 8:04, so Oscar's action happened first."order-0.000034146/66full transcript →
google/gemma-3-27b-it✓PASSanswered "Therefore, the answer is **first**."order-0.000035142/69full transcript →
google/gemma-3-27b-it✓PASSanswered "Answer: **first**"order-0.000048213/90full transcript →
google/gemma-3-27b-it✓PASSanswered "Therefore, Maya's action happened first."order-0.000032152/56full transcript →
google/gemma-3-27b-it✓PASSanswered "Answer: **first**"order-0.000056219/113full transcript →
google/gemma-3-27b-it✓PASSanswered "8:00 is before 8:04, so Sana moving the tablet to the bre..."order-0.000035144/69full transcript →
google/gemma-3-27b-it✓PASSanswered "**Answer: first**"order-0.000053216/103full transcript →
google/gemma-3-27b-it✓PASSanswered "8:00 is before 8:08, so Noor's action happened first."order-0.000043218/69full transcript →
google/gemma-3-27b-it✓PASSanswered "**Answer: first**"order-0.000047299/56full transcript →
google/gemma-3-27b-it✓PASSanswered "The answer is **first**."order-0.000058302/94full transcript →
google/gemma-3-27b-it✓PASSanswered "**Answer: first**"order-0.000064313/108full transcript →
google/gemma-3-27b-it✓PASSanswered "The answer is **first**."order-0.000052298/74full transcript →
google/gemma-3-27b-it✓PASSanswered "**Answer: first**"order-0.000051311/66full transcript →
google/gemma-3-27b-it✓PASSanswered "**Answer: first**"order-0.000066391/91full transcript →
google/gemma-3-27b-it✓PASSanswered "Answer: **first**"order-0.000069394/98full transcript →
google/gemma-3-27b-it✓PASSanswered "**Answer: first**"order-0.000108217/289full transcript →
google/gemma-3-27b-it✓PASSanswered "**first**"order-0.000072469/83full transcript →
google/gemma-3-27b-it✓PASSanswered "**Answer: first**"order-0.000077472/99full transcript →
google/gemma-3-27b-it✓PASSanswered "Therefore, Kira moving the box happened first."order-0.000064461/60full transcript →
google/gemma-3-27b-it✓PASSanswered "Therefore, the answer is **first**."order-0.000061401/69full transcript →
google/gemma-3-27b-it✓PASSanswered "**Answer: first**"order-0.000074467/90full transcript →
google/gemma-3-27b-it✓PASSanswered "Answer: **first**"order-0.000069400/97full transcript →
google/gemma-3-27b-it✓PASSanswered "**Answer: first**"order-0.000081461/116full transcript →
google/gemma-3-27b-it✓PASSanswered "Therefore, the answer is **first**."order-0.000075404/115full transcript →
google/gemma-3-27b-it✓PASSanswered "**first**"order-0.000034143/64full transcript →
google/gemma-3-27b-it✓PASSanswered "Answer: **first**"order-0.000039146/81full transcript →
google/gemma-3-27b-it✓PASSanswered "The answer is **first**."order-0.000042145/90full transcript →
google/gemma-3-27b-it✓PASSanswered "The answer is **first**."order-0.000041140/91full transcript →
google/gemma-3-27b-it✓PASSanswered "8:00 is before 8:05, so moving the box happened first."order-0.000031146/56full transcript →
google/gemma-3-27b-it✓PASSanswered "Dana's action happened at 8:00, which is before Wyatt's a..."order-0.000038146/79full transcript →
google/gemma-3-27b-it✓PASSanswered "Therefore, the answer is **first**."order-0.000069149/180full transcript →
google/gemma-3-27b-it✓PASSanswered "**Answer: first**"order-0.000047143/108full transcript →
google/gemma-3-27b-it✓PASSanswered "**Answer: first**"order-0.000067146/174full transcript →
google/gemma-3-27b-it✓PASSanswered "**Answer: first**"order-0.000079151/214full transcript →
google/gemma-3-27b-it✓PASSanswered "**Answer: first**"order-0.000065143/170full transcript →
google/gemma-3-27b-it✓PASSanswered "**Answer: first**"order-0.000069149/179full transcript →
google/gemma-3-27b-it✓PASSanswered "The answer is **first**."order-0.000078147/210full transcript →
google/gemma-3-27b-it✓PASSanswered "Therefore, the answer is **first**."order-0.000035138/72full transcript →
google/gemma-3-27b-it✓PASSanswered "first"order-0.000015141/3full transcript →
google/gemma-3-27b-it✓PASSanswered "**Answer: first**"order-0.000068142/179full transcript →
google/gemma-3-27b-it✓PASSanswered "**Answer: first**"order-0.000068150/176full transcript →
google/gemma-3-27b-it✓PASSanswered "**first**"order-0.000060148/151full transcript →
google/gemma-3-27b-it✓PASSanswered "**first**"order-0.000064149/164full transcript →
google/gemma-3-27b-it✓PASSanswered "Therefore, the answer is **first**."order-0.000082145/226full transcript →
google/gemma-3-27b-it✓PASSanswered "Since 8:00 is earlier than 8:05, Sana moving the folder h..."order-0.000036152/68full transcript →
google/gemma-3-27b-it✓PASSanswered "**Answer: first**"order-0.000064142/167full transcript →
google/gemma-3-27b-it✓PASSanswered "**Answer: first**"order-0.000066150/170full transcript →
google/gemma-3-27b-it✓PASSanswered "**Answer: first**"order-0.000069148/182full transcript →
google/gemma-3-27b-it✕FAILanswered "**second**"verified-stableorder-0.000062143/159full transcript →
google/gemma-3-27b-it✓PASSanswered "8:00 is before 8:04, so moving the folder happened first."order-0.000034147/63full transcript →
google/gemma-3-27b-it✓PASSanswered "The answer is **first**."order-0.000037142/76full transcript →
google/gemma-3-27b-it✓PASSanswered "**Answer: first**"order-0.000050263/79full transcript →
google/gemma-3-27b-it✓PASSanswered "Therefore, the answer is **first**."order-0.000043142/97full transcript →
google/gemma-3-27b-it✓PASSanswered "Therefore, the answer is **first**."order-0.000041280/45full transcript →
google/gemma-3-27b-it✓PASSanswered "Therefore, the answer is **first**."order-0.000047272/65full transcript →
google/gemma-3-27b-it✓PASSanswered "**Answer: first**"order-0.000054280/86full transcript →
google/gemma-3-27b-it✓PASSanswered "The answer is **first**."order-0.000039148/79full transcript →
google/gemma-3-27b-it✓PASSanswered "**Answer: first**"order-0.000075274/160full transcript →
google/gemma-3-27b-it✓PASSanswered "**first**"order-0.000046143/105full transcript →
google/gemma-3-27b-it✓PASSanswered "**Answer: first**"order-0.000063386/83full transcript →
google/gemma-3-27b-it✓PASSanswered "Answer: **first**"order-0.000071401/102full transcript →
google/gemma-3-27b-it✓PASSanswered "**Answer: first**"order-0.000067394/92full transcript →
google/gemma-3-27b-it✓PASSanswered "Answer: first"order-0.000065381/89full transcript →
google/gemma-3-27b-it✓PASSanswered "**Answer: first**"order-0.000082506/106full transcript →
google/gemma-3-27b-it✓PASSanswered "Answer: **first**"order-0.000081513/98full transcript →
google/gemma-3-27b-it✓PASSanswered "**Answer: first**"order-0.000091543/122full transcript →
google/gemma-3-27b-it✓PASSanswered "**Answer: first**"order-0.000078524/85full transcript →
google/gemma-3-27b-it✓PASSanswered "8:00 is before 8:32, so Willa moving the box happened first."order-0.000087653/71full transcript →
google/gemma-3-27b-it✓PASSanswered "Answer: **first**"order-0.000088627/84full transcript →
google/gemma-3-27b-it✓PASSanswered "Answer: **first**"order-0.000077526/83full transcript →
google/gemma-3-27b-it✓PASSanswered "Answer: **first**"order-0.000093651/93full transcript →
google/gemma-3-27b-it✓PASSanswered "Therefore, the answer is **first**."order-0.000101660/115full transcript →
google/gemma-3-27b-it✓PASSanswered "**Answer: first**"order-0.000068394/94full transcript →
google/gemma-3-27b-it✓PASSanswered "Answer: first"order-0.000088612/90full transcript →
google/gemma-3-27b-it✓PASSanswered "first"order-0.000016147/3full transcript →
google/gemma-3-27b-it✓PASSanswered "8:00 is before 8:05, so Finn moving the folder to the uti..."order-0.000033148/62full transcript →
google/gemma-3-27b-it✓PASSanswered "**Answer: first**"order-0.000047146/109full transcript →
google/gemma-3-27b-it✓PASSanswered "**Answer: first**"order-0.000036152/71full transcript →
google/gemma-3-27b-it✓PASSanswered "Therefore, the answer is **first**."order-0.000039144/81full transcript →