Data to …… trials

Run camp-premise-order-gemma4qat-001

83% PASS

What this run revealed, then every individual trial as evidence.

24
Trials
83%
20/24 scored
over trials
Pass rate
4
17%
Failures
$0.0023
Run cost

Run camp-premise-order-gemma4qat-001: 83% pass rate on order queries.

Of 24 scored trials (order query kind), the model passed 83% and failed 17%.

4 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Diffuse weakness (below 90% but no outright flip) on: relational_time (pass rate 78% at the high end), temporal (pass rate 78% at the high end).

Notable failures

  • record 3931b103 — asked "Which happened first: Marcus moving the old tablet to the photo studio or Quinn's aside? Answer 'first' if the former, 'second' if the latter."
    expected: first
    actual: Marcus moving the tablet to the photo studio (8:04) happened **second**.

    the model recovered narration order, not the true event timestamps

  • record 74fab2e6 — asked "Which happened first: Petra moving the dusty tablet to the west wing or Vera's aside? Answer 'first' if the former, 'second' if the latter."
    expected: first
    actual: second

    the model recovered narration order, not the true event timestamps

  • record 82fe6fe1 — asked "Which happened first: Bruno moving the blue folder to the greenhouse or Bruno's aside? Answer 'first' if the former, 'second' if the latter."
    expected: first
    actual: first

    the model recovered narration order, not the true event timestamps

Trials24 trials
Per-trial table (24 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000027261/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000026252/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000026253/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000025242/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000026256/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000026251/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000025244/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000108239/279full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000133241/362full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000111236/292full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "Marcus moving the tablet to the photo studio (8:04) happe..."unverifiedorder-0.000131232/359full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000142237/396full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000109255/278full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000128234/348full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "second"unverifiedorder-0.000130227/356full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000127228/346full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000130242/353full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "first"unverifiedorder-0.000134226/372full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000121232/327full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000149217/424full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000158212/457full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "Since "Wren's aside" does not exist in the text, the ques..."unverifiedorder-0.000068246/144full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000126210/351full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000142219/402full transcript →