Data to …… trials

Run camp-full_map-gemma4qat-001/order

99% PASS

What this run revealed, then every individual trial as evidence.

80
Trials
99%
79/80 scored
over trials
Pass rate
1
1 stable / 0 noise
1%
Failures
$0.0023
Run cost

Run camp-full_map-gemma4qat-001/order: passed essentially everything (99%) -- this test did not challenge the model.

Of 80 scored trials (order query kind), the model passed 99%. That's saturation: the difficulty range probed here isn't finding this model's limits. Consider a harder preset (breaking_point or adversarial_belief) or a wider dimension range.

1 failures recorded; all 1 survived verification.

Notable failures

  • record 42b3e6bf — asked "Which happened first: Wren moving the cracked folder to the mail room or Wren's aside? Answer 'first' if the former, 'second' if the latter."
    expected: first
    actual: Since neither event described in your question actually occurred as stated, the question cannot b...

    the model recovered narration order, not the true event timestamps

Trials80 trials
Per-trial table (80 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000016156/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000015148/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000023220/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000016153/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000016150/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000015146/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000031303/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000023223/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000023222/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000023221/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000022217/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000032317/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000031306/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000032315/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000031302/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000040395/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000041404/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000041408/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000040398/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000041405/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000048473/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000048476/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000047465/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000048471/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000047465/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000015144/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000015147/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000016150/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000016150/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000016149/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000016153/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000016150/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000016150/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000015147/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000016153/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000016151/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000016155/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000015146/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000015147/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000015146/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000016154/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000015142/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000015145/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000016153/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000016154/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000015147/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000016152/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000016152/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✕FAILanswered "Since neither event described in your question actually o..."verified-stableorder-0.000044149/97full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000016156/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000015147/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000016152/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000016151/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000015146/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000015146/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000027267/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000029284/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000028276/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000028278/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000029284/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000040390/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000041405/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000040398/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000040398/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000039385/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000052510/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000052517/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000055547/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000054530/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000053528/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000066657/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000064631/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000067664/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000066655/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000062616/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000016151/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000015148/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000016152/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000016150/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "first"order-0.000016156/2full transcript →