Run camp-premise-order-gemma4qat-001
83% PASSWhat this run revealed, then every individual trial as evidence.
24
Trials
83%
20/24 scored
over trials
Pass rate
4
Failures
$0.0023
Run cost
Run camp-premise-order-gemma4qat-001: 83% pass rate on order queries.
Of 24 scored trials (order query kind), the model passed 83% and failed 17%.
4 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).
Diffuse weakness (below 90% but no outright flip) on: relational_time (pass rate 78% at the high end), temporal (pass rate 78% at the high end).
Notable failures
- record 3931b103 expected: firstactual: Marcus moving the tablet to the photo studio (8:04) happened **second**.
- record 74fab2e6 expected: firstactual: second
- record 82fe6fe1 expected: firstactual: first
Trials
Per-trial table
| model | verdict | kind | difficulty | cost ($) | tokens in/out | |
|---|---|---|---|---|---|---|
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "first" | order | - | 0.000027 | 261/2 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "first" | order | - | 0.000026 | 252/2 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "first" | order | - | 0.000026 | 253/2 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "first" | order | - | 0.000025 | 242/2 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "first" | order | - | 0.000026 | 256/2 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "first" | order | - | 0.000026 | 251/2 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "first" | order | - | 0.000025 | 244/2 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "first" | order | - | 0.000108 | 239/279 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "first" | order | - | 0.000133 | 241/362 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "first" | order | - | 0.000111 | 236/292 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "Marcus moving the tablet to the photo studio (8:04) happe..."unverified | order | - | 0.000131 | 232/359 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "first" | order | - | 0.000142 | 237/396 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "first" | order | - | 0.000109 | 255/278 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "first" | order | - | 0.000128 | 234/348 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "second"unverified | order | - | 0.000130 | 227/356 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "first" | order | - | 0.000127 | 228/346 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "first" | order | - | 0.000130 | 242/353 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "first"unverified | order | - | 0.000134 | 226/372 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "first" | order | - | 0.000121 | 232/327 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "first" | order | - | 0.000149 | 217/424 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "first" | order | - | 0.000158 | 212/457 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "Since "Wren's aside" does not exist in the text, the ques..."unverified | order | - | 0.000068 | 246/144 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "first" | order | - | 0.000126 | 210/351 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "first" | order | - | 0.000142 | 219/402 | full transcript → |