Run camp-consistency-gemma4qat-002
62% PASSWhat this run revealed, then every individual trial as evidence.
40
Trials
62%
25/40 scored
over trials
Pass rate
15
Failures
$0.0100
Run cost
Run camp-consistency-gemma4qat-002: 62% pass rate on tool queries.
Of 40 scored trials (tool query kind), the model passed 62% and failed 38%.
15 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).
Notable failures
- record 99adb007 expected: (unknown)actual: ]
- record fe9b4933 expected: (unknown)actual: ]
- record 27325b75 expected: (unknown)actual: ]
Trials
Per-trial table
| model | verdict | kind | difficulty | cost ($) | tokens in/out | |
|---|---|---|---|---|---|---|
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000248 | 1337/382 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000248 | 1337/382 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000248 | 1337/382 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000248 | 1337/382 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000248 | 1337/382 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "]" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000249 | 1341/384 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "]" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000249 | 1341/384 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "]" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000249 | 1341/384 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "]" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000249 | 1341/384 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000249 | 1338/383 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000249 | 1338/383 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "]" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000249 | 1341/384 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000249 | 1338/383 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "]" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000249 | 1340/384 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000249 | 1338/383 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000249 | 1338/383 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "]" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000249 | 1340/384 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "]" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000249 | 1340/384 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "]" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000249 | 1340/384 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "]" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000249 | 1340/384 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000250 | 1350/384 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000250 | 1350/384 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000250 | 1350/384 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000250 | 1350/384 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000250 | 1350/384 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000250 | 1343/386 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000250 | 1343/386 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000250 | 1343/386 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000250 | 1343/386 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000250 | 1343/386 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "]" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000250 | 1343/387 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "]" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000250 | 1343/387 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "]" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000250 | 1343/387 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "]" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000250 | 1343/387 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000249 | 1341/384 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "]" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000250 | 1343/387 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000249 | 1341/384 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000249 | 1341/384 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000249 | 1341/384 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000249 | 1341/384 | full transcript → |