Run camp-tool-depth-gemma4qat-003
84% PASSWhat this run revealed, then every individual trial as evidence.
106
Trials
84%
89/106 scored
over trials
Pass rate
17
Failures
$0.0247
Run cost
Run camp-tool-depth-gemma4qat-003: 84% pass rate on tool queries.
Of 106 scored trials (tool query kind), the model passed 84% and failed 16%.
17 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).
Diffuse weakness (below 90% but no outright flip) on: catalog_size (pass rate 62% at the high end), chain_length (pass rate 62% at the high end), id_collisions_on (pass rate 68% at the high end), realized_plan_depth (pass rate 62% at the high end).
Notable failures
- record b092f6da expected: (unknown)actual: ]
- record 58567a80 expected: (unknown)actual: ]
- record 1096d6e3 expected: (unknown)actual: ]
Trials
Per-trial table
| model | verdict | kind | difficulty | cost ($) | tokens in/out | |
|---|---|---|---|---|---|---|
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000127 | 776/163 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000126 | 773/163 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000126 | 773/163 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000126 | 773/163 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000126 | 775/163 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000126 | 775/163 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000126 | 773/163 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000158 | 927/219 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000159 | 928/219 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000158 | 927/219 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000158 | 927/219 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000159 | 931/219 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000158 | 927/219 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000158 | 927/219 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000191 | 1082/275 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000191 | 1081/275 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000191 | 1083/275 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000191 | 1081/275 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000191 | 1081/275 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000191 | 1081/275 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000191 | 1083/275 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000208 | 1166/303 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000207 | 1161/303 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000207 | 1158/303 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000207 | 1160/303 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000207 | 1160/303 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000207 | 1158/303 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000207 | 1158/303 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000247 | 1336/379 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000250 | 1347/384 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "]" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000249 | 1339/382 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000248 | 1337/380 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000249 | 1338/383 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000249 | 1340/382 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000248 | 1340/381 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "]" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000282 | 1500/441 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "]" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000282 | 1496/442 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000284 | 1504/445 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "```" | tool_plan | - | 0.000284 | 1503/447 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000285 | 1503/449 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "```" | tool_plan | - | 0.000286 | 1503/451 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000282 | 1501/441 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000318 | 1662/507 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000318 | 1665/504 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "```" | tool_plan | - | 0.000322 | 1671/515 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "```" | tool_plan | - | 0.000319 | 1664/510 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "```" | tool_plan | - | 0.000319 | 1662/509 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "```" | tool_plan | - | 0.000319 | 1659/509 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "]" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000317 | 1661/504 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000334 | 1740/534 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000337 | 1748/541 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "```" | tool_plan | - | 0.000336 | 1741/539 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "```" | tool_plan | - | 0.000337 | 1742/542 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000337 | 1743/542 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "```" | tool_plan | - | 0.000337 | 1744/541 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000336 | 1740/539 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000126 | 773/163 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000126 | 775/163 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000126 | 773/163 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000126 | 773/163 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000127 | 776/163 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000126 | 775/163 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000126 | 773/163 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000159 | 928/219 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000158 | 927/219 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000158 | 927/219 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000158 | 927/219 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000159 | 931/219 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000158 | 927/219 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000191 | 1082/275 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000191 | 1081/275 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000191 | 1081/275 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000191 | 1081/275 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000191 | 1081/275 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000191 | 1083/275 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000207 | 1161/303 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000207 | 1158/303 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000207 | 1160/303 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000207 | 1158/303 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000207 | 1160/303 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000208 | 1166/303 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000249 | 1340/382 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000249 | 1338/383 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000247 | 1336/379 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000248 | 1337/380 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "]" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000249 | 1339/382 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000250 | 1347/384 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000248 | 1340/381 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "]" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000282 | 1496/442 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000284 | 1504/445 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "]" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000282 | 1500/441 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000285 | 1503/449 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "```" | tool_plan | - | 0.000286 | 1503/451 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "```" | tool_plan | - | 0.000284 | 1503/447 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "```" | tool_plan | - | 0.000319 | 1659/509 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "```" | tool_plan | - | 0.000319 | 1664/510 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "```" | tool_plan | - | 0.000319 | 1662/509 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "```" | tool_plan | - | 0.000322 | 1671/515 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000318 | 1665/504 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000318 | 1662/507 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "```" | tool_plan | - | 0.000336 | 1741/539 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000337 | 1743/542 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "```" | tool_plan | - | 0.000337 | 1742/542 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000337 | 1748/541 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ✓PASSanswered "]" | tool_plan | - | 0.000334 | 1740/534 | full transcript → |
| gemma-4-31b-it-qat-w4a16-ct | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000344 | 1740/568 | full transcript → |