Run camp-tool-deep-gemma-001
43% PASSWhat this run revealed, then every individual trial as evidence.
70
Trials
43%
30/70 scored
over trials
Pass rate
40
Failures
$0.0170
Run cost
Run camp-tool-deep-gemma-001: 43% pass rate on tool queries.
Of 70 scored trials (tool query kind), the model passed 43% and failed 57%.
40 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).
Notable failures
- record f527118b expected: (unknown)actual: ```
- record 239543b0 expected: (unknown)actual: ```
- record ad8f008a expected: (unknown)actual: ```
Trials
Per-trial table
| model | verdict | kind | difficulty | cost ($) | tokens in/out | |
|---|---|---|---|---|---|---|
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000133 | 790/179 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000133 | 786/181 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000126 | 787/159 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000139 | 784/202 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000133 | 786/181 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000141 | 791/207 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000126 | 791/155 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000192 | 1103/273 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000199 | 1099/298 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000201 | 1106/300 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000198 | 1096/296 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000201 | 1107/300 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000200 | 1102/301 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000200 | 1103/298 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000181 | 1339/156 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000242 | 1339/359 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000249 | 1334/387 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000250 | 1337/389 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000248 | 1333/383 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000250 | 1335/387 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000250 | 1336/388 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000310 | 1657/481 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000318 | 1658/506 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000320 | 1667/511 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000319 | 1658/512 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000319 | 1656/510 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000319 | 1656/510 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000316 | 1652/504 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000248 | 1497/329 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "]" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000282 | 1495/443 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "]" | tool_plan | - | 0.000281 | 1493/440 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000284 | 1495/449 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000282 | 1490/445 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000284 | 1494/448 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000284 | 1493/448 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000290 | 1569/445 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000300 | 1570/475 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "]" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000300 | 1577/476 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000300 | 1573/477 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000300 | 1571/476 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000300 | 1571/478 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000302 | 1576/481 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000283 | 1496/444 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000285 | 1496/450 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000286 | 1502/451 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000282 | 1490/445 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000283 | 1492/446 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000284 | 1495/449 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000285 | 1496/450 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000281 | 1487/441 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000285 | 1494/451 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000282 | 1490/445 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000283 | 1492/446 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "]" | tool_plan | - | 0.000282 | 1492/441 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000290 | 1494/470 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000282 | 1490/445 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "]" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000193 | 1491/147 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000285 | 1496/451 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000282 | 1489/444 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000284 | 1494/449 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000283 | 1491/446 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000276 | 1492/422 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000282 | 1489/445 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000142 | 794/208 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000118 | 785/131 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000125 | 787/153 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000132 | 782/179 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000151 | 788/241 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000118 | 785/133 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000125 | 787/154 | full transcript → |