Run camp-tool-depth-gemma-003
64% PASSWhat this run revealed, then every individual trial as evidence.
77
Trials
64%
49/77 scored
over trials
Pass rate
28
Failures
$0.0154
Run cost
Run camp-tool-depth-gemma-003: 64% pass rate on tool queries.
Of 77 scored trials (tool query kind), the model passed 64% and failed 36%.
28 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).
Notable failures
- record 3fec9dbe expected: (unknown)actual: ```
- record 1f68c34c expected: (unknown)actual: ```
- record c4b61dd3 expected: (unknown)actual: ```
Trials
Per-trial table
| model | verdict | kind | difficulty | cost ($) | tokens in/out | |
|---|---|---|---|---|---|---|
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=0)unverified | tool_plan | - | 0.000121 | 769/146 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "]" | tool_plan | - | 0.000126 | 769/163 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000128 | 772/168 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000128 | 771/168 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000127 | 769/168 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000128 | 771/168 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000127 | 769/168 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=0)unverified | tool_plan | - | 0.000153 | 923/202 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=0)unverified | tool_plan | - | 0.000153 | 927/202 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000160 | 923/224 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000160 | 923/224 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000160 | 924/224 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000160 | 923/224 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000160 | 923/224 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000192 | 1077/280 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000192 | 1077/280 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000192 | 1077/280 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000192 | 1078/280 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000192 | 1077/280 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=0)unverified | tool_plan | - | 0.000199 | 1079/304 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000192 | 1079/280 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000205 | 1156/298 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=0)unverified | tool_plan | - | 0.000202 | 1157/286 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "]" | tool_plan | - | 0.000206 | 1154/303 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000209 | 1162/308 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000208 | 1154/308 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000208 | 1156/308 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "]" | tool_plan | - | 0.000206 | 1154/303 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "]" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000230 | 1336/323 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000250 | 1335/387 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000251 | 1343/388 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000241 | 1332/359 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000249 | 1333/385 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000250 | 1334/388 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000249 | 1336/386 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "]" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000222 | 1261/321 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000230 | 1257/349 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000234 | 1259/359 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000234 | 1263/360 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000233 | 1258/357 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000233 | 1261/358 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "]" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000222 | 1256/322 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=0)unverified | tool_plan | - | 0.000217 | 1235/312 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000224 | 1233/336 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000224 | 1233/336 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000225 | 1239/336 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "]" | tool_plan | - | 0.000223 | 1234/331 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000224 | 1237/336 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000224 | 1234/336 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000210 | 1261/280 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000234 | 1259/360 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000234 | 1264/360 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000234 | 1264/360 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000234 | 1260/359 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000224 | 1253/329 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000234 | 1259/360 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000234 | 1262/360 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "]" | tool_plan | - | 0.000231 | 1258/349 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000234 | 1260/360 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000234 | 1259/359 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000232 | 1257/354 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000232 | 1256/356 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000235 | 1263/362 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000234 | 1263/360 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000234 | 1264/359 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000246 | 1258/400 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000234 | 1263/360 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000226 | 1261/333 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000233 | 1260/355 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=1)unverified | tool_plan | - | 0.000233 | 1263/357 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000127 | 769/168 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000127 | 769/168 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000127 | 769/168 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000127 | 770/168 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=0)unverified | tool_plan | - | 0.000121 | 775/144 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000127 | 770/168 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: id_collisions_on=0)unverified | tool_plan | - | 0.000115 | 775/124 | full transcript → |