Run camp-tool-confuse-gemma-001
82% PASSWhat this run revealed, then every individual trial as evidence.
56
Trials
82%
46/56 scored
over trials
Pass rate
10
Failures
$0.0131
Run cost
Run camp-tool-confuse-gemma-001: 82% pass rate on tool queries.
Of 56 scored trials (tool query kind), the model passed 82% and failed 18%.
10 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).
Diffuse weakness (below 90% but no outright flip) on: plan_overhead (pass rate 80% at the high end).
Notable failures
- record 42722abd expected: (unknown)actual: ```
- record 5fdc4259 expected: (unknown)actual: ```
- record 0a749c2f expected: (unknown)actual: ```
Trials
Per-trial table
| model | verdict | kind | difficulty | cost ($) | tokens in/out | |
|---|---|---|---|---|---|---|
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000115 | 716/146 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000115 | 709/146 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000115 | 708/146 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000115 | 713/146 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000115 | 713/146 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000115 | 708/146 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000115 | 715/146 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: dishonest_decoys_on=1)unverified | tool_plan | - | 0.000141 | 1045/122 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: dishonest_decoys_on=1)unverified | tool_plan | - | 0.000141 | 1042/122 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000149 | 1049/146 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000148 | 1045/146 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000148 | 1045/146 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000151 | 1069/146 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000151 | 1068/146 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000179 | 1351/146 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000179 | 1351/146 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: dishonest_decoys_on=1)unverified | tool_plan | - | 0.000173 | 1355/124 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000180 | 1362/146 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000179 | 1355/146 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000221 | 1349/286 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000180 | 1359/146 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: dishonest_decoys_on=1)unverified | tool_plan | - | 0.000206 | 1689/124 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000215 | 1711/146 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000213 | 1691/146 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000213 | 1689/146 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000213 | 1692/146 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: dishonest_decoys_on=1)unverified | tool_plan | - | 0.000207 | 1695/124 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: dishonest_decoys_on=1)unverified | tool_plan | - | 0.000213 | 1696/146 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000248 | 2040/146 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000248 | 2039/146 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: dishonest_decoys_on=1)unverified | tool_plan | - | 0.000248 | 2034/148 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000249 | 2048/146 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000247 | 2030/146 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: dishonest_decoys_on=1)unverified | tool_plan | - | 0.000261 | 2032/194 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: dishonest_decoys_on=1)unverified | tool_plan | - | 0.000289 | 2031/285 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000281 | 2375/146 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000282 | 2385/146 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000284 | 2402/146 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000284 | 2407/146 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000283 | 2390/146 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000296 | 2375/194 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000297 | 2389/194 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000313 | 2691/146 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000314 | 2705/146 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000355 | 2683/288 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000357 | 2706/288 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000312 | 2685/146 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000341 | 2685/241 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000355 | 2684/288 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "```" (world seeded for: dishonest_decoys_on=1)unverified | tool_plan | - | 0.000344 | 3063/124 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000335 | 2909/146 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000336 | 2918/146 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000335 | 2917/146 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000335 | 2916/146 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000378 | 2911/290 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "```" | tool_plan | - | 0.000337 | 2930/146 | full transcript → |