Run camp-code-lookalike-gemma-001
93% PASSWhat this run revealed, then every individual trial as evidence.
56
Trials
93%
52/56 scored
over trials
Pass rate
4
Failures
$0.0130
Run cost
Run camp-code-lookalike-gemma-001: 93% pass rate on code queries.
Of 56 scored trials (code query kind), the model passed 93% and failed 7%.
4 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).
Diffuse weakness (below 90% but no outright flip) on: identifier_confusability (pass rate 89% at the high end), n_statements (pass rate 86% at the high end), n_variables (pass rate 89% at the high end), trace_length (pass rate 84% at the high end).
Notable failures
- record 6aad64e1 expected: (unknown)actual: ANSWER: 13
- record 8ad82d45 expected: (unknown)actual: ANSWER: [9, 1, 2, 4, 52]
- record cd48bc4c expected: (unknown)actual: ANSWER: 26
Trials
Per-trial table
| model | verdict | kind | difficulty | cost ($) | tokens in/out | |
|---|---|---|---|---|---|---|
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 27" | code_trace | - | 0.000065 | 107/181 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 23" | code_trace | - | 0.000062 | 90/176 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 21" | code_trace | - | 0.000083 | 103/244 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 24" | code_trace | - | 0.000104 | 126/305 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: [5, 5, 8, 8, 5]" | code_trace | - | 0.000108 | 118/320 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: [2, 9, 4, 3]" | code_trace | - | 0.000128 | 139/381 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 15" | code_trace | - | 0.000093 | 113/274 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 16" | code_trace | - | 0.000095 | 153/267 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "ANSWER: 13" (world seeded for: aliasing_on=0)unverified | code_trace | - | 0.000096 | 141/274 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 15" | code_trace | - | 0.000097 | 143/277 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 17" | code_trace | - | 0.000091 | 128/260 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: [0, 2, 1, 0, 3]" | code_trace | - | 0.000116 | 173/329 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 16" | code_trace | - | 0.000140 | 143/419 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 15" | code_trace | - | 0.000093 | 132/267 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 15" | code_trace | - | 0.000121 | 198/339 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 14" | code_trace | - | 0.000133 | 220/371 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 21" | code_trace | - | 0.000141 | 167/413 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 45" | code_trace | - | 0.000178 | 195/528 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: [1, 7, 8, 8, 3]" | code_trace | - | 0.000216 | 212/648 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: [3, 3, 1, 4, 2, 18]" | code_trace | - | 0.000226 | 217/680 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 17" | code_trace | - | 0.000128 | 203/358 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 9" | code_trace | - | 0.000170 | 226/493 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 16" | code_trace | - | 0.000185 | 261/531 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 48" | code_trace | - | 0.000257 | 249/772 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 6" | code_trace | - | 0.000342 | 262/1051 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: [5, 4, 0, 4, 2, 4, 9, 4, 0]" | code_trace | - | 0.000276 | 295/822 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 5" | code_trace | - | 0.000178 | 236/516 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: [5, 8, 8, 2, 5, 5, 7, 7, 6, 6, 0, 0, 3, 3, 6, 6, ..." | code_trace | - | 0.000356 | 280/1093 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 17" | code_trace | - | 0.000145 | 289/387 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 15" | code_trace | - | 0.000203 | 298/579 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 14" | code_trace | - | 0.000262 | 294/777 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: [8, 4, 3, 3]" | code_trace | - | 0.000226 | 346/638 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: [0, 7, 4, 1, 3]" | code_trace | - | 0.000224 | 325/639 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 21" | code_trace | - | 0.000376 | 311/1151 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: [5, 2, 3, 9, 9, 4, 4, 1, 1, 0, 0, 3, 3, 5, 9, 9, ..." | code_trace | - | 0.000440 | 290/1370 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 82" | code_trace | - | 0.000243 | 298/712 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 14" | code_trace | - | 0.000264 | 328/771 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 20" | code_trace | - | 0.000318 | 349/944 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 18" | code_trace | - | 0.000369 | 334/1120 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 30" | code_trace | - | 0.000266 | 310/782 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: [2, 0, 1, 3]" | code_trace | - | 0.000284 | 416/808 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 8" | code_trace | - | 0.000278 | 331/817 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "ANSWER: [9, 1, 2, 4, 52]" (world seeded for: aliasing_on=0)unverified | code_trace | - | 0.000284 | 374/822 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "ANSWER: 26" (world seeded for: aliasing_on=0)unverified | code_trace | - | 0.000291 | 369/846 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 18" | code_trace | - | 0.000314 | 402/913 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: [6, 1, 4, 24, 3, 8]" | code_trace | - | 0.000312 | 375/914 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 22" | code_trace | - | 0.000358 | 356/1075 | full transcript → |
| google/gemma-3-27b-it | ?FAILanswered "ANSWER: 70" (world seeded for: aliasing_on=0)unverified | code_trace | - | 0.000482 | 406/1470 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 9" | code_trace | - | 0.000281 | 345/822 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 16" | code_trace | - | 0.000275 | 397/784 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 25" | code_trace | - | 0.000317 | 405/922 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 18" | code_trace | - | 0.000315 | 433/907 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 30" | code_trace | - | 0.000397 | 459/1169 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 44" | code_trace | - | 0.000426 | 426/1277 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: 23" | code_trace | - | 0.000432 | 435/1295 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "ANSWER: [1, 3, 5, -12, 2]" | code_trace | - | 0.000295 | 414/847 | full transcript → |