Data to …… trials

Run camp-code-traps-gemma-001

64% PASS

What this run revealed, then every individual trial as evidence.

watch live →compare across models →
133
Trials
64%
85/133 scored
over trials
Pass rate
48
36%
Failures
$0.0306
Run cost

Run camp-code-traps-gemma-001: 64% pass rate on code queries.

Of 133 scored trials (code query kind), the model passed 64% and failed 36%.

48 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record fc08591f — asked "After this program runs, what is the value of `rate`? Work through it carefully, then give the final answer on its own last line as `ANSWER: <python literal>`."
    expected: (unknown)
    actual: ANSWER: 99

    answer did not match the verified ground truth

  • record ba4bd0a7 — asked "After this program runs, what is the value of `rate`? Work through it carefully, then give the final answer on its own last line as `ANSWER: <python literal>`."
    expected: (unknown)
    actual: ANSWER: 69

    answer did not match the verified ground truth

  • record 616c2fbe — asked "After this program runs, what is the value of `offset`? Work through it carefully, then give the final answer on its own last line as `ANSWER: <python literal>`."
    expected: (unknown)
    actual: ANSWER: 93

    answer did not match the verified ground truth

Trials133 trials
Per-trial table (133 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
google/gemma-3-27b-it✓PASSanswered "ANSWER: 9"code_trace-0.00005699/155full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 16"code_trace-0.000090111/262full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [0, 3, 2, 8, 4, 3]"code_trace-0.000112140/328full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [5, 9, 0, 2]"code_trace-0.000099106/295full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 9"code_trace-0.000099108/293full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [0, 4, 0, 9, 4, 4]"code_trace-0.000126121/379full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [8, 0, 5, 0]"code_trace-0.000098106/291full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 17"code_trace-0.000070118/194full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 8"code_trace-0.000087131/247full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 49"code_trace-0.000077157/203full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 29"code_trace-0.000100131/289full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 47"code_trace-0.000112138/329full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [3, 6, 15, 8]"code_trace-0.000116136/343full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 28"code_trace-0.000107145/307full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 12"code_trace-0.000098156/276full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 8"code_trace-0.000097167/268full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 5"code_trace-0.000096163/266full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 39"code_trace-0.000136156/403full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [8, 2, 8, 12, 9]"code_trace-0.000130170/375full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [3, 4, 2, 2]"code_trace-0.000173159/525full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 8"code_trace-0.000083148/228full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 4"code_trace-0.000128180/368full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 10"code_trace-0.000128196/360full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [7, 3, 9, 3]"code_trace-0.000129191/367full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 15"code_trace-0.000143199/409full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 9"code_trace-0.000148180/434full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 25"code_trace-0.000162213/468full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 17"code_trace-0.000141199/403full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 99" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000252288/744full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 69" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000260294/770full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 93" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000263295/779full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 96" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000224312/643full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [6, 8, 1, 1, 18, 16, 23, 0, 1, 2, 3, 4, 2, 3, 7, 3]"code_trace-0.000399291/1232full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [2, 7, 0, 2, 4, 32, 10, 0, 1, 2, 3, 4, 5, -1, 7, ..."code_trace-0.000409313/1258full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: [8, 1, 3, 3, 2, 2, 4, 4, 5, 5, 13, 8, 27, 0, 1, 2..." (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000489310/1527full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 112"code_trace-0.000254287/750full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 85"code_trace-0.000270296/800full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: [2, 5, 6, 10, 18, 8, 0, 1, 2, 3, 4, -1, 14, 0, 2]" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000319282/968full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 124" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000273286/813full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 67" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000254285/752full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [8, 18, 35, 20, 0, 1, 2, 3, 4, 5, -1, 2, 6, 4]"code_trace-0.000329300/997full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: [0, 6, 16, 30, 39, 3, 4, 5, -3, 2, 9, 4]" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000340329/1023full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 6"code_trace-0.000126206/350full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [7, 4, 7, 6]"code_trace-0.000134184/385full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 44"code_trace-0.000179214/524full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [4, 2, 0, 5]"code_trace-0.000196214/581full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 31"code_trace-0.000202198/606full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 7"code_trace-0.000147217/419full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 11"code_trace-0.000239205/730full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 58"code_trace-0.000240281/708full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 112" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000259281/771full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 72" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000271316/797full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 68" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000249296/733full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 83" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000250276/742full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [2, 4, 8, 10, 25, 13, 0, 1, 2, -2, 3, 10, 12]"code_trace-0.000356303/1085full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 136"code_trace-0.000297318/883full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 58" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000221304/634full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: [9, 4, 4, 24, 23, 0, 1, 2, 3, 0, 3, 2, 11]" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000301303/903full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 95"code_trace-0.000260312/762full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: [9, 7, 7, 8, 18, 27, 9, 0, 1, 2, 3, 4, 5, -2, 9, ..." (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000310299/933full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 88"code_trace-0.000248283/731full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 115" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000259283/770full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: [3, 8, 5, 8, 11, 16, 49, 0, 1, 2, 3, 4, 5, -2, 11..." (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000415309/1279full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [7, 7, 4, 2, 0, 0]"code_trace-0.000108115/322full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: -23" (world seeded for: aliasing_on=0)unverifiedcode_trace-0.000145116/446full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 20"code_trace-0.00005990/167full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 10"code_trace-0.000064113/176full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 17"code_trace-0.000088103/259full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 8"code_trace-0.000058105/157full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [3, 2, 2, 3, 3]"code_trace-0.000092130/265full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 128"code_trace-0.000266269/796full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 105" (world seeded for: aliasing_on=0)unverifiedcode_trace-0.000303297/910full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 82"code_trace-0.000223258/656full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: [8, 2, 6, 50, 15, 3, 4, -3, 3, 2, 5]" (world seeded for: aliasing_on=0)unverifiedcode_trace-0.000282300/841full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 80"code_trace-0.000242270/717full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [3, 4, 9, 0, 1, 9, 9, 21, 25, 0, 1, 2, 3, 4, -2, ..."code_trace-0.000478290/1497full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [6, 2, 48, 17, 0, 1, 2, 3, 3, 2, 10, 6]"code_trace-0.000269287/800full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 52"code_trace-0.000225304/649full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 85" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000232267/683full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 90" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000228267/672full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 69" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000223258/659full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 50" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000248270/737full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 45" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000246276/729full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 64" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000248273/737full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 47" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000249255/744full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 39" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000253293/746full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: [1, 7, 9, 2, 2, 4, 24, 0, 1, 2, 3, 4, 5, 3, 3, 8]" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000398286/1232full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: [1, 3, 10, 0, 1, 2, 3, 4, 5, -2, 2, 0, -1]" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000348251/1076full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 86"code_trace-0.000352268/1084full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 116" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000289250/879full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 67"code_trace-0.000295262/897full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 53" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000189243/550full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 49"code_trace-0.000208252/611full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [7, 4, 4, 4, 2, 14, 25, 13, 0, 1, 2, 1, 3]"code_trace-0.000319276/971full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: [2, 3, 8, 63, 10, 0, 1, 2, 3, 2, 4, 5]" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000246252/735full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 79"code_trace-0.000256270/762full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [0, 9, 8, 7, 12, 24, 27, 1, 2, -2, 3]"code_trace-0.000262258/788full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 92"code_trace-0.000308243/946full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 72" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000238282/701full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [2, 5, 9, 19, 14, 27, 1, 2, -1, 5, 7, 8]"code_trace-0.000295291/888full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 107" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000310268/943full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 53"code_trace-0.000287272/867full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 99"code_trace-0.000270265/811full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 75" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000282263/851full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: [11, 7, 9, 14, 20, 0, 1, 2, 3, 4, 5, -1, 12, 4, 4]" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000397306/1223full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [8, 6, 9, 19, 12, 16, 0, 1, 2, 3, -3, 14, 3, 3]"code_trace-0.000289287/867full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [2, 2, 16, 42, 20, 0, 1, 2, 3, 4, 5, 1, 2, 6, 1]"code_trace-0.000333286/1015full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [9, 4, 6, 14, 63, 18, 0, 1, 2, 3, 4, 5, -2, 11, 2..."code_trace-0.000295302/883full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [18, 5, 6, 3, 40, 11, 0, 1, 2, 3, 4, 5, -1, 13, 3..."code_trace-0.000353277/1084full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 94"code_trace-0.000283275/851full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 89" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000215264/630full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 114" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000267279/798full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 80"code_trace-0.000278275/835full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: [8, 3, 1, 14, 12, 35, 23, 1, 2, 2]" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000259254/780full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 136"code_trace-0.000338293/1029full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 74" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000292269/884full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [7, 1, 8, 10, 24, 27, 0, 1, 2, 2, 1, 4]"code_trace-0.000323299/977full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 77"code_trace-0.000265279/790full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 76"code_trace-0.000204265/593full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 46"code_trace-0.000325275/990full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 35" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000212230/629full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 60"code_trace-0.000236261/698full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 50" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000223244/663full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 103" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000233283/683full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 47"code_trace-0.000259272/772full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 87"code_trace-0.000258241/778full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: [8, 8, 1, 0, 18, 6, 15, 11, 0, 1, 2, 3, 4, -2, 11]" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000278267/837full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 126"code_trace-0.000257246/774full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 35" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000221231/661full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 47" (world seeded for: aliasing_on=1)unverifiedcode_trace-0.000247260/737full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 77"code_trace-0.000243253/725full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [3, 0, 9, 1, 7, 10, 16, 0, 1, 2, 3, 2, 3]"code_trace-0.000321261/983full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [9, 7, 7, 7, 19, 15, 0, 1, 2, 3, -2, 3, 3, 1]"code_trace-0.000271278/812full transcript →