Data to …… trials

Run camp-code-lookalike-gemma-001

93% PASS

What this run revealed, then every individual trial as evidence.

56
Trials
93%
52/56 scored
over trials
Pass rate
4
7%
Failures
$0.0130
Run cost

Run camp-code-lookalike-gemma-001: 93% pass rate on code queries.

Of 56 scored trials (code query kind), the model passed 93% and failed 7%.

4 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Diffuse weakness (below 90% but no outright flip) on: identifier_confusability (pass rate 89% at the high end), n_statements (pass rate 86% at the high end), n_variables (pass rate 89% at the high end), trace_length (pass rate 84% at the high end).

Notable failures

  • record 6aad64e1 — asked "After this program runs, what is the value of `step`? Work through it carefully, then give the final answer on its own last line as `ANSWER: <python literal>`."
    expected: (unknown)
    actual: ANSWER: 13

    answer did not match the verified ground truth

  • record 8ad82d45 — asked "After this program runs, what is the value of `bins`? Work through it carefully, then give the final answer on its own last line as `ANSWER: <python literal>`."
    expected: (unknown)
    actual: ANSWER: [9, 1, 2, 4, 52]

    answer did not match the verified ground truth

  • record cd48bc4c — asked "After this program runs, what is the value of `extra`? Work through it carefully, then give the final answer on its own last line as `ANSWER: <python literal>`."
    expected: (unknown)
    actual: ANSWER: 26

    answer did not match the verified ground truth

Trials56 trials
Per-trial table (56 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
google/gemma-3-27b-it✓PASSanswered "ANSWER: 27"code_trace-0.000065107/181full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 23"code_trace-0.00006290/176full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 21"code_trace-0.000083103/244full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 24"code_trace-0.000104126/305full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [5, 5, 8, 8, 5]"code_trace-0.000108118/320full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [2, 9, 4, 3]"code_trace-0.000128139/381full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 15"code_trace-0.000093113/274full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 16"code_trace-0.000095153/267full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 13" (world seeded for: aliasing_on=0)unverifiedcode_trace-0.000096141/274full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 15"code_trace-0.000097143/277full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 17"code_trace-0.000091128/260full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [0, 2, 1, 0, 3]"code_trace-0.000116173/329full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 16"code_trace-0.000140143/419full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 15"code_trace-0.000093132/267full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 15"code_trace-0.000121198/339full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 14"code_trace-0.000133220/371full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 21"code_trace-0.000141167/413full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 45"code_trace-0.000178195/528full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [1, 7, 8, 8, 3]"code_trace-0.000216212/648full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [3, 3, 1, 4, 2, 18]"code_trace-0.000226217/680full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 17"code_trace-0.000128203/358full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 9"code_trace-0.000170226/493full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 16"code_trace-0.000185261/531full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 48"code_trace-0.000257249/772full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 6"code_trace-0.000342262/1051full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [5, 4, 0, 4, 2, 4, 9, 4, 0]"code_trace-0.000276295/822full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 5"code_trace-0.000178236/516full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [5, 8, 8, 2, 5, 5, 7, 7, 6, 6, 0, 0, 3, 3, 6, 6, ..."code_trace-0.000356280/1093full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 17"code_trace-0.000145289/387full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 15"code_trace-0.000203298/579full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 14"code_trace-0.000262294/777full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [8, 4, 3, 3]"code_trace-0.000226346/638full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [0, 7, 4, 1, 3]"code_trace-0.000224325/639full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 21"code_trace-0.000376311/1151full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [5, 2, 3, 9, 9, 4, 4, 1, 1, 0, 0, 3, 3, 5, 9, 9, ..."code_trace-0.000440290/1370full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 82"code_trace-0.000243298/712full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 14"code_trace-0.000264328/771full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 20"code_trace-0.000318349/944full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 18"code_trace-0.000369334/1120full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 30"code_trace-0.000266310/782full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [2, 0, 1, 3]"code_trace-0.000284416/808full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 8"code_trace-0.000278331/817full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: [9, 1, 2, 4, 52]" (world seeded for: aliasing_on=0)unverifiedcode_trace-0.000284374/822full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 26" (world seeded for: aliasing_on=0)unverifiedcode_trace-0.000291369/846full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 18"code_trace-0.000314402/913full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [6, 1, 4, 24, 3, 8]"code_trace-0.000312375/914full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 22"code_trace-0.000358356/1075full transcript →
google/gemma-3-27b-it?FAILanswered "ANSWER: 70" (world seeded for: aliasing_on=0)unverifiedcode_trace-0.000482406/1470full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 9"code_trace-0.000281345/822full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 16"code_trace-0.000275397/784full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 25"code_trace-0.000317405/922full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 18"code_trace-0.000315433/907full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 30"code_trace-0.000397459/1169full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 44"code_trace-0.000426426/1277full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: 23"code_trace-0.000432435/1295full transcript →
google/gemma-3-27b-it✓PASSanswered "ANSWER: [1, 3, 5, -12, 2]"code_trace-0.000295414/847full transcript →