Data to …… trials

Run camp-ef-tools-gemma4qat-001

92% PASS

What this run revealed, then every individual trial as evidence.

watch live →compare across models →
40
Trials
92%
37/40 scored
over trials
Pass rate
3
8%
Failures
$0.0094
Run cost

Run camp-ef-tools-gemma4qat-001, 4 arms: clean on A, C; below 97% on B, D.

Arm A passes cleanly (100%, 10 trials).

Arm B: 80% pass rate across 10 trials.

(B) 2 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Arm C passes cleanly (100%, 10 trials).

Arm D: 90% pass rate across 10 trials.

(D) 1 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record 7f694fdc — asked "You may use at most 26 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

  • record 2aa4fd9f — asked "You may use at most 26 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

  • record 8e652102 — asked "You may use at most 26 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

Trials40 trials
Per-trial table (40 trials -- prompt, reasoning, verdict, answer key)
modelstageverdictkinddifficultycost ($)tokens in/out
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "]"tool_plan-0.0002231237/331full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "]"tool_plan-0.0002231237/331full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "]"tool_plan-0.0002231237/331full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "]"tool_plan-0.0002231237/331full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "]"tool_plan-0.0002231237/331full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "]"tool_plan-0.0002231237/331full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "]"tool_plan-0.0002231237/331full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "]"tool_plan-0.0002231237/331full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "]"tool_plan-0.0002231241/331full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "]"tool_plan-0.0002231237/331full transcript →
gemma-4-31b-it-qat-w4a16-ctB✓PASSanswered "]"tool_plan-0.0002311260/351full transcript →
gemma-4-31b-it-qat-w4a16-ctB✓PASSanswered "]"tool_plan-0.0002321262/352full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002331264/354full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002321262/353full transcript →
gemma-4-31b-it-qat-w4a16-ctB✓PASSanswered "]"tool_plan-0.0002311260/350full transcript →
gemma-4-31b-it-qat-w4a16-ctB✓PASSanswered "]"tool_plan-0.0002321262/352full transcript →
gemma-4-31b-it-qat-w4a16-ctB✓PASSanswered "]"tool_plan-0.0002331263/355full transcript →
gemma-4-31b-it-qat-w4a16-ctB✓PASSanswered "]"tool_plan-0.0002311259/350full transcript →
gemma-4-31b-it-qat-w4a16-ctB✓PASSanswered "]"tool_plan-0.0002331267/355full transcript →
gemma-4-31b-it-qat-w4a16-ctB✓PASSanswered "]"tool_plan-0.0002311261/351full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "```"tool_plan-0.0002451384/356full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "```"tool_plan-0.0002461388/357full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "```"tool_plan-0.0002471392/359full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "]"tool_plan-0.0002451388/353full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "]"tool_plan-0.0002431384/350full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "```"tool_plan-0.0002461388/357full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "```"tool_plan-0.0002471390/360full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "]"tool_plan-0.0002431382/350full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "```"tool_plan-0.0002471394/360full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "]"tool_plan-0.0002441386/351full transcript →
gemma-4-31b-it-qat-w4a16-ctD✓PASSanswered "]"tool_plan-0.0002341291/351full transcript →
gemma-4-31b-it-qat-w4a16-ctD✓PASSanswered "]"tool_plan-0.0002351292/352full transcript →
gemma-4-31b-it-qat-w4a16-ctD?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002361294/354full transcript →
gemma-4-31b-it-qat-w4a16-ctD✓PASSanswered "]"tool_plan-0.0002351293/353full transcript →
gemma-4-31b-it-qat-w4a16-ctD✓PASSanswered "]"tool_plan-0.0002341290/350full transcript →
gemma-4-31b-it-qat-w4a16-ctD✓PASSanswered "]"tool_plan-0.0002351292/352full transcript →
gemma-4-31b-it-qat-w4a16-ctD✓PASSanswered "]"tool_plan-0.0002361295/355full transcript →
gemma-4-31b-it-qat-w4a16-ctD✓PASSanswered "]"tool_plan-0.0002341290/350full transcript →
gemma-4-31b-it-qat-w4a16-ctD✓PASSanswered "]"tool_plan-0.0002361299/355full transcript →
gemma-4-31b-it-qat-w4a16-ctD✓PASSanswered "]"tool_plan-0.0002341291/351full transcript →