Data to …… trials

Run camp-ef-tools-gemma4qat-001/B

80% PASS

What this run revealed, then every individual trial as evidence.

10
Trials
80%
8/10 scored
over trials
Pass rate
2
20%
Failures
$0.0023
Run cost

Run camp-ef-tools-gemma4qat-001/B: 80% pass rate on tool queries.

Of 10 scored trials (tool query kind), the model passed 80% and failed 20%.

2 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record 7f694fdc — asked "You may use at most 26 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

  • record 2aa4fd9f — asked "You may use at most 26 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

Trials10 trials
Per-trial table (10 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002311260/351full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002321262/352full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002331264/354full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002321262/353full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002311260/350full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002321262/352full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002331263/355full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002311259/350full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002331267/355full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002311261/351full transcript →