Data to …… trials

Run camp-ef-tools-gemma4qat-001/D

90% PASS

What this run revealed, then every individual trial as evidence.

watch live →compare across models →
10
Trials
90%
9/10 scored
over trials
Pass rate
1
10%
Failures
$0.0023
Run cost

Run camp-ef-tools-gemma4qat-001/D: 90% pass rate on tool queries.

Of 10 scored trials (tool query kind), the model passed 90% and failed 10%.

1 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record 8e652102 — asked "You may use at most 26 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

Trials10 trials
Per-trial table (10 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002341291/351full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002351292/352full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002361294/354full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002351293/353full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002341290/350full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002351292/352full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002361295/355full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002341290/350full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002361299/355full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002341291/351full transcript →