Data to …… trials

Run camp-consistency-gemma4qat-002

62% PASS

What this run revealed, then every individual trial as evidence.

40
Trials
62%
25/40 scored
over trials
Pass rate
15
38%
Failures
$0.0100
Run cost

Run camp-consistency-gemma4qat-002: 62% pass rate on tool queries.

Of 40 scored trials (tool query kind), the model passed 62% and failed 38%.

15 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record 99adb007 — asked "You may use at most 28 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

  • record fe9b4933 — asked "You may use at most 28 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

  • record 27325b75 — asked "You may use at most 28 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

Trials40 trials
Per-trial table (40 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002481337/382full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002481337/382full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002481337/382full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002481337/382full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002481337/382full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002491341/384full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002491341/384full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002491341/384full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002491341/384full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002491338/383full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002491338/383full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002491341/384full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002491338/383full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002491340/384full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002491338/383full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002491338/383full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002491340/384full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002491340/384full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002491340/384full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002491340/384full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002501350/384full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002501350/384full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002501350/384full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002501350/384full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002501350/384full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002501343/386full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002501343/386full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002501343/386full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002501343/386full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002501343/386full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002501343/387full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002501343/387full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002501343/387full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002501343/387full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002491341/384full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002501343/387full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002491341/384full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002491341/384full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002491341/384full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002491341/384full transcript →