Data to …… trials

Run camp-consistency-gemma4qat-001

47% PASS

What this run revealed, then every individual trial as evidence.

20
1 error
Trials
47%
9/19 scored
over trials
Pass rate
10
50%
Failures
$0.0047
Run cost

Run camp-consistency-gemma4qat-001: 47% pass rate on tool queries.

Of 19 scored trials (tool query kind), the model passed 47% and failed 53%.

1 additional trial(s) ended in ERROR (infra failures, cost cap, or cancellation) and are excluded from the pass rate above.

10 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record 552de0d5 — asked "You may use at most 28 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

  • record d38ee364 — asked "You may use at most 28 tool calls."
    expected: (unknown)
    actual: ```

    answer did not match the verified ground truth

  • record 080612c5 — asked "You may use at most 28 tool calls."
    expected: (unknown)
    actual: ```

    answer did not match the verified ground truth

Trials20 trials
Per-trial table (20 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002481337/382full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002481337/382full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002481337/382full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORtimeouttool_plan-0.0000000/0full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002481337/382full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002491341/384full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002511341/389full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002511341/389full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002491341/384full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002491341/384full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002491338/383full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002491338/383full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002491338/383full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002491338/383full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002491338/383full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002491340/384full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002491340/384full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002491340/384full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002491340/384full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002491340/384full transcript →