Data to …… trials

Run camp-tool-depth-gemma4qat-003

84% PASS

What this run revealed, then every individual trial as evidence.

106
Trials
84%
89/106 scored
over trials
Pass rate
17
16%
Failures
$0.0247
Run cost

Run camp-tool-depth-gemma4qat-003: 84% pass rate on tool queries.

Of 106 scored trials (tool query kind), the model passed 84% and failed 16%.

17 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Diffuse weakness (below 90% but no outright flip) on: catalog_size (pass rate 62% at the high end), chain_length (pass rate 62% at the high end), id_collisions_on (pass rate 68% at the high end), realized_plan_depth (pass rate 62% at the high end).

Notable failures

  • record b092f6da — asked "You may use at most 28 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

  • record 58567a80 — asked "You may use at most 32 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

  • record 1096d6e3 — asked "You may use at most 32 tool calls."
    expected: (unknown)
    actual: ]

    answer did not match the verified ground truth

Trials106 trials
Per-trial table (106 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.000127776/163full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.000126773/163full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.000126773/163full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.000126773/163full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.000126775/163full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.000126775/163full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.000126773/163full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.000158927/219full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.000159928/219full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.000158927/219full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.000158927/219full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.000159931/219full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.000158927/219full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.000158927/219full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0001911082/275full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0001911081/275full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0001911083/275full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0001911081/275full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0001911081/275full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0001911081/275full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0001911083/275full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002081166/303full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002071161/303full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002071158/303full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002071160/303full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002071160/303full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002071158/303full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002071158/303full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002471336/379full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002501347/384full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002491339/382full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002481337/380full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002491338/383full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002491340/382full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002481340/381full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002821500/441full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002821496/442full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002841504/445full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "```"tool_plan-0.0002841503/447full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002851503/449full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "```"tool_plan-0.0002861503/451full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002821501/441full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0003181662/507full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0003181665/504full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "```"tool_plan-0.0003221671/515full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "```"tool_plan-0.0003191664/510full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "```"tool_plan-0.0003191662/509full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "```"tool_plan-0.0003191659/509full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0003171661/504full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0003341740/534full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0003371748/541full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "```"tool_plan-0.0003361741/539full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "```"tool_plan-0.0003371742/542full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0003371743/542full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "```"tool_plan-0.0003371744/541full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0003361740/539full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.000126773/163full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.000126775/163full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.000126773/163full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.000126773/163full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.000127776/163full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.000126775/163full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.000126773/163full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.000159928/219full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.000158927/219full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.000158927/219full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.000158927/219full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.000159931/219full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.000158927/219full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0001911082/275full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0001911081/275full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0001911081/275full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0001911081/275full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0001911081/275full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0001911083/275full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002071161/303full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002071158/303full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002071160/303full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002071158/303full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002071160/303full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002081166/303full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002491340/382full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002491338/383full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002471336/379full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002481337/380full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002491339/382full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002501347/384full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002481340/381full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002821496/442full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0002841504/445full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002821500/441full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002851503/449full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "```"tool_plan-0.0002861503/451full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "```"tool_plan-0.0002841503/447full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "```"tool_plan-0.0003191659/509full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "```"tool_plan-0.0003191664/510full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "```"tool_plan-0.0003191662/509full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "```"tool_plan-0.0003221671/515full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0003181665/504full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0003181662/507full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "```"tool_plan-0.0003361741/539full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0003371743/542full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "```"tool_plan-0.0003371742/542full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0003371748/541full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "]"tool_plan-0.0003341740/534full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0003441740/568full transcript →