Data to …… trials

Run camp-tool-depth-gemma-003

64% PASS

What this run revealed, then every individual trial as evidence.

77
Trials
64%
49/77 scored
over trials
Pass rate
28
36%
Failures
$0.0154
Run cost

Run camp-tool-depth-gemma-003: 64% pass rate on tool queries.

Of 77 scored trials (tool query kind), the model passed 64% and failed 36%.

28 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record 3fec9dbe — asked "You may use at most 14 tool calls."
    expected: (unknown)
    actual: ```

    answer did not match the verified ground truth

  • record 1f68c34c — asked "You may use at most 18 tool calls."
    expected: (unknown)
    actual: ```

    answer did not match the verified ground truth

  • record c4b61dd3 — asked "You may use at most 18 tool calls."
    expected: (unknown)
    actual: ```

    answer did not match the verified ground truth

Trials77 trials
Per-trial table (77 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=0)unverifiedtool_plan-0.000121769/146full transcript →
google/gemma-3-27b-it✓PASSanswered "]"tool_plan-0.000126769/163full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.000128772/168full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.000128771/168full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.000127769/168full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.000128771/168full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.000127769/168full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=0)unverifiedtool_plan-0.000153923/202full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=0)unverifiedtool_plan-0.000153927/202full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.000160923/224full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.000160923/224full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.000160924/224full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.000160923/224full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.000160923/224full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0001921077/280full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0001921077/280full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0001921077/280full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0001921078/280full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0001921077/280full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=0)unverifiedtool_plan-0.0001991079/304full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0001921079/280full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002051156/298full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=0)unverifiedtool_plan-0.0002021157/286full transcript →
google/gemma-3-27b-it✓PASSanswered "]"tool_plan-0.0002061154/303full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002091162/308full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002081154/308full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002081156/308full transcript →
google/gemma-3-27b-it✓PASSanswered "]"tool_plan-0.0002061154/303full transcript →
google/gemma-3-27b-it?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002301336/323full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002501335/387full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002511343/388full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002411332/359full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002491333/385full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002501334/388full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002491336/386full transcript →
google/gemma-3-27b-it?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002221261/321full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002301257/349full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002341259/359full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002341263/360full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002331258/357full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002331261/358full transcript →
google/gemma-3-27b-it?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002221256/322full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=0)unverifiedtool_plan-0.0002171235/312full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002241233/336full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002241233/336full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002251239/336full transcript →
google/gemma-3-27b-it✓PASSanswered "]"tool_plan-0.0002231234/331full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002241237/336full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002241234/336full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002101261/280full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002341259/360full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002341264/360full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002341264/360full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002341260/359full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002241253/329full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002341259/360full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002341262/360full transcript →
google/gemma-3-27b-it✓PASSanswered "]"tool_plan-0.0002311258/349full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002341260/360full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002341259/359full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002321257/354full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002321256/356full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002351263/362full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002341263/360full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002341264/359full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002461258/400full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002341263/360full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002261261/333full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002331260/355full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002331263/357full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.000127769/168full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.000127769/168full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.000127769/168full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.000127770/168full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=0)unverifiedtool_plan-0.000121775/144full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.000127770/168full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=0)unverifiedtool_plan-0.000115775/124full transcript →