Data to …… trials

Run camp-tool-confuse-gemma-001

82% PASS

What this run revealed, then every individual trial as evidence.

56
Trials
82%
46/56 scored
over trials
Pass rate
10
18%
Failures
$0.0131
Run cost

Run camp-tool-confuse-gemma-001: 82% pass rate on tool queries.

Of 56 scored trials (tool query kind), the model passed 82% and failed 18%.

10 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Diffuse weakness (below 90% but no outright flip) on: plan_overhead (pass rate 80% at the high end).

Notable failures

  • record 42722abd — asked "You may use at most 12 tool calls."
    expected: (unknown)
    actual: ```

    answer did not match the verified ground truth

  • record 5fdc4259 — asked "You may use at most 12 tool calls."
    expected: (unknown)
    actual: ```

    answer did not match the verified ground truth

  • record 0a749c2f — asked "You may use at most 12 tool calls."
    expected: (unknown)
    actual: ```

    answer did not match the verified ground truth

Trials56 trials
Per-trial table (56 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.000115716/146full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.000115709/146full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.000115708/146full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.000115713/146full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.000115713/146full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.000115708/146full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.000115715/146full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: dishonest_decoys_on=1)unverifiedtool_plan-0.0001411045/122full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: dishonest_decoys_on=1)unverifiedtool_plan-0.0001411042/122full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0001491049/146full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0001481045/146full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0001481045/146full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0001511069/146full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0001511068/146full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0001791351/146full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0001791351/146full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: dishonest_decoys_on=1)unverifiedtool_plan-0.0001731355/124full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0001801362/146full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0001791355/146full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002211349/286full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0001801359/146full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: dishonest_decoys_on=1)unverifiedtool_plan-0.0002061689/124full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002151711/146full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002131691/146full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002131689/146full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002131692/146full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: dishonest_decoys_on=1)unverifiedtool_plan-0.0002071695/124full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: dishonest_decoys_on=1)unverifiedtool_plan-0.0002131696/146full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002482040/146full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002482039/146full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: dishonest_decoys_on=1)unverifiedtool_plan-0.0002482034/148full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002492048/146full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002472030/146full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: dishonest_decoys_on=1)unverifiedtool_plan-0.0002612032/194full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: dishonest_decoys_on=1)unverifiedtool_plan-0.0002892031/285full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002812375/146full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002822385/146full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002842402/146full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002842407/146full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002832390/146full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002962375/194full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002972389/194full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0003132691/146full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0003142705/146full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0003552683/288full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0003572706/288full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0003122685/146full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0003412685/241full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0003552684/288full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: dishonest_decoys_on=1)unverifiedtool_plan-0.0003443063/124full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0003352909/146full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0003362918/146full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0003352917/146full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0003352916/146full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0003782911/290full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0003372930/146full transcript →