Data to …… trials

Run camp-tool-deep-gemma-001

43% PASS

What this run revealed, then every individual trial as evidence.

70
Trials
43%
30/70 scored
over trials
Pass rate
40
57%
Failures
$0.0170
Run cost

Run camp-tool-deep-gemma-001: 43% pass rate on tool queries.

Of 70 scored trials (tool query kind), the model passed 43% and failed 57%.

40 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record f527118b — asked "You may use at most 14 tool calls."
    expected: (unknown)
    actual: ```

    answer did not match the verified ground truth

  • record 239543b0 — asked "You may use at most 14 tool calls."
    expected: (unknown)
    actual: ```

    answer did not match the verified ground truth

  • record ad8f008a — asked "You may use at most 22 tool calls."
    expected: (unknown)
    actual: ```

    answer did not match the verified ground truth

Trials70 trials
Per-trial table (70 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.000133790/179full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.000133786/181full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.000126787/159full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.000139784/202full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.000133786/181full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.000141791/207full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.000126791/155full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0001921103/273full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0001991099/298full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002011106/300full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0001981096/296full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002011107/300full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002001102/301full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002001103/298full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0001811339/156full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002421339/359full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002491334/387full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002501337/389full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002481333/383full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002501335/387full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002501336/388full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0003101657/481full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0003181658/506full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0003201667/511full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0003191658/512full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0003191656/510full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0003191656/510full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0003161652/504full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002481497/329full transcript →
google/gemma-3-27b-it?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002821495/443full transcript →
google/gemma-3-27b-it✓PASSanswered "]"tool_plan-0.0002811493/440full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002841495/449full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002821490/445full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002841494/448full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002841493/448full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002901569/445full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0003001570/475full transcript →
google/gemma-3-27b-it?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0003001577/476full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0003001573/477full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0003001571/476full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0003001571/478full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0003021576/481full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002831496/444full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002851496/450full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002861502/451full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002821490/445full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002831492/446full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002841495/449full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002851496/450full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002811487/441full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002851494/451full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002821490/445full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002831492/446full transcript →
google/gemma-3-27b-it✓PASSanswered "]"tool_plan-0.0002821492/441full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002901494/470full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002821490/445full transcript →
google/gemma-3-27b-it?FAILanswered "]" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0001931491/147full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002851496/451full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002821489/444full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.0002841494/449full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002831491/446full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002761492/422full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.0002821489/445full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.000142794/208full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.000118785/131full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.000125787/153full transcript →
google/gemma-3-27b-it✓PASSanswered "```"tool_plan-0.000132782/179full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.000151788/241full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.000118785/133full transcript →
google/gemma-3-27b-it?FAILanswered "```" (world seeded for: id_collisions_on=1)unverifiedtool_plan-0.000125787/154full transcript →