Run smoke-gemma-interactive-text

FAILED

What this run revealed, then every individual trial as evidence.

Trials

0/2 scored

Pass rate

100%

Failures

$0.0012

Run cost

Run smoke-gemma-interactive-text: 0% pass rate on interactive queries.

Of 2 scored trials (interactive query kind), the model passed 0% and failed 100%.

2 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

record b45d9b00 — asked "You may make at most 14 tool calls."
expected: (unknown)
actual: [{"step": 1, "tool": "provision_service", "args": {"service_id": "svc_0"}, "ok": false, "error_ki...
answer did not match the verified ground truth
record aac03a25 — asked "You may make at most 18 tool calls."
expected: (unknown)
actual: [{"step": 1, "tool": "provision_service", "args": {"service_id": "svc_0"}, "ok": false, "error_ki...
answer did not match the verified ground truth

Trials2 trials

Per-trial table (2 trials -- prompt, reasoning, verdict, answer key)

model	verdict	kind	difficulty	cost ($)	tokens in/out
google/gemma-3-27b-it	?FAILanswered "[{"step": 1, "tool": "provision_service", "args": {"servi..."unverified	interactive	-	0.000674	5426/437	full transcript →
google/gemma-3-27b-it	?FAILanswered "[{"step": 1, "tool": "provision_service", "args": {"servi..."unverified	interactive	-	0.000508	3970/370	full transcript →