Run camp-ef-tickets-gemma4qat-001/D

10% PASS

What this run revealed, then every individual trial as evidence.

watch live →compare across models →

Trials

10%

1/10 scored

over trials

Pass rate

90%

Failures

$0.0071

Run cost

Run camp-ef-tickets-gemma4qat-001/D: 10% pass rate on interactive queries.

Of 10 scored trials (interactive query kind), the model passed 10% and failed 90%.

9 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

record 69940033 — asked "You may make at most 22 tool calls."
expected: (unknown)
actual: [{"step": 1, "tool": "advance_0", "args": {"service": "svc_0"}, "ok": true, "error_kind": null, "...
answer did not match the verified ground truth
record fcb0f90a — asked "You may make at most 22 tool calls."
expected: (unknown)
actual: [{"step": 1, "tool": "advance_0", "args": {"service": "svc_0"}, "ok": true, "error_kind": null, "...
answer did not match the verified ground truth
record 58a4960f — asked "You may make at most 22 tool calls."
expected: (unknown)
actual: [{"step": 1, "tool": "advance_0", "args": {"service": "svc_0"}, "ok": true, "error_kind": null, "...
answer did not match the verified ground truth

Trials10 trials

Per-trial table (10 trials -- prompt, reasoning, verdict, answer key)

model	verdict	kind	difficulty	cost ($)	tokens in/out
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverified	interactive	-	0.000709	5360/578	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverified	interactive	-	0.000718	5379/600	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverified	interactive	-	0.000680	5181/539	full transcript →
gemma-4-31b-it-qat-w4a16-ct	✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."	interactive	-	0.000774	5407/777	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverified	interactive	-	0.000657	5256/439	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "[{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag..."unverified	interactive	-	0.000703	5354/559	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverified	interactive	-	0.000724	5406/611	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverified	interactive	-	0.000712	5342/592	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverified	interactive	-	0.000716	5382/594	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverified	interactive	-	0.000713	5349/593	full transcript →