Run smoke-vi-gemma4qat

33% PASS

What this run revealed, then every individual trial as evidence.

3 errors

Trials

33%

1/3 scored

Pass rate

33%

Failures

$0.0263

Run cost

Run smoke-vi-gemma4qat: 33% pass rate on repo queries.

Of 3 scored trials (repo query kind), the model passed 33% and failed 67%.

3 additional trial(s) ended in ERROR (infra failures, cost cap, or cancellation) and are excluded from the pass rate above.

2 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

record 71f19214 — asked "tests/test_scheduling.py"
expected: (unknown)
actual: {"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "test...
answer did not match the verified ground truth
record 4c6ae52a — asked "tests/test_scheduling.py"
expected: (unknown)
actual: {"steps": [{"step": 1, "tool": "read_file", "args": {"path": "scheduling.py"}, "ok": true, "kind"...
answer did not match the verified ground truth

Trials6 trials

Per-trial table (6 trials -- prompt, reasoning, verdict, answer key)

model	verdict	kind	difficulty	cost ($)	tokens in/out
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverified	repo_fix	-	0.002206	7765/4764	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "{"steps": [{"step": 1, "tool": "read_file", "args": {"pat..."unverified	repo_fix	-	0.003958	18525/7019	full transcript →
gemma-4-31b-it-qat-w4a16-ct	✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."	repo_fix	-	0.005470	17041/12552	full transcript →
gemma-4-31b-it-qat-w4a16-ct	✕ERRORhttp 303: b''	repo_fix	-	0.004905	1050/16000	full transcript →
gemma-4-31b-it-qat-w4a16-ct	✕ERRORhttp 303: b''	repo_fix	-	0.004906	1063/16000	full transcript →
gemma-4-31b-it-qat-w4a16-ct	✕ERRORhttp 303: b''	repo_fix	-	0.004904	1043/16000	full transcript →