Run smoke-vi-gemma4qat-4

FAILED

What this run revealed, then every individual trial as evidence.

1 error

Trials

0/5 scored

over trials

Pass rate

83%

Failures

$0.0456

Run cost

Run smoke-vi-gemma4qat-4: 0% pass rate on repo queries.

Of 5 scored trials (repo query kind), the model passed 0% and failed 100%.

1 additional trial(s) ended in ERROR (infra failures, cost cap, or cancellation) and are excluded from the pass rate above.

5 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

record 2c7e07b0 — asked "tests/test_scheduling.py"
expected: (unknown)
actual: {"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "test...
answer did not match the verified ground truth
record 7d3ea34e — asked "tests/test_accounts.py"
expected: (unknown)
actual: {"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "test...
answer did not match the verified ground truth
record 8f90d577 — asked "tests/test_budgeting.py"
expected: (unknown)
actual: {"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "test...
answer did not match the verified ground truth

Trials6 trials

Per-trial table (6 trials -- prompt, reasoning, verdict, answer key)

model	verdict	kind	difficulty	cost ($)	tokens in/out
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverified	repo_fix	-	0.003513	34597/179	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverified	repo_fix	-	0.003708	35373/569	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverified	repo_fix	-	0.004735	46449/299	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverified	repo_fix	-	0.001972	11714/2670	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverified	repo_fix	-	0.004985	41344/2835	full transcript →
gemma-4-31b-it-qat-w4a16-ct	✕ERRORlength	repo_fix	-	0.026718	246637/6846	full transcript →