Run camp-ef-repair-gemma4qat-001/X

11% PASS

What this run revealed, then every individual trial as evidence.

watch live →compare across models →

1 error

Trials

11%

1/9 scored

over trials

Pass rate

80%

Failures

$0.0959

Run cost

Run camp-ef-repair-gemma4qat-001/X: 11% pass rate on repo queries.

Of 9 scored trials (repo query kind), the model passed 11% and failed 89%.

1 additional trial(s) ended in ERROR (infra failures, cost cap, or cancellation) and are excluded from the pass rate above.

8 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

record 3b5ccb4e — asked "tests/test_audit.py"
expected: (unknown)
actual: {"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "test...
answer did not match the verified ground truth
record af506f00 — asked "tests/test_reporting.py"
expected: (unknown)
actual: {"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "test...
answer did not match the verified ground truth
record 43e72710 — asked "tests/test_rules.py"
expected: (unknown)
actual: {"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "test...
answer did not match the verified ground truth

Trials10 trials

Per-trial table (10 trials -- prompt, reasoning, verdict, answer key)

model	verdict	kind	difficulty	cost ($)	tokens in/out
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverified	repo_fix	-	0.013132	129910/471	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverified	repo_fix	-	0.003432	31785/844	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverified	repo_fix	-	0.011448	112974/502	full transcript →
gemma-4-31b-it-qat-w4a16-ct	✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."	repo_fix	-	0.014287	120124/7583	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverified	repo_fix	-	0.004805	44410/1215	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverified	repo_fix	-	0.006283	61510/440	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverified	repo_fix	-	0.014820	128729/6489	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverified	repo_fix	-	0.011265	111368/429	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverified	repo_fix	-	0.015260	123243/9785	full transcript →
gemma-4-31b-it-qat-w4a16-ct	✕ERRORtimeout read_timeout:360s attempts:1 elapsed:360.6s	repo_fix	-	0.001132	11050/89	full transcript →