Run camp-ef-repair-gemma4qat-001/C

30% PASS

What this run revealed, then every individual trial as evidence.

watch live →compare across models →

Trials

30%

3/10 scored

over trials

Pass rate

70%

Failures

$0.1023

Run cost

Run camp-ef-repair-gemma4qat-001/C: 30% pass rate on repo queries.

Of 10 scored trials (repo query kind), the model passed 30% and failed 70%.

7 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

record 7bcdefa6 — asked "tests/test_audit.py"
expected: (unknown)
actual: {"steps": [{"step": 1, "tool": "list_files", "args": {}, "ok": true, "kind": null, "output": "acc...
answer did not match the verified ground truth
record b3464cd7 — asked "tests/test_reporting.py"
expected: (unknown)
actual: {"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "test...
answer did not match the verified ground truth
record 5bf6ef7b — asked "tests/test_ranking.py"
expected: (unknown)
actual: {"steps": [{"step": 1, "tool": "list_files", "args": {}, "ok": true, "kind": null, "output": "inv...
answer did not match the verified ground truth

Trials10 trials

Per-trial table (10 trials -- prompt, reasoning, verdict, answer key)

model	verdict	kind	difficulty	cost ($)	tokens in/out
gemma-4-31b-it-qat-w4a16-ct	✓PASSanswered "{"steps": [{"step": 1, "tool": "read_file", "args": {"pat..."	repo_fix	-	0.009822	93545/1558	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "{"steps": [{"step": 1, "tool": "list_files", "args": {}, ..."unverified	repo_fix	-	0.018065	171265/3128	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverified	repo_fix	-	0.010003	98579/483	full transcript →
gemma-4-31b-it-qat-w4a16-ct	✓PASSanswered "{"steps": [{"step": 1, "tool": "list_files", "args": {}, ..."	repo_fix	-	0.007245	71409/348	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "{"steps": [{"step": 1, "tool": "list_files", "args": {}, ..."unverified	repo_fix	-	0.007733	57405/6640	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "{"steps": [{"step": 1, "tool": "read_file", "args": {"pat..."unverified	repo_fix	-	0.015456	153100/486	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverified	repo_fix	-	0.010313	101517/536	full transcript →
gemma-4-31b-it-qat-w4a16-ct	✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."	repo_fix	-	0.005687	55860/336	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "{"steps": [{"step": 1, "tool": "list_files", "args": {}, ..."unverified	repo_fix	-	0.011780	116348/485	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverified	repo_fix	-	0.006162	43081/6179	full transcript →