Run camp-repo-graph-gemma4qat-001/graph_off

14% PASS

What this run revealed, then every individual trial as evidence.

1 error

Trials

14%

1/7 scored

over trials

Pass rate

75%

Failures

$0.0778

Run cost

Run camp-repo-graph-gemma4qat-001/graph_off: 14% pass rate on repo queries.

Of 7 scored trials (repo query kind), the model passed 14% and failed 86%.

1 additional trial(s) ended in ERROR (infra failures, cost cap, or cancellation) and are excluded from the pass rate above.

6 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

record 4046631d — asked "utils.py"
expected: (unknown)
actual: {"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "test...
answer did not match the verified ground truth
record 85456b73 — asked "utils.py"
expected: (unknown)
actual: {"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "test...
answer did not match the verified ground truth
record 3e99ef85 — asked "tests/test_ledger.py"
expected: (unknown)
actual: {"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "test...
answer did not match the verified ground truth

Trials8 trials

Per-trial table (8 trials -- prompt, reasoning, verdict, answer key)

model	verdict	kind	difficulty	cost ($)	tokens in/out
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverified	repo_fix	-	0.008746	86214/416	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverified	repo_fix	-	0.009122	89868/451	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverified	repo_fix	-	0.010711	104342/923	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverified	repo_fix	-	0.012072	112478/2749	full transcript →
gemma-4-31b-it-qat-w4a16-ct	✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."	repo_fix	-	0.009629	92415/1292	full transcript →
gemma-4-31b-it-qat-w4a16-ct	✕ERRORlength	repo_fix	-	0.006723	48561/6222	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverified	repo_fix	-	0.010864	103756/1627	full transcript →
gemma-4-31b-it-qat-w4a16-ct	?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverified	repo_fix	-	0.009932	96758/853	full transcript →