Data to …… trials

Run camp-repo-control-gemma4qat-001

35% PASS

What this run revealed, then every individual trial as evidence.

watch live →compare across models →
24
7 errors
Trials
35%
6/17 scored
over trials
Pass rate
11
46%
Failures
$0.2229
Run cost

Run camp-repo-control-gemma4qat-001: 35% pass rate on repo queries.

Of 17 scored trials (repo query kind), the model passed 35% and failed 65%.

7 additional trial(s) ended in ERROR (infra failures, cost cap, or cancellation) and are excluded from the pass rate above.

11 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record 13803960 — asked "tests/test_scoring.py"
    expected: (unknown)
    actual: {"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "test...

    answer did not match the verified ground truth

  • record 675fc0dc — asked "tests/test_grading.py"
    expected: (unknown)
    actual: {"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "test...

    answer did not match the verified ground truth

  • record af87ccab — asked "tests/test_scoring.py"
    expected: (unknown)
    actual: {"steps": [{"step": 1, "tool": "read_file", "args": {"path": "scoring.py"}, "ok": true, "kind": n...

    answer did not match the verified ground truth

Trials24 trials
Per-trial table (24 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00435141691/605full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.010277101111/553full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "read_file", "args": {"pat..."unverifiedrepo_fix-0.00723463688/2884full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.01041394997/3044full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00981196492/539full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00889087389/504full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORlengthrepo_fix-0.00539135360/6183full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "{"steps": [{"step": 1, "tool": "read_file", "args": {"pat..."repo_fix-0.01007898912/622full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.00732869396/1294full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.00956881254/4807full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORlengthrepo_fix-0.00753256480/6281full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.00797874267/1837full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.016236160762/532full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORlengthrepo_fix-0.020600186218/6595full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORlengthrepo_fix-0.00793260488/6276full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORlengthrepo_fix-0.01004681608/6285full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.01158497812/6008full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00434341691/580full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00453643420/646full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.014323141866/455full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "read_file", "args": {"pat..."unverifiedrepo_fix-0.00589550071/2959full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "{"steps": [{"step": 1, "tool": "read_file", "args": {"pat..."repo_fix-0.01063596998/3118full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORlengthrepo_fix-0.012074101610/6378full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORlengthrepo_fix-0.00586840074/6203full transcript →