Data to …… trials

Run camp-repo-graph-gemma4qat-001

25% PASS

What this run revealed, then every individual trial as evidence.

watch live →compare across models →
16
4 errors
Trials
25%
3/12 scored
over trials
Pass rate
9
56%
Failures
$0.1469
Run cost

Run camp-repo-graph-gemma4qat-001, 2 arms: clean on none; below 97% on graph_off, graph_on.

Arm graph_off: 14% pass rate across 8 trials.

(graph_off) 6 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Arm graph_on: 40% pass rate across 8 trials.

(graph_on) 3 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record 4046631d — asked "utils.py"
    expected: (unknown)
    actual: {"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "test...

    answer did not match the verified ground truth

  • record 85456b73 — asked "utils.py"
    expected: (unknown)
    actual: {"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "test...

    answer did not match the verified ground truth

  • record 736f858a — asked "tests/test_ledger.py"
    expected: (unknown)
    actual: {"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "test...

    answer did not match the verified ground truth

Trials16 trials
Per-trial table (16 trials -- prompt, reasoning, verdict, answer key)
modelstageverdictkinddifficultycost ($)tokens in/out
gemma-4-31b-it-qat-w4a16-ctgraph_off?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00874686214/416full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_off?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00912289868/451full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_off?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.010711104342/923full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_off?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.012072112478/2749full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_off✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.00962992415/1292full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_off✕ERRORlengthrepo_fix-0.00672348561/6222full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_off?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.010864103756/1627full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_off?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00993296758/853full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_on?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00464945750/245full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_on?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00936392206/474full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_on✕ERRORlengthrepo_fix-0.01094990405/6363full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_on✕ERRORlengthrepo_fix-0.00841765370/6268full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_on✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.012956112629/5642full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_on?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.010487100078/1598full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_on✓PASSanswered "{"steps": [{"step": 1, "tool": "read_file", "args": {"pat..."repo_fix-0.00640763000/355full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_on✕ERRORlengthrepo_fix-0.00582839693/6197full transcript →