Data to …… trials

Run camp-repo-graph-gemma4qat-002

26% PASS

What this run revealed, then every individual trial as evidence.

watch live →compare across models →
30
3 errors
Trials
26%
7/27 scored
over trials
Pass rate
20
67%
Failures
$0.3084
Run cost

Run camp-repo-graph-gemma4qat-002, 2 arms: clean on none; below 97% on graph_off, graph_on.

Arm graph_off: 29% pass rate across 15 trials.

(graph_off) 10 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Arm graph_on: 23% pass rate across 15 trials.

(graph_on) 10 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record 41f4cd9b — asked "tests/test_rewards.py"
    expected: (unknown)
    actual: {"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "test...

    answer did not match the verified ground truth

  • record cb5fa8d6 — asked "tests/test_audit.py"
    expected: (unknown)
    actual: {"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "test...

    answer did not match the verified ground truth

  • record 67020e0b — asked "tests/test_audit.py"
    expected: (unknown)
    actual: {"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "test...

    answer did not match the verified ground truth

Trials30 trials
Per-trial table (30 trials -- prompt, reasoning, verdict, answer key)
modelstageverdictkinddifficultycost ($)tokens in/out
gemma-4-31b-it-qat-w4a16-ctgraph_off?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00243422718/540full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_off✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.01004998827/554full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_off?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.012052117602/972full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_off✕ERRORlengthrepo_fix-0.00355817239/6113full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_off?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00380419683/6120full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_off?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00913289872/483full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_off?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.010645103431/1007full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_off?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.013082122137/2894full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_off?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00893270448/6292full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_off✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.00835781208/786full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_off?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00903071431/6291full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_off✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.010165100195/486full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_off✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.00954393942/496full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_off?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.012034114618/1906full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_off?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.014598116398/9859full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_on✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.011719115519/557full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_on✕ERRORlengthrepo_fix-0.00303812110/6089full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_on?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.011874111980/2254full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_on?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.01043898017/2120full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_on?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00936392206/474full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_on?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00464945750/245full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_on?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.010778104361/1140full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_on✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.011503113814/404full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_on✕ERRORlengthrepo_fix-0.00993962656/12243full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_on✓PASSanswered "{"steps": [{"step": 1, "tool": "read_file", "args": {"pat..."repo_fix-0.011472108449/2089full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_on?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.015868126022/10886full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_on?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.013426114483/6591full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_on?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.01341196888/12406full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_on?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.017318118935/18081full transcript →
gemma-4-31b-it-qat-w4a16-ctgraph_on?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.016214118714/14477full transcript →