Data to …… trials

Run camp-ef-repair-gemma4qat-001

29% PASS

What this run revealed, then every individual trial as evidence.

watch live →compare across models →
50
2 errors
Trials
29%
14/48 scored
over trials
Pass rate
34
68%
Failures
$0.4294
Run cost

Run camp-ef-repair-gemma4qat-001, 5 arms: clean on none; below 97% on A, B, C, D, X.

Arm A: 40% pass rate across 10 trials.

(A) 6 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Arm B: 33% pass rate across 10 trials.

(B) 6 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Arm C: 30% pass rate across 10 trials.

(C) 7 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Arm D: 30% pass rate across 10 trials.

(D) 7 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Arm X: 11% pass rate across 10 trials.

(X) 8 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record 7139aab2 — asked "tests/test_audit.py"
    expected: (unknown)
    actual: {"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "test...

    answer did not match the verified ground truth

  • record 7831bb4f — asked "utils.py"
    expected: (unknown)
    actual: {"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "test...

    answer did not match the verified ground truth

  • record 6acaf6d4 — asked "tests/test_audit.py"
    expected: (unknown)
    actual: {"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "test...

    answer did not match the verified ground truth

Trials50 trials
Per-trial table (50 trials -- prompt, reasoning, verdict, answer key)
modelstageverdictkinddifficultycost ($)tokens in/out
gemma-4-31b-it-qat-w4a16-ctA?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00323831832/184full transcript →
gemma-4-31b-it-qat-w4a16-ctA?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.011305111805/415full transcript →
gemma-4-31b-it-qat-w4a16-ctA?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00301327735/797full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.00727571410/446full transcript →
gemma-4-31b-it-qat-w4a16-ctA?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00998598361/498full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.00326931472/407full transcript →
gemma-4-31b-it-qat-w4a16-ctA?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00868985517/457full transcript →
gemma-4-31b-it-qat-w4a16-ctA?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00408922497/6132full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.00598658849/337full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.00469446058/295full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.012046119126/446full transcript →
gemma-4-31b-it-qat-w4a16-ctB✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.00953191315/1330full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00261823751/809full transcript →
gemma-4-31b-it-qat-w4a16-ctB✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.00931491626/503full transcript →
gemma-4-31b-it-qat-w4a16-ctB✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.010779106164/542full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.01148691288/7856full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00944292978/479full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00409022499/6132full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00971695761/466full transcript →
gemma-4-31b-it-qat-w4a16-ctB✕ERRORtimeout read_timeout:360s attempts:1 elapsed:511.4srepo_fix-0.00468546098/250full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "{"steps": [{"step": 1, "tool": "read_file", "args": {"pat..."repo_fix-0.00982293545/1558full transcript →
gemma-4-31b-it-qat-w4a16-ctC?FAILanswered "{"steps": [{"step": 1, "tool": "list_files", "args": {}, ..."unverifiedrepo_fix-0.018065171265/3128full transcript →
gemma-4-31b-it-qat-w4a16-ctC?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.01000398579/483full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "{"steps": [{"step": 1, "tool": "list_files", "args": {}, ..."repo_fix-0.00724571409/348full transcript →
gemma-4-31b-it-qat-w4a16-ctC?FAILanswered "{"steps": [{"step": 1, "tool": "list_files", "args": {}, ..."unverifiedrepo_fix-0.00773357405/6640full transcript →
gemma-4-31b-it-qat-w4a16-ctC?FAILanswered "{"steps": [{"step": 1, "tool": "read_file", "args": {"pat..."unverifiedrepo_fix-0.015456153100/486full transcript →
gemma-4-31b-it-qat-w4a16-ctC?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.010313101517/536full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.00568755860/336full transcript →
gemma-4-31b-it-qat-w4a16-ctC?FAILanswered "{"steps": [{"step": 1, "tool": "list_files", "args": {}, ..."unverifiedrepo_fix-0.011780116348/485full transcript →
gemma-4-31b-it-qat-w4a16-ctC?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00616243081/6179full transcript →
gemma-4-31b-it-qat-w4a16-ctD?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.013677135380/462full transcript →
gemma-4-31b-it-qat-w4a16-ctD✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.00985294770/1251full transcript →
gemma-4-31b-it-qat-w4a16-ctD?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00302827855/809full transcript →
gemma-4-31b-it-qat-w4a16-ctD✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.00768375578/416full transcript →
gemma-4-31b-it-qat-w4a16-ctD✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.00882886792/496full transcript →
gemma-4-31b-it-qat-w4a16-ctD?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00806479074/522full transcript →
gemma-4-31b-it-qat-w4a16-ctD?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.01228879909/14325full transcript →
gemma-4-31b-it-qat-w4a16-ctD?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00951593758/465full transcript →
gemma-4-31b-it-qat-w4a16-ctD?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00410022601/6132full transcript →
gemma-4-31b-it-qat-w4a16-ctD?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00899488409/510full transcript →
gemma-4-31b-it-qat-w4a16-ctX?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.013132129910/471full transcript →
gemma-4-31b-it-qat-w4a16-ctX?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00343231785/844full transcript →
gemma-4-31b-it-qat-w4a16-ctX?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.011448112974/502full transcript →
gemma-4-31b-it-qat-w4a16-ctX✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.014287120124/7583full transcript →
gemma-4-31b-it-qat-w4a16-ctX?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00480544410/1215full transcript →
gemma-4-31b-it-qat-w4a16-ctX?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00628361510/440full transcript →
gemma-4-31b-it-qat-w4a16-ctX?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.014820128729/6489full transcript →
gemma-4-31b-it-qat-w4a16-ctX?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.011265111368/429full transcript →
gemma-4-31b-it-qat-w4a16-ctX?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.015260123243/9785full transcript →
gemma-4-31b-it-qat-w4a16-ctX✕ERRORtimeout read_timeout:360s attempts:1 elapsed:360.6srepo_fix-0.00113211050/89full transcript →