Data to …… trials

Run camp-repo-control-gemma4qat-003

22% PASS

What this run revealed, then every individual trial as evidence.

watch live →compare across models →
24
1 error
Trials
22%
5/23 scored
over trials
Pass rate
18
75%
Failures
$0.2564
Run cost

Run camp-repo-control-gemma4qat-003: 22% pass rate on repo queries.

Of 23 scored trials (repo query kind), the model passed 22% and failed 78%.

1 additional trial(s) ended in ERROR (infra failures, cost cap, or cancellation) and are excluded from the pass rate above.

18 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record ff8c01b4 — asked "tests/test_scoring.py"
    expected: (unknown)
    actual: {"steps": [{"step": 1, "tool": "read_file", "args": {"path": "scoring.py"}, "ok": true, "kind": n...

    answer did not match the verified ground truth

  • record 9c504f50 — asked "tests/test_grading.py"
    expected: (unknown)
    actual: {"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "test...

    answer did not match the verified ground truth

  • record 7fd04f45 — asked "tests/test_scoring.py"
    expected: (unknown)
    actual: {"steps": [{"step": 1, "tool": "read_file", "args": {"path": "scoring.py"}, "ok": true, "kind": n...

    answer did not match the verified ground truth

Trials24 trials
Per-trial table (24 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "read_file", "args": {"pat..."unverifiedrepo_fix-0.00879686450/502full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.010870107141/521full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "read_file", "args": {"pat..."unverifiedrepo_fix-0.00855466782/6251full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00506931971/6239full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.00917489819/641full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.01127993485/6436full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.00434142521/297full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00944888868/1872full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.00353734500/290full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.00526051503/366full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "read_file", "args": {"pat..."unverifiedrepo_fix-0.023170229910/597full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.014254140966/526full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "{"steps": [{"step": 1, "tool": "read_file", "args": {"pat..."repo_fix-0.00646263652/323full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.010222100812/470full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.017359136294/12433full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "read_file", "args": {"pat..."unverifiedrepo_fix-0.00719671027/311full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.014459143119/491full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00801466033/4703full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "read_file", "args": {"pat..."unverifiedrepo_fix-0.00585057904/197full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "read_file", "args": {"pat..."unverifiedrepo_fix-0.020818170596/12527full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.01008981524/6455full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.017631169619/2230full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "read_file", "args": {"pat..."unverifiedrepo_fix-0.013335119689/4554full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORlengthrepo_fix-0.01122857671/18203full transcript →