Data to …… trials

Run camp-repo-navigate-gemma4qat-002

24% PASS

What this run revealed, then every individual trial as evidence.

watch live →compare across models →
18
1 error
Trials
24%
4/17 scored
over trials
Pass rate
13
72%
Failures
$0.1806
Run cost

Run camp-repo-navigate-gemma4qat-002: 24% pass rate on repo queries.

Of 17 scored trials (repo query kind), the model passed 24% and failed 76%.

1 additional trial(s) ended in ERROR (infra failures, cost cap, or cancellation) and are excluded from the pass rate above.

13 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record 8304acee — asked "tests/test_discounts.py"
    expected: (unknown)
    actual: {"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "test...

    answer did not match the verified ground truth

  • record ca30dbbd — asked "tests/test_discounts.py"
    expected: (unknown)
    actual: {"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "test...

    answer did not match the verified ground truth

  • record 531fc7a8 — asked "tests/test_ingest.py"
    expected: (unknown)
    actual: {"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "test...

    answer did not match the verified ground truth

Trials18 trials
Per-trial table (18 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00531651278/626full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.011003108444/530full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00877083258/1482full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.011154110161/461full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.00503448443/632full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.00942892972/435full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.014508128641/5479full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.010678105624/386full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.013884129056/3260full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.010762105582/678full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORlengthrepo_fix-0.00863367513/6273full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.00845882974/536full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00355634741/272full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "read_file", "args": {"pat..."unverifiedrepo_fix-0.014574108685/12350full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "read_file", "args": {"pat..."unverifiedrepo_fix-0.013816116987/7057full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.01000398494/512full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00752456303/6311full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.01355098378/12375full transcript →