Data to …… trials

Run camp-repo-navigate-gemma4qat-001

21% PASS

What this run revealed, then every individual trial as evidence.

watch live →compare across models →
18
4 errors
Trials
21%
3/14 scored
over trials
Pass rate
11
61%
Failures
$0.1656
Run cost

Run camp-repo-navigate-gemma4qat-001: 21% pass rate on repo queries.

Of 14 scored trials (repo query kind), the model passed 21% and failed 79%.

4 additional trial(s) ended in ERROR (infra failures, cost cap, or cancellation) and are excluded from the pass rate above.

11 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record 70c8adea — asked "tests/test_discounts.py"
    expected: (unknown)
    actual: {"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "test...

    answer did not match the verified ground truth

  • record 46b242b5 — asked "tests/test_discounts.py"
    expected: (unknown)
    actual: {"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "test...

    answer did not match the verified ground truth

  • record 92f1967d — asked "tests/test_ingest.py"
    expected: (unknown)
    actual: {"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "test...

    answer did not match the verified ground truth

Trials18 trials
Per-trial table (18 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00531651278/626full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.011007108444/543full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00890584542/1502full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.011154110161/461full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.011501113527/495full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00768674817/681full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.013663123571/4353full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.011577114232/513full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.010678105624/386full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.010292101460/487full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORlengthrepo_fix-0.01189199855/6350full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORlengthrepo_fix-0.00362417919/6107full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00355634741/272full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.00935391947/529full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.01002598860/463full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.01000398494/513full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORlengthrepo_fix-0.00914672580/6292full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORlengthrepo_fix-0.00624743832/6214full transcript →