Data to …… trials

Run camp-repo-control-gemma4qat-002

42% PASS

What this run revealed, then every individual trial as evidence.

watch live →compare across models →
24
Trials
42%
10/24 scored
over trials
Pass rate
14
58%
Failures
$0.2559
Run cost

Run camp-repo-control-gemma4qat-002: 42% pass rate on repo queries.

Of 24 scored trials (repo query kind), the model passed 42% and failed 58%.

14 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record 545fe302 — asked "tests/test_scoring.py"
    expected: (unknown)
    actual: {"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "test...

    answer did not match the verified ground truth

  • record f03b08c1 — asked "tests/test_grading.py"
    expected: (unknown)
    actual: {"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "test...

    answer did not match the verified ground truth

  • record e386fe05 — asked "tests/test_scoring.py"
    expected: (unknown)
    actual: {"steps": [{"step": 1, "tool": "run_tests", "args": {}, "ok": true, "kind": null, "output": "test...

    answer did not match the verified ground truth

Trials24 trials
Per-trial table (24 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00370335380/549full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.010277101111/553full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00688160190/2874full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.01041394997/3044full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00981196492/539full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00889087389/504full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "{"steps": [{"step": 1, "tool": "read_file", "args": {"pat..."repo_fix-0.01007898912/622full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "read_file", "args": {"pat..."unverifiedrepo_fix-0.01133093934/6456full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.00733169408/1301full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.01094990173/6440full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.00837176271/2481full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.00942574001/6749full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.014491143334/524full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "read_file", "args": {"pat..."unverifiedrepo_fix-0.025926248966/3432full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.00922086174/2007full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "read_file", "args": {"pat..."unverifiedrepo_fix-0.00589550071/2959full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.01148995496/6464full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.00453643420/646full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "{"steps": [{"step": 1, "tool": "read_file", "args": {"pat..."repo_fix-0.020302179317/7900full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.010327102031/413full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."unverifiedrepo_fix-0.012073101610/6373full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "{"steps": [{"step": 1, "tool": "read_file", "args": {"pat..."repo_fix-0.00620158761/1082full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "{"steps": [{"step": 1, "tool": "read_file", "args": {"pat..."repo_fix-0.00882175677/4178full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "{"steps": [{"step": 1, "tool": "run_tests", "args": {}, "..."repo_fix-0.019183135953/18624full transcript →