Data to …… trials

Run camp-interactive-gemma4qat-002

47% PASS

What this run revealed, then every individual trial as evidence.

watch live →compare across models →
55
Trials
47%
26/55 scored
over trials
Pass rate
29
53%
Failures
$0.0436
Run cost

Run camp-interactive-gemma4qat-002: 47% pass rate on interactive queries.

Of 55 scored trials (interactive query kind), the model passed 47% and failed 53%.

29 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Sharp boundary detected on: final_distance (high-end pass rate 0%).

Notable failures

  • record c13d6d4f — asked "You may make at most 18 tool calls."
    expected: (unknown)
    actual: [{"step": 1, "tool": "advance_0", "args": {"service": "svc_0"}, "ok": true, "error_kind": null, "...

    answer did not match the verified ground truth

  • record 3137b7e1 — asked "You may make at most 18 tool calls."
    expected: (unknown)
    actual: [{"step": 1, "tool": "advance_0", "args": {"service": "svc_0"}, "ok": true, "error_kind": null, "...

    answer did not match the verified ground truth

  • record 766854fd — asked "You may make at most 22 tool calls."
    expected: (unknown)
    actual: [{"step": 1, "tool": "advance_0", "args": {"service": "svc_0"}, "ok": true, "error_kind": null, "...

    answer did not match the verified ground truth

Trials55 trials
Per-trial table (55 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0002471488/326full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0002461478/326full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0002461478/326full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0002461484/326full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0002461480/326full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0005924450/490full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0005964461/499full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0006114433/558full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0011319699/537full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0008577023/517full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.00131511614/512full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007575617/652full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007635582/683full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0012029218/934full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0010138504/541full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0008496457/679full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0008776504/756full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0009266546/903full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.00245920369/1408full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag..."unverifiedinteractive-0.00138110476/1112full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.00304229605/273full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007795804/662full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0008245841/799full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag..."interactive-0.00161813243/979full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007105655/481full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007165664/498full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.00194117900/502full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007885796/696full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0008015794/738full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007035621/469full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007105651/483full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007615753/619full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007735794/646full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0008065801/753full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0006955584/455full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007185648/512full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007175656/506full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007935823/701full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007995783/735full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007775795/659full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0002461478/326full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0002461478/326full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0002461482/326full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0002461479/326full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0002461480/326full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0004943588/449full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0008557593/320full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0008577601/324full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0004963596/453full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0004753595/386full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0004363511/283full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0006935714/407full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0006945721/406full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0008577596/323full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0004753604/382full transcript →