Data to …… trials

Run camp-interactive-fault-gemma4qat-002

44% PASS

What this run revealed, then every individual trial as evidence.

watch live →compare across models →
55
Trials
44%
24/55 scored
over trials
Pass rate
31
56%
Failures
$0.0565
Run cost

Run camp-interactive-fault-gemma4qat-002: 44% pass rate on interactive queries.

Of 55 scored trials (interactive query kind), the model passed 44% and failed 56%.

31 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Sharp boundary detected on: final_distance (high-end pass rate 0%).

Notable failures

  • record b698cd74 — asked "You may make at most 19 tool calls."
    expected: (unknown)
    actual: [{"step": 1, "tool": "advance_0", "args": {"service": "svc_0"}, "ok": true, "error_kind": null, "...

    answer did not match the verified ground truth

  • record f68e0117 — asked "You may make at most 19 tool calls."
    expected: (unknown)
    actual: [{"step": 1, "tool": "advance_0", "args": {"service": "svc_0"}, "ok": true, "error_kind": null, "...

    answer did not match the verified ground truth

  • record 37c84961 — asked "You may make at most 19 tool calls."
    expected: (unknown)
    actual: [{"step": 1, "tool": "advance_0", "args": {"service": "svc_0"}, "ok": true, "error_kind": null, "...

    answer did not match the verified ground truth

Trials55 trials
Per-trial table (55 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0006495400/362full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0006495400/362full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0004673417/418full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0004673417/418full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0006495400/362full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0005864435/474full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0006524479/681full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0006664481/728full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0009827436/796full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.00146710864/1269full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0009758412/446full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0008345594/916full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0012549397/1048full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.00169013405/1166full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0007805544/753full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0008066328/578full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0008576486/695full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0008826531/764full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0008816296/839full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007986310/557full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007115655/484full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.00339033006/299full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007635750/626full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.00180414060/1328full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0012639599/1010full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "set_flag_0", "args": {"flag": "flag..."unverifiedinteractive-0.0007145647/497full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007195641/517full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007715789/640full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0008075805/754full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0012669580/1025full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007015601/470full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007715791/640full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007085625/485full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0013839918/1304full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.00223518400/1317full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0006875564/435full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.00336132736/293full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.00215120370/381full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007705796/634full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007725798/639full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007105659/480full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007725799/640full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0008145819/774full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0008035815/738full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.00285726607/654full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0004673417/418full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0006505412/362full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0004673417/418full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0004673417/418full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0006495400/362full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007835821/670full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007555672/627full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007075625/481full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.00215320398/376full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0007125665/485full transcript →