Data to …… trials

Run camp-ticket-trap-gemma4qat-001

60% PASS

What this run revealed, then every individual trial as evidence.

watch live →compare across models →
5
Trials
60%
3/5 scored
over trials
Pass rate
2
40%
Failures
$0.0051
Run cost

Run camp-ticket-trap-gemma4qat-001, 1 arms: clean on none; below 97% on ablate.

Arm ablate: 25% pass rate across 48 trials.

(ablate) 36 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record 6dc2cbbb — asked "You may make at most 30 tool calls."
    expected: (unknown)
    actual: [{"step": 1, "tool": "advance_0", "args": {"service": "svc_0"}, "ok": true, "error_kind": null, "...

    answer did not match the verified ground truth

  • record 40bf4b7b — asked "You may make at most 30 tool calls."
    expected: (unknown)
    actual: [{"step": 1, "tool": "advance_0", "args": {"service": "svc_0"}, "ok": true, "error_kind": null, "...

    answer did not match the verified ground truth

Trials5 trials
Per-trial table (5 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0009877466/800full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0009947483/818full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0009987489/831full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0010527538/995full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0010407522/958full transcript →