Data to …… trials

Run camp-ticket-mechanism-gemma4qat-001/ablate

27% PASS

What this run revealed, then every individual trial as evidence.

watch live →compare across models →
48
Trials
27%
13/48 scored
over trials
Pass rate
35
73%
Failures
$0.0419
Run cost

Run camp-ticket-mechanism-gemma4qat-001/ablate: 27% pass rate on interactive queries.

Of 48 scored trials (interactive query kind), the model passed 27% and failed 73%.

35 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record 8c52814a — asked "You may make at most 30 tool calls."
    expected: (unknown)
    actual: [{"step": 1, "tool": "advance_0", "args": {"service": "svc_0"}, "ok": true, "error_kind": null, "...

    answer did not match the verified ground truth

  • record adcdeec0 — asked "You may make at most 30 tool calls."
    expected: (unknown)
    actual: [{"step": 1, "tool": "advance_0", "args": {"service": "svc_0"}, "ok": true, "error_kind": null, "...

    answer did not match the verified ground truth

  • record 3a886f0a — asked "You may make at most 30 tool calls."
    expected: (unknown)
    actual: [{"step": 1, "tool": "advance_0", "args": {"service": "svc_0"}, "ok": true, "error_kind": null, "...

    answer did not match the verified ground truth

Trials48 trials
Per-trial table (48 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0010087587/831full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0010087587/831full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0010087587/831full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0010087587/831full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0010087587/831full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0010087587/831full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0011167761/1132full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0011167761/1132full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0011167761/1132full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0011167761/1132full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0011167761/1132full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0011167761/1132full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0010127625/833full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0010127625/833full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0010127625/833full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0010127625/833full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0010127625/833full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0010127625/833full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[]"unverifiedinteractive-0.0003163114/15full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[]"unverifiedinteractive-0.0003163114/15full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[]"unverifiedinteractive-0.0003163114/15full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[]"unverifiedinteractive-0.0003163114/15full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[]"unverifiedinteractive-0.0003163114/15full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[]"unverifiedinteractive-0.0003163114/15full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0010107614/830full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0010107614/830full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0010107614/830full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0010107614/830full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0010107614/830full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0010107614/830full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0011717877/1277full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0011717877/1277full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0011717877/1277full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0011717877/1277full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0011717877/1277full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0011717877/1277full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."interactive-0.0011367648/1239full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0010147650/830full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0010147650/830full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0010147650/830full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0010147650/830full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[{"step": 1, "tool": "advance_0", "args": {"service": "sv..."unverifiedinteractive-0.0010147650/830full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[]"unverifiedinteractive-0.0003153094/17full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[]"unverifiedinteractive-0.0003153094/17full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[]"unverifiedinteractive-0.0003153094/17full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[]"unverifiedinteractive-0.0003153094/17full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[]"unverifiedinteractive-0.0003153094/17full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[]"unverifiedinteractive-0.0003153094/17full transcript →