Data to …… trials

Run camp-interactive-lookalike-gemma4qat-001

97% PASS

What this run revealed, then every individual trial as evidence.

watch live →compare across models →
55
19 errors
Trials
97%
35/36 scored
over trials
Pass rate
1
2%
Failures
$0.1240
Run cost

Run camp-interactive-lookalike-gemma4qat-001: passed essentially everything (97%) -- this test did not challenge the model.

Of 36 scored trials (interactive query kind), the model passed 97%. That's saturation: the difficulty range probed here isn't finding this model's limits. Consider a harder preset (breaking_point or adversarial_belief) or a wider dimension range.

19 additional trial(s) ended in ERROR (infra failures, cost cap, or cancellation) and are excluded from the pass rate above.

1 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Sharp boundary detected on: final_distance (high-end pass rate 0%).

Notable failures

  • record ead7d15a — asked "You may make at most 42 tool calls."
    expected: (unknown)
    actual: []

    answer did not match the verified ground truth

Trials55 trials
Per-trial table (55 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
gemma-4-31b-it-qat-w4a16-ct✕ERRORhttp 400: b'{"error":{"message":"\\"auto\\" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set","type":"BadRequestError","param":null,"code":400}}'interactive-0.0000000/0full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORhttp 400: b'{"error":{"message":"\\"auto\\" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set","type":"BadRequestError","param":null,"code":400}}'interactive-0.0000000/0full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORhttp 400: b'{"error":{"message":"\\"auto\\" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set","type":"BadRequestError","param":null,"code":400}}'interactive-0.0000000/0full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORhttp 400: b'{"error":{"message":"\\"auto\\" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set","type":"BadRequestError","param":null,"code":400}}'interactive-0.0000000/0full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORhttp 400: b'{"error":{"message":"\\"auto\\" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set","type":"BadRequestError","param":null,"code":400}}'interactive-0.0000000/0full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORtimeoutinteractive-0.0000000/0full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORtimeoutinteractive-0.0000000/0full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORtimeoutinteractive-0.0000000/0full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORtimeoutinteractive-0.0000000/0full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_o0", "args": {"service": "s..."interactive-0.00118711422/148full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "adva_nce_0", "args": {"service": "s..."interactive-0.0002011497/172full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "adva-nce_0", "args": {"service": "s..."interactive-0.0002521500/340full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "adva-nce_0", "args": {"service": "s..."interactive-0.0002511494/340full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "adva_nce_0", "args": {"service": "s..."interactive-0.0002511493/339full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_o1", "args": {"service": "s..."interactive-0.00128611457/469full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "adva_nce_0", "args": {"service": "s..."interactive-0.0002982237/246full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "adva_nce_0", "args": {"service": "s..."interactive-0.00281727389/260full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_o0", "args": {"service": "s..."interactive-0.0003932234/564full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "adva_nce_0", "args": {"service": "s..."interactive-0.0003942239/566full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "adva_nce_0", "args": {"service": "s..."interactive-0.00282227445/259full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "adva_nce_0", "args": {"service": "s..."interactive-0.00466645645/337full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "adva-nce_0", "args": {"service": "s..."interactive-0.00467245697/342full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "adva-nce_0", "args": {"service": "s..."interactive-0.00467045683/339full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "adva_nce_0", "args": {"service": "s..."interactive-0.00465245515/336full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "adva-nce_0", "args": {"service": "s..."interactive-0.00464745463/337full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "adva-nce_0", "args": {"service": "s..."interactive-0.0005334018/437full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "adva_nce_0", "args": {"service": "s..."interactive-0.0005334016/438full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_o0", "args": {"service": "s..."interactive-0.0006894010/960full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "adva_nce_0", "args": {"service": "s..."interactive-0.0006924018/966full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "adva-nce_0", "args": {"service": "s..."interactive-0.0006914016/964full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "[]"unverifiedinteractive-0.0005065006/18full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "adva_nce_0", "args": {"service": "s..."interactive-0.0006565016/514full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "adva-nce_0", "args": {"service": "s..."interactive-0.0006565016/514full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "adva-nce_0", "args": {"service": "s..."interactive-0.0006565011/517full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "adva_nce_0", "args": {"service": "s..."interactive-0.011773116137/530full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "adva_nce_0", "args": {"service": "s..."interactive-0.018131179361/648full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "adva-nce_0", "args": {"service": "s..."interactive-0.018143179498/644full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_o0", "args": {"service": "s..."interactive-0.0010706531/1391full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_o0", "args": {"service": "s..."interactive-0.0010686527/1383full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "advance_o0", "args": {"service": "s..."interactive-0.0010716533/1393full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "adva_nce_0", "args": {"service": "s..."interactive-0.0010488277/733full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORtimeoutinteractive-0.0000000/0full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORtimeoutinteractive-0.0000000/0full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORtimeoutinteractive-0.0000000/0full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "adva_nce_0", "args": {"service": "s..."interactive-0.0013158274/1624full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORtimeoutinteractive-0.0000000/0full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORtimeoutinteractive-0.0000000/0full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORtimeoutinteractive-0.0000000/0full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORtimeoutinteractive-0.0000000/0full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "adva_nce_0", "args": {"service": "s..."interactive-0.0011629239/794full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "adva-nce_0", "args": {"service": "s..."interactive-0.028804285702/780full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORtimeoutinteractive-0.0000000/0full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORtimeoutinteractive-0.0000000/0full transcript →
gemma-4-31b-it-qat-w4a16-ct✕ERRORtimeoutinteractive-0.0000000/0full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "[{"step": 1, "tool": "adva_nce_0", "args": {"service": "s..."interactive-0.0013828752/1690full transcript →