Data to …… trials

Run camp-trivial-belief-gemma4qat-002

98% PASS

What this run revealed, then every individual trial as evidence.

40
Trials
98%
39/40 scored
over trials
Pass rate
1
2%
Failures
$0.0015
Run cost

Run camp-trivial-belief-gemma4qat-002: passed essentially everything (98%) -- this test did not challenge the model.

Of 40 scored trials (belief query kind), the model passed 98%. That's saturation: the difficulty range probed here isn't finding this model's limits. Consider a harder preset (breaking_point or adversarial_belief) or a wider dimension range.

1 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

1 belief failure(s) were classified against known shortcuts, but none of them dominate -- the failures don't share one obvious wrong-answer pattern.

Notable failures

  • record 6ec8e0ea — asked "Where does Nadia believe the dusty wrench is? Answer with only the location name (or 'unknown' if they have no belief)."
    expected: the basement
    actual: gym

    the model's answer matches no known shortcut -- an unclassified miss

Trials40 trials
Per-trial table (40 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "archive room"belief-0.000015145/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "supply closet"belief-0.000015145/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "conference room"belief-0.000016148/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "south stairwell"belief-0.000016152/4full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "archive room"belief-0.000014134/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "lobby"belief-0.000022210/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "mail room"belief-0.000022211/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "rooftop terrace"belief-0.000022202/5full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "greenhouse"belief-0.000022213/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "observation deck"belief-0.000022209/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "gym"belief-0.000028276/2full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "gym"unverifiedbelief-0.000027259/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "storage room"belief-0.000027259/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "greenhouse"belief-0.000027265/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "basement"belief-0.000028267/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "supply closet"belief-0.000036354/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "reception area"belief-0.000036348/4full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "server room"belief-0.000035342/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "atrium"belief-0.000034335/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "basement"belief-0.000036347/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "server room"belief-0.000040392/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "gym"belief-0.000042411/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "loading dock"belief-0.000041401/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "front office"belief-0.000039384/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "observation deck"belief-0.000042411/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "lobby"belief-0.000050491/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "workshop"belief-0.000050495/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "supply closet"belief-0.000050490/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "conference room"belief-0.000048471/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "utility closet"belief-0.000048475/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "kitchen"belief-0.000056550/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "archive room"belief-0.000054534/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "security office"belief-0.000054527/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "north stairwell"belief-0.000055538/4full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "rooftop terrace"belief-0.000053517/5full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "security office"belief-0.000056552/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "the gym"belief-0.000059578/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "courtyard"belief-0.000055539/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "rooftop terrace"belief-0.000058562/5full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "front office"belief-0.000053523/3full transcript →