Data to …… trials

Run camp-ef-belief-gemma4qat-001/D

50% PASS

What this run revealed, then every individual trial as evidence.

30
Trials
50%
15/30 scored
over trials
Pass rate
15
50%
Failures
$0.0011
Run cost

Run camp-ef-belief-gemma4qat-001/D: 50% pass rate on belief queries.

Of 30 scored trials (belief query kind), the model passed 50% and failed 50%.

15 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

In 14 of 15 classifiable belief failures the model reported where the object ACTUALLY was, not where the queried entity BELIEVED it was -- it collapsed their perspective into reality.

Notable failures

  • record 00bd25e4 — asked "Where does Uma believe the large basket is? Answer with only the location name (or 'unknown' if they have no belief)."
    expected: the supply closet
    actual: west wing

    the model reported the object's TRUE final location, not what the queried entity actually believed -- it collapsed belief into reality

  • record e5dfdbec — asked "Where does Victor believe the worn folder is? Answer with only the location name (or 'unknown' if they have no belief)."
    expected: the boiler room
    actual: utility closet

    the model reported the object's TRUE final location, not what the queried entity actually believed -- it collapsed belief into reality

  • record e3a04310 — asked "Where does Piers believe the dusty wrench is? Answer with only the location name (or 'unknown' if they have no belief)."
    expected: the workshop
    actual: photo studio

    the model reported the object's TRUE final location, not what the queried entity actually believed -- it collapsed belief into reality

Trials30 trials
Per-trial table (30 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "west wing"unverifiedbelief-0.000035345/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "north stairwell"belief-0.000039379/4full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "kitchen"belief-0.000037362/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "storage room"belief-0.000040392/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "utility closet"unverifiedbelief-0.000037361/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "archive room"belief-0.000039383/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "workshop"belief-0.000039384/2full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "photo studio"unverifiedbelief-0.000038370/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "cafeteria"belief-0.000036352/4full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "photo studio"belief-0.000036347/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "the kitchen"belief-0.000036346/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "print room"belief-0.000034332/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "basement"belief-0.000036354/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "break room"unverifiedbelief-0.000036352/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "rooftop terrace"unverifiedbelief-0.000041395/5full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "west wing"unverifiedbelief-0.000035339/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "the gym"belief-0.000041400/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "front office"belief-0.000039383/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "archive room"unverifiedbelief-0.000037360/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "supply closet"belief-0.000032314/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "lobby"unverifiedbelief-0.000039383/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "print room"belief-0.000036355/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "supply closet"belief-0.000036355/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "server room"unverifiedbelief-0.000033322/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "break room"unverifiedbelief-0.000037361/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "garage"unverifiedbelief-0.000033324/2full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "archive room"unverifiedbelief-0.000036355/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "wine cellar"unverifiedbelief-0.000038372/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "north stairwell"unverifiedbelief-0.000038370/4full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "archive room"unverifiedbelief-0.000038366/3full transcript →