Data to …… trials

Run camp-ef-belief-gemma4qat-001/B

37% PASS

What this run revealed, then every individual trial as evidence.

30
Trials
37%
11/30 scored
over trials
Pass rate
19
63%
Failures
$0.0010
Run cost

Run camp-ef-belief-gemma4qat-001/B: 37% pass rate on belief queries.

Of 30 scored trials (belief query kind), the model passed 37% and failed 63%.

19 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

In 16 of 19 classifiable belief failures the model reported where the object ACTUALLY was, not where the queried entity BELIEVED it was -- it collapsed their perspective into reality.

Notable failures

  • record 02cf5b79 — asked "Where does Uma believe the large basket is? Answer with only the location name (or 'unknown' if they have no belief)."
    expected: the supply closet
    actual: west wing

    the model reported the object's TRUE final location, not what the queried entity actually believed -- it collapsed belief into reality

  • record 5cbcc354 — asked "Where does Victor believe the worn folder is? Answer with only the location name (or 'unknown' if they have no belief)."
    expected: the boiler room
    actual: utility closet

    the model reported the object's TRUE final location, not what the queried entity actually believed -- it collapsed belief into reality

  • record be2c3fa3 — asked "Where does Petra believe the heavy tablet is? Answer with only the location name (or 'unknown' if they have no belief)."
    expected: the archive room
    actual: security office

    the model reported the object's TRUE final location, not what the queried entity actually believed -- it collapsed belief into reality

Trials30 trials
Per-trial table (30 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "west wing"unverifiedbelief-0.000032315/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "north stairwell"belief-0.000036343/4full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "kitchen"belief-0.000034329/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "storage room"belief-0.000036353/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "utility closet"unverifiedbelief-0.000034328/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "security office"unverifiedbelief-0.000035344/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "courtyard"unverifiedbelief-0.000035345/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "workshop"belief-0.000034334/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "cafeteria"belief-0.000033319/4full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "photo studio"belief-0.000033317/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "south stairwell"unverifiedbelief-0.000033313/4full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "workshop"unverifiedbelief-0.000031302/2full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "supply closet"unverifiedbelief-0.000033321/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "break room"unverifiedbelief-0.000033319/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "archive room"belief-0.000036356/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "west wing"unverifiedbelief-0.000032309/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "print room"unverifiedbelief-0.000037358/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "front office"belief-0.000035344/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "archive room"unverifiedbelief-0.000034327/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "supply closet"belief-0.000030287/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "lobby"unverifiedbelief-0.000035347/2full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "meditation room"unverifiedbelief-0.000033322/4full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "supply closet"belief-0.000033322/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "server room"unverifiedbelief-0.000030295/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "break room"unverifiedbelief-0.000033325/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "garage"unverifiedbelief-0.000030294/2full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "archive room"unverifiedbelief-0.000033322/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "wine cellar"unverifiedbelief-0.000034336/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "north stairwell"unverifiedbelief-0.000035334/4full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "security office"belief-0.000034330/3full transcript →