Data to …… trials

Run camp-ef-belief-gemma4qat-001/C

90% PASS

What this run revealed, then every individual trial as evidence.

30
Trials
90%
27/30 scored
over trials
Pass rate
3
10%
Failures
$0.0016
Run cost

Run camp-ef-belief-gemma4qat-001/C: 90% pass rate on belief queries.

Of 30 scored trials (belief query kind), the model passed 90% and failed 10%.

3 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Diffuse weakness (below 90% but no outright flip) on: coreference (pass rate 89% at the high end), evidence_span (pass rate 80% at the high end).

In 3 of 3 classifiable belief failures the model reported where the object ACTUALLY was, not where the queried entity BELIEVED it was -- it collapsed their perspective into reality.

Notable failures

  • record 88e752b6 — asked "Where does Quentin believe the blue folder is? Answer with only the location name (or 'unknown' if they have no belief)."
    expected: the kitchen
    actual: south stairwell

    the model reported the object's TRUE final location, not what the queried entity actually believed -- it collapsed belief into reality

  • record d067ef31 — asked "Where does Beatrix believe the dusty wrench is? Answer with only the location name (or 'unknown' if they have no belief)."
    expected: the basement
    actual: library

    the model reported the object's TRUE final location, not what the queried entity actually believed -- it collapsed belief into reality

  • record c7aa9710 — asked "Where does Omar believe the worn box is? Answer with only the location name (or 'unknown' if they have no belief)."
    expected: the gym
    actual: west wing

    the model reported the object's TRUE final location, not what the queried entity actually believed -- it collapsed belief into reality

Trials30 trials
Per-trial table (30 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "supply closet"belief-0.000051503/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "north stairwell"belief-0.000058567/4full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "kitchen"belief-0.000054536/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "storage room"belief-0.000060586/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "boiler room"belief-0.000054529/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "archive room"belief-0.000058569/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "workshop"belief-0.000058572/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "workshop"belief-0.000055543/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "cafeteria"belief-0.000053514/4full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "photo studio"belief-0.000053519/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "south stairwell"unverifiedbelief-0.000052505/4full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "print room"belief-0.000049484/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "library"unverifiedbelief-0.000053522/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "boiler room"belief-0.000053520/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "archive room"belief-0.000060592/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "west wing"unverifiedbelief-0.000050490/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "gym"belief-0.000061599/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "front office"belief-0.000058567/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "south stairwell"belief-0.000054529/4full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "supply closet"belief-0.000046450/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "conference room"belief-0.000058573/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "print room"belief-0.000053522/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "supply closet"belief-0.000054526/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "cafeteria"belief-0.000048466/4full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "observation deck"belief-0.000054529/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "photo studio"belief-0.000048470/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "supply closet"belief-0.000053524/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "greenhouse"belief-0.000056550/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "garage"belief-0.000055547/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "security office"belief-0.000055538/3full transcript →