Data to …… trials

Run camp-ef-belief-gemma4qat-001

69% PASS

What this run revealed, then every individual trial as evidence.

120
Trials
69%
83/120 scored
over trials
Pass rate
37
31%
Failures
$0.0048
Run cost

Run camp-ef-belief-gemma4qat-001, 4 arms: clean on A; below 97% on B, C, D.

Arm A passes cleanly (100%, 30 trials).

Arm B: 37% pass rate across 30 trials.

(B) 19 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Arm C: 90% pass rate across 30 trials.

(C) 3 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Arm D: 50% pass rate across 30 trials.

(D) 15 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record 02cf5b79 — asked "Where does Uma believe the large basket is? Answer with only the location name (or 'unknown' if they have no belief)."
    expected: the supply closet
    actual: west wing

    the model reported the object's TRUE final location, not what the queried entity actually believed -- it collapsed belief into reality

  • record 5cbcc354 — asked "Where does Victor believe the worn folder is? Answer with only the location name (or 'unknown' if they have no belief)."
    expected: the boiler room
    actual: utility closet

    the model reported the object's TRUE final location, not what the queried entity actually believed -- it collapsed belief into reality

  • record 88e752b6 — asked "Where does Quentin believe the blue folder is? Answer with only the location name (or 'unknown' if they have no belief)."
    expected: the kitchen
    actual: south stairwell

    the model reported the object's TRUE final location, not what the queried entity actually believed -- it collapsed belief into reality

Trials120 trials
Per-trial table (120 trials -- prompt, reasoning, verdict, answer key)
modelstageverdictkinddifficultycost ($)tokens in/out
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "west wing"belief-0.000034328/3full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "lobby"belief-0.000036357/2full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "workshop"belief-0.000035342/2full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "library"belief-0.000037367/2full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "utility closet"belief-0.000035343/3full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "security office"belief-0.000037359/3full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "courtyard"belief-0.000037360/3full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "photo studio"belief-0.000036349/3full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "boiler room"belief-0.000034332/3full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "storage room"belief-0.000034332/3full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "south stairwell"belief-0.000034328/4full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "workshop"belief-0.000032315/2full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "library"belief-0.000034336/2full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "break room"belief-0.000034334/3full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "rooftop terrace"belief-0.000039370/5full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "west wing"belief-0.000033324/3full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "north stairwell"belief-0.000039373/4full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "supply closet"belief-0.000037359/3full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "archive room"belief-0.000035340/3full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "workshop"belief-0.000031300/2full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "observation deck"belief-0.000037362/3full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "meditation room"belief-0.000035337/4full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "kitchen"belief-0.000034337/2full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "server room"belief-0.000032310/3full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "break room"belief-0.000035338/3full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "garage"belief-0.000031309/2full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "archive room"belief-0.000035337/3full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "wine cellar"belief-0.000036351/3full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "north stairwell"belief-0.000036349/4full transcript →
gemma-4-31b-it-qat-w4a16-ctA✓PASSanswered "archive room"belief-0.000035345/3full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "west wing"unverifiedbelief-0.000032315/3full transcript →
gemma-4-31b-it-qat-w4a16-ctB✓PASSanswered "north stairwell"belief-0.000036343/4full transcript →
gemma-4-31b-it-qat-w4a16-ctB✓PASSanswered "kitchen"belief-0.000034329/2full transcript →
gemma-4-31b-it-qat-w4a16-ctB✓PASSanswered "storage room"belief-0.000036353/3full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "utility closet"unverifiedbelief-0.000034328/3full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "security office"unverifiedbelief-0.000035344/3full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "courtyard"unverifiedbelief-0.000035345/3full transcript →
gemma-4-31b-it-qat-w4a16-ctB✓PASSanswered "workshop"belief-0.000034334/2full transcript →
gemma-4-31b-it-qat-w4a16-ctB✓PASSanswered "cafeteria"belief-0.000033319/4full transcript →
gemma-4-31b-it-qat-w4a16-ctB✓PASSanswered "photo studio"belief-0.000033317/3full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "south stairwell"unverifiedbelief-0.000033313/4full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "workshop"unverifiedbelief-0.000031302/2full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "supply closet"unverifiedbelief-0.000033321/3full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "break room"unverifiedbelief-0.000033319/3full transcript →
gemma-4-31b-it-qat-w4a16-ctB✓PASSanswered "archive room"belief-0.000036356/3full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "west wing"unverifiedbelief-0.000032309/3full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "print room"unverifiedbelief-0.000037358/3full transcript →
gemma-4-31b-it-qat-w4a16-ctB✓PASSanswered "front office"belief-0.000035344/3full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "archive room"unverifiedbelief-0.000034327/3full transcript →
gemma-4-31b-it-qat-w4a16-ctB✓PASSanswered "supply closet"belief-0.000030287/3full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "lobby"unverifiedbelief-0.000035347/2full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "meditation room"unverifiedbelief-0.000033322/4full transcript →
gemma-4-31b-it-qat-w4a16-ctB✓PASSanswered "supply closet"belief-0.000033322/3full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "server room"unverifiedbelief-0.000030295/3full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "break room"unverifiedbelief-0.000033325/3full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "garage"unverifiedbelief-0.000030294/2full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "archive room"unverifiedbelief-0.000033322/3full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "wine cellar"unverifiedbelief-0.000034336/3full transcript →
gemma-4-31b-it-qat-w4a16-ctB?FAILanswered "north stairwell"unverifiedbelief-0.000035334/4full transcript →
gemma-4-31b-it-qat-w4a16-ctB✓PASSanswered "security office"belief-0.000034330/3full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "supply closet"belief-0.000051503/3full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "north stairwell"belief-0.000058567/4full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "kitchen"belief-0.000054536/2full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "storage room"belief-0.000060586/3full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "boiler room"belief-0.000054529/3full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "archive room"belief-0.000058569/3full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "workshop"belief-0.000058572/2full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "workshop"belief-0.000055543/2full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "cafeteria"belief-0.000053514/4full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "photo studio"belief-0.000053519/3full transcript →
gemma-4-31b-it-qat-w4a16-ctC?FAILanswered "south stairwell"unverifiedbelief-0.000052505/4full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "print room"belief-0.000049484/3full transcript →
gemma-4-31b-it-qat-w4a16-ctC?FAILanswered "library"unverifiedbelief-0.000053522/2full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "boiler room"belief-0.000053520/3full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "archive room"belief-0.000060592/3full transcript →
gemma-4-31b-it-qat-w4a16-ctC?FAILanswered "west wing"unverifiedbelief-0.000050490/3full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "gym"belief-0.000061599/2full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "front office"belief-0.000058567/3full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "south stairwell"belief-0.000054529/4full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "supply closet"belief-0.000046450/3full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "conference room"belief-0.000058573/3full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "print room"belief-0.000053522/3full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "supply closet"belief-0.000054526/3full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "cafeteria"belief-0.000048466/4full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "observation deck"belief-0.000054529/3full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "photo studio"belief-0.000048470/3full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "supply closet"belief-0.000053524/3full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "greenhouse"belief-0.000056550/3full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "garage"belief-0.000055547/2full transcript →
gemma-4-31b-it-qat-w4a16-ctC✓PASSanswered "security office"belief-0.000055538/3full transcript →
gemma-4-31b-it-qat-w4a16-ctD?FAILanswered "west wing"unverifiedbelief-0.000035345/3full transcript →
gemma-4-31b-it-qat-w4a16-ctD✓PASSanswered "north stairwell"belief-0.000039379/4full transcript →
gemma-4-31b-it-qat-w4a16-ctD✓PASSanswered "kitchen"belief-0.000037362/2full transcript →
gemma-4-31b-it-qat-w4a16-ctD✓PASSanswered "storage room"belief-0.000040392/3full transcript →
gemma-4-31b-it-qat-w4a16-ctD?FAILanswered "utility closet"unverifiedbelief-0.000037361/3full transcript →
gemma-4-31b-it-qat-w4a16-ctD✓PASSanswered "archive room"belief-0.000039383/3full transcript →
gemma-4-31b-it-qat-w4a16-ctD✓PASSanswered "workshop"belief-0.000039384/2full transcript →
gemma-4-31b-it-qat-w4a16-ctD?FAILanswered "photo studio"unverifiedbelief-0.000038370/3full transcript →
gemma-4-31b-it-qat-w4a16-ctD✓PASSanswered "cafeteria"belief-0.000036352/4full transcript →
gemma-4-31b-it-qat-w4a16-ctD✓PASSanswered "photo studio"belief-0.000036347/3full transcript →
gemma-4-31b-it-qat-w4a16-ctD✓PASSanswered "the kitchen"belief-0.000036346/3full transcript →
gemma-4-31b-it-qat-w4a16-ctD✓PASSanswered "print room"belief-0.000034332/3full transcript →
gemma-4-31b-it-qat-w4a16-ctD✓PASSanswered "basement"belief-0.000036354/3full transcript →
gemma-4-31b-it-qat-w4a16-ctD?FAILanswered "break room"unverifiedbelief-0.000036352/3full transcript →
gemma-4-31b-it-qat-w4a16-ctD?FAILanswered "rooftop terrace"unverifiedbelief-0.000041395/5full transcript →
gemma-4-31b-it-qat-w4a16-ctD?FAILanswered "west wing"unverifiedbelief-0.000035339/3full transcript →
gemma-4-31b-it-qat-w4a16-ctD✓PASSanswered "the gym"belief-0.000041400/3full transcript →
gemma-4-31b-it-qat-w4a16-ctD✓PASSanswered "front office"belief-0.000039383/3full transcript →
gemma-4-31b-it-qat-w4a16-ctD?FAILanswered "archive room"unverifiedbelief-0.000037360/3full transcript →
gemma-4-31b-it-qat-w4a16-ctD✓PASSanswered "supply closet"belief-0.000032314/3full transcript →
gemma-4-31b-it-qat-w4a16-ctD?FAILanswered "lobby"unverifiedbelief-0.000039383/2full transcript →
gemma-4-31b-it-qat-w4a16-ctD✓PASSanswered "print room"belief-0.000036355/3full transcript →
gemma-4-31b-it-qat-w4a16-ctD✓PASSanswered "supply closet"belief-0.000036355/3full transcript →
gemma-4-31b-it-qat-w4a16-ctD?FAILanswered "server room"unverifiedbelief-0.000033322/3full transcript →
gemma-4-31b-it-qat-w4a16-ctD?FAILanswered "break room"unverifiedbelief-0.000037361/3full transcript →
gemma-4-31b-it-qat-w4a16-ctD?FAILanswered "garage"unverifiedbelief-0.000033324/2full transcript →
gemma-4-31b-it-qat-w4a16-ctD?FAILanswered "archive room"unverifiedbelief-0.000036355/3full transcript →
gemma-4-31b-it-qat-w4a16-ctD?FAILanswered "wine cellar"unverifiedbelief-0.000038372/3full transcript →
gemma-4-31b-it-qat-w4a16-ctD?FAILanswered "north stairwell"unverifiedbelief-0.000038370/4full transcript →
gemma-4-31b-it-qat-w4a16-ctD?FAILanswered "archive room"unverifiedbelief-0.000038366/3full transcript →