Data to …… trials

Run camp-adversarial_belief-gemma4qat-001

58% PASS

What this run revealed, then every individual trial as evidence.

105
Trials
58%
61/105 scored
over trials
Pass rate
44
42%
Failures
$0.0026
Run cost

Run camp-adversarial_belief-gemma4qat-001: 58% pass rate on belief queries.

Of 105 scored trials (belief query kind), the model passed 58% and failed 42%.

44 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

In 44 of 44 classifiable belief failures the model reported where the object ACTUALLY was, not where the queried entity BELIEVED it was -- it collapsed their perspective into reality.

Notable failures

  • record bf805e40 — asked "Where does Opal believe the faded wrench is? Answer with only the location name (or 'unknown' if they have no belief)."
    expected: the cafeteria
    actual: kitchen

    the model reported the object's TRUE final location, not what the queried entity actually believed -- it collapsed belief into reality

  • record 76ea30fa — asked "Where does Opal believe the blue folder is? Answer with only the location name (or 'unknown' if they have no belief)."
    expected: the north stairwell
    actual: lobby

    the model reported the object's TRUE final location, not what the queried entity actually believed -- it collapsed belief into reality

  • record afd539f9 — asked "Where does Sofia believe the small wrench is? Answer with only the location name (or 'unknown' if they have no belief)."
    expected: the observation deck
    actual: storage room

    the model reported the object's TRUE final location, not what the queried entity actually believed -- it collapsed belief into reality

Trials105 trials
Per-trial table (105 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "north stairwell"belief-0.000017159/4full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "mail room"belief-0.000016152/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "utility closet"belief-0.000016150/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "supply closet"belief-0.000018169/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "kitchen" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone)unverifiedbelief-0.000017165/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "north stairwell"belief-0.000022207/4full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "lobby" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone, most_frequent_location, query_agent_own_last_move)unverifiedbelief-0.000022218/2full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "storage room" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone, most_frequent_location, query_agent_own_last_move)unverifiedbelief-0.000023220/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "wine cellar" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone, most_frequent_location, query_agent_own_last_move)unverifiedbelief-0.000022213/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "library" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone, most_frequent_location, query_agent_own_last_move)unverifiedbelief-0.000021201/2full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "observation deck" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone, most_frequent_location, query_agent_own_last_move)unverifiedbelief-0.000027263/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "south stairwell" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone, most_frequent_location, query_agent_own_last_move)unverifiedbelief-0.000029274/4full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "supply closet" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone, most_frequent_location, query_agent_own_last_move)unverifiedbelief-0.000029276/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "library"belief-0.000030291/2full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "south stairwell" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone, most_frequent_location, query_agent_own_last_move)unverifiedbelief-0.000029281/4full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "greenhouse" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone, most_frequent_location, query_agent_own_last_move)unverifiedbelief-0.000035339/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "break room" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone, most_frequent_location, query_agent_own_last_move)unverifiedbelief-0.000034331/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "meditation room" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone, most_frequent_location, query_agent_own_last_move)unverifiedbelief-0.000032307/4full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "reception area" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone, most_frequent_location, query_agent_own_last_move)unverifiedbelief-0.000033316/4full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "break room" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone, most_frequent_location, query_agent_own_last_move)unverifiedbelief-0.000042414/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "workshop" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone, most_frequent_location, query_agent_own_last_move)unverifiedbelief-0.000044431/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "east wing"belief-0.000036352/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "mail room" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone, most_frequent_location, query_agent_own_last_move)unverifiedbelief-0.000040387/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "library" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone, most_frequent_location, query_agent_own_last_move)unverifiedbelief-0.000038375/2full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "library" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone, most_frequent_location, query_agent_own_last_move)unverifiedbelief-0.000038371/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "mail room"belief-0.000018169/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "observation deck"belief-0.000016151/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "utility closet"belief-0.000016152/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "photo studio"belief-0.000017159/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "front office"belief-0.000018167/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "north stairwell"belief-0.000018169/4full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "basement"belief-0.000017165/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "break room"belief-0.000018168/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "storage room" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone)unverifiedbelief-0.000017163/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "conference room"belief-0.000018172/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "print room"belief-0.000016153/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "basement"belief-0.000016147/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "greenhouse" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone)unverifiedbelief-0.000016151/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "photo studio" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone)unverifiedbelief-0.000019176/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "break room" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone)unverifiedbelief-0.000018169/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "garage"belief-0.000017159/2full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "boiler room" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone)unverifiedbelief-0.000017165/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "library"belief-0.000015148/2full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "photo studio" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone)unverifiedbelief-0.000018166/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "archive room"belief-0.000016152/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "archive room" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone)unverifiedbelief-0.000018172/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "gym"belief-0.000016152/2full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "north stairwell" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone)unverifiedbelief-0.000018172/4full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "archive room" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone)unverifiedbelief-0.000018168/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "rooftop terrace"belief-0.000017153/5full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "photo studio"belief-0.000016153/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "conference room"belief-0.000018170/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "courtyard"belief-0.000018167/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "cafeteria" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone)unverifiedbelief-0.000018164/4full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "west wing" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone)unverifiedbelief-0.000017165/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "atrium"belief-0.000017164/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "mail room"belief-0.000019181/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "atrium"belief-0.000018167/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "storage room" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone)unverifiedbelief-0.000019181/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "storage room"belief-0.000020193/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "loading dock"belief-0.000019184/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "south stairwell"belief-0.000021196/4full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "utility closet"belief-0.000020191/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "utility closet"belief-0.000020191/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "archive room"belief-0.000020190/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "lobby"belief-0.000021204/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "reception area"belief-0.000024224/4full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "east wing"belief-0.000023220/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "storage room"belief-0.000023217/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "photo studio"belief-0.000023219/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "storage room"belief-0.000024234/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "loading dock"belief-0.000025242/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "greenhouse"belief-0.000024235/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "wine cellar"belief-0.000024231/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "meditation room"belief-0.000025238/4full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "wine cellar"belief-0.000018168/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "south stairwell"belief-0.000016152/4full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "reception area" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone)unverifiedbelief-0.000018170/4full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "conference room"belief-0.000018175/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "photo studio"belief-0.000018168/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "west wing" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone)unverifiedbelief-0.000025236/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "mail room"belief-0.000025243/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "north stairwell" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone)unverifiedbelief-0.000024225/4full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "cafeteria"belief-0.000025236/4full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "archive room" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone)unverifiedbelief-0.000023224/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "wine cellar" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone)unverifiedbelief-0.000033324/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "library" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone)unverifiedbelief-0.000033319/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "atrium"belief-0.000033324/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "south stairwell" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone)unverifiedbelief-0.000035341/4full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "basement" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone)unverifiedbelief-0.000041399/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "security office"belief-0.000032306/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "gym" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone)unverifiedbelief-0.000042410/2full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "basement" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone)unverifiedbelief-0.000042413/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "observation deck" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone)unverifiedbelief-0.000042406/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "west wing"belief-0.000043418/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "lobby"belief-0.000051503/2full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "lobby" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone)unverifiedbelief-0.000048472/2full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "storage room" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone)unverifiedbelief-0.000050488/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "utility closet"belief-0.000049485/3full transcript →
gemma-4-31b-it-qat-w4a16-ct?FAILanswered "greenhouse" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone)unverifiedbelief-0.000049478/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "wine cellar"belief-0.000016153/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "kitchen"belief-0.000017165/2full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "wine cellar"belief-0.000016154/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "break room"belief-0.000018168/3full transcript →
gemma-4-31b-it-qat-w4a16-ct✓PASSanswered "greenhouse"belief-0.000016151/3full transcript →