Data to …… trials

Run camp-full_map-gpt-oss-120b-001/belief

97% PASS

What this run revealed, then every individual trial as evidence.

160
1 error
Trials
97%
154/159 scored
over trials
Pass rate
5
5 stable / 0 noise
3%
Failures
$0.0351
Run cost

Run camp-full_map-gpt-oss-120b-001/belief: 97% pass rate on belief queries.

Of 159 scored trials (belief query kind), the model passed 97% and failed 3%.

1 additional trial(s) ended in ERROR (infra failures, cost cap, or cancellation) and are excluded from the pass rate above.

5 failures recorded; all 5 survived verification.

Diffuse weakness (below 90% but no outright flip) on: witness_gap (pass rate 80% at the high end).

In 3 of 5 classifiable belief failures the model reported where the object ACTUALLY was, not where the queried entity BELIEVED it was -- it collapsed their perspective into reality.

Notable failures

  • record 89641036 — asked "Where does Dexter believe the faded box is? Answer with only the location name (or 'unknown' if they have no belief)."
    expected: the archive room
    actual: print room

    the model reported the object's TRUE final location, not what the queried entity actually believed -- it collapsed belief into reality

  • record 52ea26a5 — asked "Where does Elliot believe the dusty wrench is? Answer with only the location name (or 'unknown' if they have no belief)."
    expected: the east wing
    actual: conference room

    the model reported the object's TRUE final location, not what the queried entity actually believed -- it collapsed belief into reality

  • record 04f097b3 — asked "Where does Quinn believe the narrow wrench is? Answer with only the location name (or 'unknown' if they have no belief)."
    expected: the utility closet
    actual: library

    the model reported the object's TRUE final location, not what the queried entity actually believed -- it collapsed belief into reality

Trials160 trials
Per-trial table (160 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
openai/gpt-oss-120b✓PASSanswered "east wing"belief-0.000151205/201full transcript →
openai/gpt-oss-120b✓PASSanswered "lobby"belief-0.000130203/166full transcript →
openai/gpt-oss-120b✓PASSanswered "workshop"belief-0.000162218/215full transcript →
openai/gpt-oss-120b✓PASSanswered "boiler room"belief-0.000139219/177full transcript →
openai/gpt-oss-120b✓PASSanswered "garage"belief-0.000208272/279full transcript →
openai/gpt-oss-120b✕FAILanswered "print room"verified-stablebelief-0.000357267/528full transcript →
openai/gpt-oss-120b✓PASSanswered "supply closet"belief-0.000164206/222full transcript →
openai/gpt-oss-120b✓PASSanswered "kitchen"belief-0.000219276/296full transcript →
openai/gpt-oss-120b✓PASSanswered "basement"belief-0.000213268/288full transcript →
openai/gpt-oss-120b✓PASSanswered "observation deck"belief-0.000165284/204full transcript →
openai/gpt-oss-120b✓PASSanswered "reception area"belief-0.000315328/443full transcript →
openai/gpt-oss-120b✓PASSanswered "break room"belief-0.000208300/271full transcript →
openai/gpt-oss-120b✓PASSanswered "garage"belief-0.000331338/468full transcript →
openai/gpt-oss-120b✓PASSanswered "break room"belief-0.000471329/703full transcript →
openai/gpt-oss-120b✓PASSanswered "library"belief-0.000579363/874full transcript →
openai/gpt-oss-120b✓PASSanswered "north stairwell"belief-0.000313406/420full transcript →
openai/gpt-oss-120b✓PASSanswered "security office"belief-0.000439402/631full transcript →
openai/gpt-oss-120b✓PASSanswered "rooftop terrace"belief-0.000411392/587full transcript →
openai/gpt-oss-120b✓PASSanswered "boiler room"belief-0.000318313/451full transcript →
openai/gpt-oss-120b✓PASSanswered "cafeteria"belief-0.000508418/743full transcript →
openai/gpt-oss-120b✓PASSanswered "reception area"belief-0.000430385/620full transcript →
openai/gpt-oss-120b✓PASSanswered "garage"belief-0.000378401/529full transcript →
openai/gpt-oss-120b✓PASSanswered "west wing"belief-0.000626400/943full transcript →
openai/gpt-oss-120b✓PASSanswered "basement"belief-0.000463413/668full transcript →
openai/gpt-oss-120b✓PASSanswered "server room"belief-0.000417406/594full transcript →
openai/gpt-oss-120b✓PASSanswered "break room"belief-0.000122222/148full transcript →
openai/gpt-oss-120b✓PASSanswered "photo studio"belief-0.000117219/141full transcript →
openai/gpt-oss-120b✓PASSanswered "security office"belief-0.000163201/222full transcript →
openai/gpt-oss-120b✓PASSanswered "loading dock"belief-0.000145226/186full transcript →
openai/gpt-oss-120b✓PASSanswered "north stairwell"belief-0.000425203/657full transcript →
openai/gpt-oss-120b✓PASSanswered "rooftop terrace"belief-0.000207218/291full transcript →
openai/gpt-oss-120b✓PASSanswered "loading dock"belief-0.000263205/387full transcript →
openai/gpt-oss-120b✓PASSanswered "library"belief-0.000181206/250full transcript →
openai/gpt-oss-120b✕FAILanswered "conference room"verified-stablebelief-0.000548216/859full transcript →
openai/gpt-oss-120b✓PASSanswered "photo studio"belief-0.000157220/207full transcript →
openai/gpt-oss-120b✓PASSanswered "archive room"belief-0.000181205/251full transcript →
openai/gpt-oss-120b✓PASSanswered "front office"belief-0.000185212/256full transcript →
openai/gpt-oss-120b✓PASSanswered "rooftop terrace"belief-0.000226220/321full transcript →
openai/gpt-oss-120b✓PASSanswered "workshop"belief-0.000318209/478full transcript →
openai/gpt-oss-120b✓PASSanswered "front office"belief-0.000228218/325full transcript →
openai/gpt-oss-120b✓PASSanswered "storage room"belief-0.000175204/240full transcript →
openai/gpt-oss-120b✓PASSanswered "security office"belief-0.000228222/324full transcript →
openai/gpt-oss-120b✓PASSanswered "server room"belief-0.000273207/403full transcript →
openai/gpt-oss-120b✓PASSanswered "observation deck"belief-0.000449214/694full transcript →
openai/gpt-oss-120b✓PASSanswered "loading dock"belief-0.000280224/410full transcript →
openai/gpt-oss-120b✓PASSanswered "west wing"belief-0.000307216/458full transcript →
openai/gpt-oss-120b✓PASSanswered "greenhouse"belief-0.000440217/679full transcript →
openai/gpt-oss-120b✓PASSanswered "archive room"belief-0.000254223/367full transcript →
openai/gpt-oss-120b✓PASSanswered "cafeteria"belief-0.000201219/280full transcript →
openai/gpt-oss-120b✓PASSanswered "utility closet"belief-0.000210227/294full transcript →
openai/gpt-oss-120b✓PASSanswered "loading dock"belief-0.000189223/259full transcript →
openai/gpt-oss-120b✓PASSanswered "cafeteria"belief-0.000141217/181full transcript →
openai/gpt-oss-120b✓PASSanswered "kitchen"belief-0.000144222/184full transcript →
openai/gpt-oss-120b✓PASSanswered "break room"belief-0.000168206/228full transcript →
openai/gpt-oss-120b✓PASSanswered "atrium"belief-0.000152236/194full transcript →
openai/gpt-oss-120b✓PASSanswered "north stairwell"belief-0.000158232/205full transcript →
openai/gpt-oss-120b✓PASSanswered "east wing"belief-0.000175216/238full transcript →
openai/gpt-oss-120b✓PASSanswered "workshop"belief-0.000102214/116full transcript →
openai/gpt-oss-120b✓PASSanswered "server room"belief-0.000208221/291full transcript →
openai/gpt-oss-120b✓PASSanswered "supply closet"belief-0.000178225/240full transcript →
openai/gpt-oss-120b✓PASSanswered "garage"belief-0.000230226/327full transcript →
openai/gpt-oss-120b✓PASSanswered "storage room"belief-0.000140243/173full transcript →
openai/gpt-oss-120b✓PASSanswered "library"belief-0.000142243/176full transcript →
openai/gpt-oss-120b✓PASSanswered "conference room"belief-0.000178241/237full transcript →
openai/gpt-oss-120b✓PASSanswered "supply closet"belief-0.000105251/112full transcript →
openai/gpt-oss-120b✓PASSanswered "south stairwell"belief-0.000130253/153full transcript →
openai/gpt-oss-120b✓PASSanswered "kitchen"belief-0.000192268/253full transcript →
openai/gpt-oss-120b✓PASSanswered "west wing"belief-0.000133267/155full transcript →
openai/gpt-oss-120b✓PASSanswered "observation deck"belief-0.000312275/452full transcript →
openai/gpt-oss-120b✓PASSanswered "north stairwell"belief-0.000212283/282full transcript →
openai/gpt-oss-120b✓PASSanswered "security office"belief-0.000220287/295full transcript →
openai/gpt-oss-120b✓PASSanswered "workshop"belief-0.000186267/243full transcript →
openai/gpt-oss-120b✓PASSanswered "greenhouse"belief-0.000146279/174full transcript →
openai/gpt-oss-120b✓PASSanswered "boiler room"belief-0.000216286/288full transcript →
openai/gpt-oss-120b✓PASSanswered "conference room"belief-0.000265286/370full transcript →
openai/gpt-oss-120b✓PASSanswered "south stairwell"belief-0.000145219/187full transcript →
openai/gpt-oss-120b✓PASSanswered "server room"belief-0.000153213/202full transcript →
openai/gpt-oss-120b✓PASSanswered "lobby"belief-0.000189213/261full transcript →
openai/gpt-oss-120b✓PASSanswered "security office"belief-0.000161223/212full transcript →
openai/gpt-oss-120b✓PASSanswered "break room"belief-0.000173220/234full transcript →
openai/gpt-oss-120b✓PASSanswered "lobby"belief-0.000155270/191full transcript →
openai/gpt-oss-120b✓PASSanswered "courtyard"belief-0.000249287/343full transcript →
openai/gpt-oss-120b✓PASSanswered "photo studio"belief-0.000236280/323full transcript →
openai/gpt-oss-120b✓PASSanswered "archive room"belief-0.000233284/318full transcript →
openai/gpt-oss-120b✓PASSanswered "photo studio"belief-0.000141370/143full transcript →
openai/gpt-oss-120b✓PASSanswered "photo studio"belief-0.000155351/171full transcript →
openai/gpt-oss-120b✓PASSanswered "security office"belief-0.000195275/257full transcript →
openai/gpt-oss-120b✓PASSanswered "north stairwell"belief-0.000171362/195full transcript →
openai/gpt-oss-120b✓PASSanswered "basement"belief-0.000169357/193full transcript →
openai/gpt-oss-120b✓PASSanswered "front office"belief-0.000154369/164full transcript →
openai/gpt-oss-120b✓PASSanswered "reception area"belief-0.000161448/157full transcript →
openai/gpt-oss-120b✓PASSanswered "rooftop terrace"belief-0.000140454/120full transcript →
openai/gpt-oss-120b✓PASSanswered "storage room"belief-0.000144457/126full transcript →
openai/gpt-oss-120b✓PASSanswered "south stairwell"belief-0.000151448/139full transcript →
openai/gpt-oss-120b✓PASSanswered "east wing"belief-0.000238551/259full transcript →
openai/gpt-oss-120b✓PASSanswered "archive room"belief-0.000141462/119full transcript →
openai/gpt-oss-120b✓PASSanswered "meditation room"belief-0.000182537/169full transcript →
openai/gpt-oss-120b✓PASSanswered "wine cellar"belief-0.000190516/188full transcript →
openai/gpt-oss-120b✓PASSanswered "reception area"belief-0.000148512/118full transcript →
openai/gpt-oss-120b✓PASSanswered "courtyard"belief-0.000322520/406full transcript →
openai/gpt-oss-120b✓PASSanswered "basement"belief-0.000119216/144full transcript →
openai/gpt-oss-120b✓PASSanswered "gym"belief-0.000145218/188full transcript →
openai/gpt-oss-120b✓PASSanswered "observation deck"belief-0.000178219/242full transcript →
openai/gpt-oss-120b✓PASSanswered "south stairwell"belief-0.000108203/130full transcript →
openai/gpt-oss-120b✓PASSanswered "kitchen"belief-0.000221216/314full transcript →
openai/gpt-oss-120b✓PASSanswered "server room"belief-0.000184205/256full transcript →
openai/gpt-oss-120b✕FAILanswered "library"verified-stablebelief-0.000238217/343full transcript →
openai/gpt-oss-120b✓PASSanswered "kitchen"belief-0.000161219/214full transcript →
openai/gpt-oss-120b✓PASSanswered "library"belief-0.000168205/229full transcript →
openai/gpt-oss-120b✓PASSanswered "print room"belief-0.000175209/240full transcript →
openai/gpt-oss-120b✓PASSanswered "workshop"belief-0.000116223/138full transcript →
openai/gpt-oss-120b✓PASSanswered "east wing"belief-0.000187206/261full transcript →
openai/gpt-oss-120b✕FAILanswered "reception"verified-stablebelief-0.000138217/176full transcript →
openai/gpt-oss-120b✓PASSanswered "library"belief-0.000185200/259full transcript →
openai/gpt-oss-120b✓PASSanswered "boiler room"belief-0.000133212/169full transcript →
openai/gpt-oss-120b✓PASSanswered "mail room"belief-0.000109205/131full transcript →
openai/gpt-oss-120b✕ERRORhttp 429: b'{"error":{"message":"Rate limit reached for model `openai/gpt-oss-120b` in organization `org_01jpwfsbh0f7atrs0qxaewb6b7` service tier `on_demand` on tokens per minute (TPM): Limit 8000, Used 7368, Req'belief-0.0000000/0full transcript →
openai/gpt-oss-120b✓PASSanswered "observation deck"belief-0.000147202/195full transcript →
openai/gpt-oss-120b✓PASSanswered "wine cellar"belief-0.000267221/390full transcript →
openai/gpt-oss-120b✓PASSanswered "courtyard"belief-0.000117203/144full transcript →
openai/gpt-oss-120b✓PASSanswered "south stairwell"belief-0.000233209/336full transcript →
openai/gpt-oss-120b✓PASSanswered "storage room"belief-0.000103201/122full transcript →
openai/gpt-oss-120b✓PASSanswered "utility closet"belief-0.000147207/194full transcript →
openai/gpt-oss-120b✓PASSanswered "cafeteria"belief-0.000164213/220full transcript →
openai/gpt-oss-120b✓PASSanswered "rooftop terrace"belief-0.000144205/188full transcript →
openai/gpt-oss-120b✓PASSanswered "atrium"belief-0.000193226/266full transcript →
openai/gpt-oss-120b✓PASSanswered "south stairwell"belief-0.000194226/267full transcript →
openai/gpt-oss-120b✓PASSanswered "workshop"belief-0.000143210/186full transcript →
openai/gpt-oss-120b✓PASSanswered "greenhouse"belief-0.000173201/238full transcript →
openai/gpt-oss-120b✓PASSanswered "boiler room"belief-0.000180217/245full transcript →
openai/gpt-oss-120b✓PASSanswered "atrium"belief-0.000181201/251full transcript →
openai/gpt-oss-120b✓PASSanswered "library"belief-0.000261224/379full transcript →
openai/gpt-oss-120b✓PASSanswered "south stairwell"belief-0.000217215/308full transcript →
openai/gpt-oss-120b✓PASSanswered "south stairwell"belief-0.000190210/265full transcript →
openai/gpt-oss-120b✓PASSanswered "server room"belief-0.000143221/183full transcript →
openai/gpt-oss-120b✓PASSanswered "observation deck"belief-0.000152206/202full transcript →
openai/gpt-oss-120b✓PASSanswered "utility closet"belief-0.000126223/155full transcript →
openai/gpt-oss-120b✓PASSanswered "reception area"belief-0.000120223/145full transcript →
openai/gpt-oss-120b✓PASSanswered "library"belief-0.000120202/150full transcript →
openai/gpt-oss-120b✓PASSanswered "south stairwell"belief-0.000274213/404full transcript →
openai/gpt-oss-120b✓PASSanswered "front office"belief-0.000155224/203full transcript →
openai/gpt-oss-120b✓PASSanswered "print room"belief-0.000162218/216full transcript →
openai/gpt-oss-120b✓PASSanswered "mail room"belief-0.000141217/181full transcript →
openai/gpt-oss-120b✓PASSanswered "security office"belief-0.000127221/157full transcript →
openai/gpt-oss-120b✓PASSanswered "supply closet"belief-0.000180222/244full transcript →
openai/gpt-oss-120b✓PASSanswered "front office"belief-0.000159210/213full transcript →
openai/gpt-oss-120b✓PASSanswered "meditation room"belief-0.000128219/158full transcript →
openai/gpt-oss-120b✓PASSanswered "security office"belief-0.000131215/165full transcript →
openai/gpt-oss-120b✓PASSanswered "west wing"belief-0.000196222/272full transcript →
openai/gpt-oss-120b✓PASSanswered "observation deck"belief-0.000119202/148full transcript →
openai/gpt-oss-120b✓PASSanswered "utility closet"belief-0.000137209/176full transcript →
openai/gpt-oss-120b✓PASSanswered "courtyard"belief-0.000120221/145full transcript →
openai/gpt-oss-120b✓PASSanswered "loading dock"belief-0.000151224/195full transcript →
openai/gpt-oss-120b✓PASSanswered "rooftop terrace"belief-0.000094201/107full transcript →
openai/gpt-oss-120b✓PASSanswered "break room"belief-0.000186222/255full transcript →
openai/gpt-oss-120b✓PASSanswered "courtyard"belief-0.000492762/630full transcript →
openai/gpt-oss-120b✓PASSanswered "security office"belief-0.000279779/270full transcript →
openai/gpt-oss-120b✕FAILanswered "unknown"verified-stablebelief-0.000513752/667full transcript →
openai/gpt-oss-120b✓PASSanswered "garage"belief-0.000486763/620full transcript →
openai/gpt-oss-120b✓PASSanswered "greenhouse"belief-0.001263770/1913full transcript →