Data to …… trials

Run camp-belief-120b-001

53% PASS

What this run revealed, then every individual trial as evidence.

214
Trials
53%
113/214 scored
over trials
Pass rate
101
47%
Failures
$0.0595
Run cost

Run camp-belief-120b-001: 53% pass rate on belief queries.

Of 214 scored trials (belief query kind), the model passed 53% and failed 47%.

101 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

101 belief FAIL(s) were recorded, but they predate the generator-hardening fix and can't be safely re-classified against regenerated ground truth.

Notable failures

  • record 03b07440 — asked "Where does Gideon believe the worn tablet is? Answer with only the location name (or 'unknown' if they have no belief)."
    expected: (unknown)
    actual: unknown

    answer did not match the verified ground truth

  • record 14d4516a — asked "Where does Victor believe the heavy box is? Answer with only the location name (or 'unknown' if they have no belief)."
    expected: (unknown)
    actual: unknown

    answer did not match the verified ground truth

  • record c1b85f09 — asked "Where does Elena believe the shiny tablet is? Answer with only the location name (or 'unknown' if they have no belief)."
    expected: (unknown)
    actual: unknown

    answer did not match the verified ground truth

Trials214 trials
Per-trial table (214 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000156195/211full transcript →
openai/gpt-oss-120b✓PASSanswered "storage room"belief-0.000112200/136full transcript →
openai/gpt-oss-120b✓PASSanswered "break room"belief-0.000127191/164full transcript →
openai/gpt-oss-120b✓PASSanswered "kitchen"belief-0.000099189/117full transcript →
openai/gpt-oss-120b✓PASSanswered "photo studio"belief-0.000119199/149full transcript →
openai/gpt-oss-120b✓PASSanswered "storage room"belief-0.000105227/119full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000126231/152full transcript →
openai/gpt-oss-120b✓PASSanswered "front office"belief-0.000148226/191full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000134229/166full transcript →
openai/gpt-oss-120b✓PASSanswered "library"belief-0.000164225/217full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000261255/371full transcript →
openai/gpt-oss-120b✓PASSanswered "reception area"belief-0.000387266/579full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000093261/89full transcript →
openai/gpt-oss-120b✓PASSanswered "wine cellar"belief-0.000254271/355full transcript →
openai/gpt-oss-120b✓PASSanswered "west wing"belief-0.000285258/411full transcript →
openai/gpt-oss-120b✓PASSanswered "atrium"belief-0.000130297/143full transcript →
openai/gpt-oss-120b✓PASSanswered "archive room"belief-0.000288310/402full transcript →
openai/gpt-oss-120b✓PASSanswered "supply closet"belief-0.000230299/309full transcript →
openai/gpt-oss-120b✓PASSanswered "courtyard"belief-0.000366301/534full transcript →
openai/gpt-oss-120b✓PASSanswered "east wing"belief-0.000439295/658full transcript →
openai/gpt-oss-120b?FAILanswered "greenhouse"unverifiedbelief-0.000388325/565full transcript →
openai/gpt-oss-120b✓PASSanswered "front office"belief-0.000426326/629full transcript →
openai/gpt-oss-120b?FAILanswered "utility closet"unverifiedbelief-0.000144330/157full transcript →
openai/gpt-oss-120b✓PASSanswered "basement"belief-0.000177323/214full transcript →
openai/gpt-oss-120b?FAILanswered "mail room"unverifiedbelief-0.000231334/302full transcript →
openai/gpt-oss-120b✓PASSanswered "kitchen"belief-0.000117199/145full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000121197/152full transcript →
openai/gpt-oss-120b✓PASSanswered "greenhouse"belief-0.000101197/119full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000219194/317full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000130195/168full transcript →
openai/gpt-oss-120b✓PASSanswered "workshop"belief-0.000174194/242full transcript →
openai/gpt-oss-120b✓PASSanswered "gym"belief-0.000386190/596full transcript →
openai/gpt-oss-120b✓PASSanswered "utility closet"belief-0.000203200/288full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000243190/357full transcript →
openai/gpt-oss-120b✓PASSanswered "utility closet"belief-0.000362193/555full transcript →
openai/gpt-oss-120b✓PASSanswered "lobby"belief-0.000258190/383full transcript →
openai/gpt-oss-120b✓PASSanswered "utility closet"belief-0.000401194/620full transcript →
openai/gpt-oss-120b✓PASSanswered "photo studio"belief-0.000498195/782full transcript →
openai/gpt-oss-120b✓PASSanswered "kitchen"belief-0.000391194/603full transcript →
openai/gpt-oss-120b✓PASSanswered "supply closet"belief-0.000208196/298full transcript →
openai/gpt-oss-120b✓PASSanswered "west wing"belief-0.000167194/230full transcript →
openai/gpt-oss-120b?FAILanswered "loading dock"unverifiedbelief-0.000393194/606full transcript →
openai/gpt-oss-120b✓PASSanswered "storage room"belief-0.000106198/128full transcript →
openai/gpt-oss-120b✓PASSanswered "photo studio"belief-0.000169199/232full transcript →
openai/gpt-oss-120b✓PASSanswered "reception area"belief-0.000263199/388full transcript →
openai/gpt-oss-120b✓PASSanswered "garage"belief-0.000114193/141full transcript →
openai/gpt-oss-120b✓PASSanswered "north stairwell"belief-0.000145194/193full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000183194/257full transcript →
openai/gpt-oss-120b✓PASSanswered "north stairwell"belief-0.000282191/423full transcript →
openai/gpt-oss-120b✓PASSanswered "greenhouse"belief-0.000320198/484full transcript →
openai/gpt-oss-120b✓PASSanswered "observation deck"belief-0.000197191/281full transcript →
openai/gpt-oss-120b✓PASSanswered "gym"belief-0.000120194/152full transcript →
openai/gpt-oss-120b✓PASSanswered "lobby"belief-0.000113194/140full transcript →
openai/gpt-oss-120b✓PASSanswered "rooftop terrace"belief-0.000127195/163full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000186195/262full transcript →
openai/gpt-oss-120b✓PASSanswered "east wing"belief-0.000101207/117full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000178205/245full transcript →
openai/gpt-oss-120b✓PASSanswered "archive room"belief-0.000199205/281full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000198202/280full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000187206/260full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000090217/96full transcript →
openai/gpt-oss-120b✓PASSanswered "mail room"belief-0.000102213/117full transcript →
openai/gpt-oss-120b✓PASSanswered "reception area"belief-0.000144217/186full transcript →
openai/gpt-oss-120b✓PASSanswered "reception area"belief-0.000168216/226full transcript →
openai/gpt-oss-120b✓PASSanswered "mail room"belief-0.000127217/157full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000236250/331full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000222242/309full transcript →
openai/gpt-oss-120b✓PASSanswered "archive room"belief-0.000116247/131full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000106244/115full transcript →
openai/gpt-oss-120b✓PASSanswered "north stairwell"belief-0.000197255/264full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000186255/247full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000110265/117full transcript →
openai/gpt-oss-120b✓PASSanswered "courtyard"belief-0.000135262/159full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000241260/337full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000160257/202full transcript →
openai/gpt-oss-120b✓PASSanswered "courtyard"belief-0.000110196/135full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000257197/379full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000063204/54full transcript →
openai/gpt-oss-120b✓PASSanswered "utility closet"belief-0.000140198/183full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000228196/331full transcript →
openai/gpt-oss-120b✓PASSanswered "courtyard"belief-0.000110261/118full transcript →
openai/gpt-oss-120b✓PASSanswered "workshop"belief-0.000111263/120full transcript →
openai/gpt-oss-120b✓PASSanswered "courtyard"belief-0.000171256/221full transcript →
openai/gpt-oss-120b✓PASSanswered "cafeteria"belief-0.000134259/159full transcript →
openai/gpt-oss-120b✓PASSanswered "meditation room"belief-0.000124263/141full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000224343/288full transcript →
openai/gpt-oss-120b✓PASSanswered "east wing"belief-0.000114348/103full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000238336/312full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000478349/709full transcript →
openai/gpt-oss-120b✓PASSanswered "workshop"belief-0.000177352/207full transcript →
openai/gpt-oss-120b✓PASSanswered "break room"belief-0.000170428/177full transcript →
openai/gpt-oss-120b✓PASSanswered "rooftop terrace"belief-0.000172432/179full transcript →
openai/gpt-oss-120b✓PASSanswered "basement"belief-0.000171422/180full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000257449/316full transcript →
openai/gpt-oss-120b✓PASSanswered "west wing"belief-0.000136417/122full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000288490/357full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000186497/185full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000297517/366full transcript →
openai/gpt-oss-120b✓PASSanswered "boiler room"belief-0.000192497/196full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000208491/224full transcript →
openai/gpt-oss-120b✓PASSanswered "cafeteria"belief-0.000212229/296full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000358233/538full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000218235/304full transcript →
openai/gpt-oss-120b✓PASSanswered "meditation room"belief-0.000133230/165full transcript →
openai/gpt-oss-120b✓PASSanswered "south stairwell"belief-0.000159236/206full transcript →
openai/gpt-oss-120b✓PASSanswered "south stairwell"belief-0.000147479/126full transcript →
openai/gpt-oss-120b✓PASSanswered "kitchen"belief-0.000269468/332full transcript →
openai/gpt-oss-120b✓PASSanswered "north stairwell"belief-0.000241483/281full transcript →
openai/gpt-oss-120b✓PASSanswered "security office"belief-0.000155464/142full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000186469/193full transcript →
openai/gpt-oss-120b?FAILanswered "server room"unverifiedbelief-0.000133517/93full transcript →
openai/gpt-oss-120b✓PASSanswered "archive room"belief-0.000148478/127full transcript →
openai/gpt-oss-120b✓PASSanswered "garage"belief-0.000162477/150full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000328471/429full transcript →
openai/gpt-oss-120b✓PASSanswered "conference room"belief-0.000198494/207full transcript →
openai/gpt-oss-120b?FAILanswered "conference room"unverifiedbelief-0.000493690/649full transcript →
openai/gpt-oss-120b✓PASSanswered "north stairwell"belief-0.000534700/715full transcript →
openai/gpt-oss-120b?FAILanswered "photo studio"unverifiedbelief-0.000546716/731full transcript →
openai/gpt-oss-120b✓PASSanswered "meditation room"belief-0.000457699/587full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000338698/389full transcript →
openai/gpt-oss-120b?FAILanswered "reception area"unverifiedbelief-0.000370703/441full transcript →
openai/gpt-oss-120b✓PASSanswered "kitchen"belief-0.000632694/880full transcript →
openai/gpt-oss-120b?FAILanswered "mail room"unverifiedbelief-0.000753699/1081full transcript →
openai/gpt-oss-120b?FAILanswered "greenhouse"unverifiedbelief-0.000758696/1090full transcript →
openai/gpt-oss-120b?FAILanswered "boiler room"unverifiedbelief-0.000259705/255full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000181196/252full transcript →
openai/gpt-oss-120b✓PASSanswered "security office"belief-0.000253198/373full transcript →
openai/gpt-oss-120b✓PASSanswered "boiler room"belief-0.000152195/205full transcript →
openai/gpt-oss-120b?FAILanswered "café."unverifiedbelief-0.000161188/222full transcript →
openai/gpt-oss-120b✓PASSanswered "wine cellar"belief-0.000089191/101full transcript →
openai/gpt-oss-120b✓PASSanswered "archive room"belief-0.000088195/98full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000076200/76full transcript →
openai/gpt-oss-120b✓PASSanswered "mail room"belief-0.000190196/268full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000230188/337full transcript →
openai/gpt-oss-120b✓PASSanswered "atrium"belief-0.000140192/185full transcript →
openai/gpt-oss-120b✓PASSanswered "workshop"belief-0.000239557/259full transcript →
openai/gpt-oss-120b?FAILanswered "print room"unverifiedbelief-0.000214560/217full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000368576/470full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000373561/482full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000739578/1088full transcript →
openai/gpt-oss-120b✓PASSanswered "kitchen"belief-0.000356625/437full transcript →
openai/gpt-oss-120b✓PASSanswered "gym"belief-0.000552655/757full transcript →
openai/gpt-oss-120b?FAILanswered "lobby"unverifiedbelief-0.000603634/846full transcript →
openai/gpt-oss-120b✓PASSanswered "north stairwell"belief-0.000606634/852full transcript →
openai/gpt-oss-120b✓PASSanswered "observation deck"belief-0.000532619/732full transcript →
openai/gpt-oss-120b?FAILanswered "lobby"unverifiedbelief-0.000178393/199full transcript →
openai/gpt-oss-120b?FAILanswered "gym"unverifiedbelief-0.000246397/311full transcript →
openai/gpt-oss-120b✓PASSanswered "gym"belief-0.000489398/715full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000174407/189full transcript →
openai/gpt-oss-120b?FAILanswered "basement"unverifiedbelief-0.000359399/499full transcript →
openai/gpt-oss-120b✓PASSanswered "basement"belief-0.000597704/819full transcript →
openai/gpt-oss-120b?FAILanswered "workshop"unverifiedbelief-0.000452716/574full transcript →
openai/gpt-oss-120b?FAILanswered "meditation room"unverifiedbelief-0.000366706/434full transcript →
openai/gpt-oss-120b✓PASSanswered "north stairwell"belief-0.000491714/640full transcript →
openai/gpt-oss-120b?FAILanswered "loading dock"unverifiedbelief-0.000436704/550full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000203696/165full transcript →
openai/gpt-oss-120b?FAILanswered "greenhouse"unverifiedbelief-0.000374712/446full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000399703/490full transcript →
openai/gpt-oss-120b?FAILanswered "gym"unverifiedbelief-0.000329704/373full transcript →
openai/gpt-oss-120b✓PASSanswered "boiler room"belief-0.000355707/415full transcript →
openai/gpt-oss-120b✓PASSanswered "break room"belief-0.000366324/529full transcript →
openai/gpt-oss-120b✓PASSanswered "library"belief-0.000335334/475full transcript →
openai/gpt-oss-120b✓PASSanswered "rooftop terrace"belief-0.000419331/616full transcript →
openai/gpt-oss-120b?FAILanswered "east wing"unverifiedbelief-0.000352334/503full transcript →
openai/gpt-oss-120b✓PASSanswered "garage"belief-0.000400322/586full transcript →
openai/gpt-oss-120b✓PASSanswered "basement"belief-0.000396717/480full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000381704/459full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000558710/752full transcript →
openai/gpt-oss-120b?FAILanswered "lobby"unverifiedbelief-0.000474697/616full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000314712/345full transcript →
openai/gpt-oss-120b✓PASSanswered "south stairwell"belief-0.000147196/196full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000077190/80full transcript →
openai/gpt-oss-120b✓PASSanswered "front office"belief-0.000116196/145full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000189195/267full transcript →
openai/gpt-oss-120b✓PASSanswered "observation deck"belief-0.000092199/103full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000199190/284full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000205193/294full transcript →
openai/gpt-oss-120b✓PASSanswered "security office"belief-0.000173190/240full transcript →
openai/gpt-oss-120b✓PASSanswered "reception area"belief-0.000143192/190full transcript →
openai/gpt-oss-120b✓PASSanswered "courtyard"belief-0.000170193/235full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000440559/593full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000780565/1158full transcript →
openai/gpt-oss-120b✓PASSanswered "north stairwell"belief-0.000237574/251full transcript →
openai/gpt-oss-120b✓PASSanswered "wine cellar"belief-0.000170569/141full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000365558/469full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000285582/330full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000487565/671full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000519569/723full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000224584/227full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000330568/408full transcript →
openai/gpt-oss-120b✓PASSanswered "break room"belief-0.000709629/1024full transcript →
openai/gpt-oss-120b✓PASSanswered "storage room"belief-0.000531639/726full transcript →
openai/gpt-oss-120b?FAILanswered "workshop"unverifiedbelief-0.000682626/981full transcript →
openai/gpt-oss-120b?FAILanswered "library"unverifiedbelief-0.000486631/652full transcript →
openai/gpt-oss-120b?FAILanswered "library"unverifiedbelief-0.000625648/879full transcript →
openai/gpt-oss-120b✓PASSanswered "garage"belief-0.000581641/808full transcript →
openai/gpt-oss-120b✓PASSanswered "front office"belief-0.000924645/1379full transcript →
openai/gpt-oss-120b?FAILanswered "garage"unverifiedbelief-0.000822630/1213full transcript →
openai/gpt-oss-120b✓PASSanswered "library"belief-0.000384641/480full transcript →
openai/gpt-oss-120b?FAILanswered "gym"unverifiedbelief-0.000959650/1436full transcript →
openai/gpt-oss-120b✓PASSanswered "workshop"belief-0.000208393/249full transcript →
openai/gpt-oss-120b✓PASSanswered "greenhouse"belief-0.000193397/222full transcript →
openai/gpt-oss-120b?FAILanswered "basement"unverifiedbelief-0.000128388/116full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000260397/334full transcript →
openai/gpt-oss-120b?FAILanswered "cafeteria"unverifiedbelief-0.000195387/229full transcript →
openai/gpt-oss-120b✓PASSanswered "east wing"belief-0.000350391/485full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000373397/523full transcript →
openai/gpt-oss-120b?FAILanswered "photo studio"unverifiedbelief-0.000185396/209full transcript →
openai/gpt-oss-120b✓PASSanswered "conference room"belief-0.000315397/425full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000117390/98full transcript →
openai/gpt-oss-120b?FAILanswered "unknown"unverifiedbelief-0.000259693/259full transcript →
openai/gpt-oss-120b?FAILanswered "server room"unverifiedbelief-0.000442714/558full transcript →
openai/gpt-oss-120b?FAILanswered "boiler room"unverifiedbelief-0.000310702/341full transcript →
openai/gpt-oss-120b?FAILanswered "loading dock"unverifiedbelief-0.000240714/221full transcript →