Data to …… trials

Run camp-full_map-nano-002/breaking_point

91% PASS

What this run revealed, then every individual trial as evidence.

56
10 errors
Trials
91%
42/46 scored
over trials
Pass rate
4
7%
Failures
$0.0298
Run cost

Run camp-full_map-nano-002/breaking_point: 91% pass rate on belief queries.

Of 46 scored trials (belief query kind), the model passed 91% and failed 9%.

10 additional trial(s) ended in ERROR (infra failures, cost cap, or cancellation) and are excluded from the pass rate above.

4 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Diffuse weakness (below 90% but no outright flip) on: temporal (pass rate 89% at the high end).

In 2 of 4 classifiable belief failures the model reported where the object ACTUALLY was, not where the queried entity BELIEVED it was -- it collapsed their perspective into reality.

Notable failures

  • record 93fd41ae — asked "Where does Quinn believe the bright basket is? Answer with only the location name (or 'unknown' if they have no belief)."
    expected: the basement
    actual: photo studio

    the model reported the object's TRUE final location, not what the queried entity actually believed -- it collapsed belief into reality

  • record 928137f8 — asked "Where does David believe the red folder is? Answer with only the location name (or 'unknown' if they have no belief)."
    expected: the basement
    actual: supply closet

    the model reported the object's TRUE final location, not what the queried entity actually believed -- it collapsed belief into reality

  • record 7d4fafff — asked "Where does Elliot believe the sturdy basket is? Answer with only the location name (or 'unknown' if they have no belief)."
    expected: the photo studio
    actual: library

    the model's answer matches the 'query_agent_own_last_move' shortcut, not the verified belief

Trials56 trials
Per-trial table (56 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "garage"belief-0.000064167/279full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "meditation room"belief-0.000095176/430full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "rooftop terrace"belief-0.000133174/624full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "east wing"belief-0.000157178/739full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "east wing"belief-0.000066163/288full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "utility closet"belief-0.000085171/381full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "atrium"belief-0.000094164/430full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "conference room"belief-0.000611514/2928full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "storage room"belief-0.000557517/2657full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✕ERRORlengthbelief-0.000826521/4000full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "workshop"belief-0.000416493/1958full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "photo studio"unverifiedbelief-0.000703521/3385full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "supply closet"unverifiedbelief-0.000363496/1693full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "storage room"belief-0.000290477/1332full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "rooftop terrace"belief-0.000289870/1225full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "observation deck"belief-0.000554869/2552full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "kitchen"belief-0.000204837/809full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "garage"belief-0.000172852/648full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "lobby"belief-0.000251830/1049full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "archive room"belief-0.000543835/2504full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "boiler room"belief-0.000534855/2456full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✕ERRORlengthbelief-0.0008611225/4000full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "gym"belief-0.0007541211/3467full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "print room"belief-0.0007841227/3615full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "reception area"belief-0.0004391201/1894full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "archive room"belief-0.0002971243/1173full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "wine cellar"belief-0.0007021192/3210full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "gym"belief-0.0003301188/1355full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✕ERRORlengthbelief-0.0008771533/4000full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "archive room"belief-0.0006601573/2907full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✕ERRORlengthbelief-0.0008751496/4000full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "greenhouse"belief-0.0007711540/3470full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "break room"belief-0.0003061514/1150full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "rooftop terrace"belief-0.0008341525/3789full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "west wing"belief-0.0005011520/2127full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "rooftop terrace"belief-0.0008531800/3815full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "workshop"belief-0.0004641799/1868full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✕ERRORlengthbelief-0.0008891770/4000full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "library"unverifiedbelief-0.0007901770/3508full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "wine cellar"belief-0.0005431765/2273full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "greenhouse"belief-0.0007931754/3524full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "server room"belief-0.0005541796/2321full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✕ERRORlengthbelief-0.0009032054/4000full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "loading dock"belief-0.0006462110/2701full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "reception area"unverifiedbelief-0.0003722051/1349full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✕ERRORlengthbelief-0.0009042089/4000full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✕ERRORlengthbelief-0.0009022039/4000full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "archive room"belief-0.0005412035/2194full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "conference room"belief-0.0007832121/3387full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "reception area"belief-0.0004122323/1479full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "supply closet"belief-0.0004452236/1666full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "supply closet"belief-0.0005272267/2066full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "server room"belief-0.0003882274/1370full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✕ERRORlengthbelief-0.0009102201/4000full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✕ERRORlengthbelief-0.0009152291/4000full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "front office"belief-0.0003182308/1014full transcript →