Data to …… trials

Run camp-full_map-super-002/breaking_point

96% PASS

What this run revealed, then every individual trial as evidence.

56
Trials
96%
54/56 scored
over trials
Pass rate
2
4%
Failures
$0.0448
Run cost

Run camp-full_map-super-002/breaking_point: 96% pass rate on belief queries.

Of 56 scored trials (belief query kind), the model passed 96% and failed 4%.

2 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

In 1 of 2 classifiable belief failures the model's wrong answer matches the 'most_frequent_location' shortcut -- a plausible but ungrounded guess rather than tracking what that entity actually witnessed.

Notable failures

  • record 26f5b058 — asked "Where does Quinn believe the bright basket is? Answer with only the location name (or 'unknown' if they have no belief)."
    expected: the basement
    actual: loading dock

    the model's answer matches the 'most_frequent_location' shortcut, not the verified belief

  • record 1b2f79d9 — asked "Where does Marcus believe the shiny box is? Answer with only the location name (or 'unknown' if they have no belief)."
    expected: the wine cellar
    actual: reception area

    the model reported the object's TRUE final location, not what the queried entity actually believed -- it collapsed belief into reality

Trials56 trials
Per-trial table (56 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "garage"belief-0.000155167/315full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "meditation room"belief-0.000168176/342full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "rooftop terrace"belief-0.000264174/555full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "east wing"belief-0.000240178/502full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "east wing"belief-0.000158163/322full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "utility closet"belief-0.000137171/274full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "atrium"belief-0.000163164/334full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "conference room"belief-0.000978514/2083full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "storage room"belief-0.001048517/2236full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "utility closet"belief-0.000348521/681full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "workshop"belief-0.000896493/1904full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "loading dock"unverifiedbelief-0.000726521/1520full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "basement"belief-0.001382496/2984full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "storage room"belief-0.000488477/999full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "rooftop terrace"belief-0.000588870/1151full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "observation deck"belief-0.000568869/1107full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "kitchen"belief-0.000683837/1369full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "garage"belief-0.000702852/1409full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "lobby"belief-0.000785830/1597full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "archive room"belief-0.000519835/1004full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "boiler room"belief-0.000575855/1125full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "front office"belief-0.0007501225/1449full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "gym"belief-0.0005471211/1000full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "print room"belief-0.0007061227/1350full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "reception area"belief-0.0005511201/1012full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "archive room"belief-0.0005131243/920full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "wine cellar"belief-0.0010391192/2097full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "gym"belief-0.0006631188/1263full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "supply closet"belief-0.0009581533/1856full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "archive room"belief-0.0007021573/1280full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "archive room"belief-0.0006661496/1215full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "greenhouse"belief-0.0006471540/1164full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "break room"belief-0.0013811514/2800full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "rooftop terrace"belief-0.0006581525/1192full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "west wing"belief-0.0008741520/1671full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "rooftop terrace"belief-0.0017441800/3555full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "workshop"belief-0.0019021799/3906full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "courtyard"belief-0.0007991770/1461full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "photo studio"belief-0.0009621770/1822full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "reception area"unverifiedbelief-0.0015591765/3150full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "greenhouse"belief-0.0007051754/1255full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "server room"belief-0.0008171796/1497full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "wine cellar"belief-0.0010382054/1942full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "loading dock"belief-0.0007292110/1245full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "meditation room"belief-0.0007822051/1374full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "south stairwell"belief-0.0005942089/948full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "workshop"belief-0.0009952039/1848full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "archive room"belief-0.0007082035/1212full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "conference room"belief-0.0017232121/3452full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "reception area"belief-0.0013702323/2632full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "supply closet"belief-0.0012842236/2456full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "supply closet"belief-0.0014672267/2857full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "server room"belief-0.0008292274/1437full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "server room"belief-0.0006632201/1083full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "wine cellar"belief-0.0010722291/1976full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "front office"belief-0.0008192308/1410full transcript →