Run camp-full_map-ultra-002/adversarial_belief
99% PASSWhat this run revealed, then every individual trial as evidence.
130
2 errors
Trials
99%
127/128 scored
over trials
Pass rate
1
1 stable / 0 noise
Failures
$0.3080
Run cost
Run camp-full_map-ultra-002/adversarial_belief: passed essentially everything (99%) -- this test did not challenge the model.
Of 128 scored trials (belief query kind), the model passed 99%. That's saturation: the difficulty range probed here isn't finding this model's limits. Consider a harder preset (breaking_point or adversarial_belief) or a wider dimension range.
2 additional trial(s) ended in ERROR (infra failures, cost cap, or cancellation) and are excluded from the pass rate above.
1 failures recorded; all 1 survived verification.
1 belief failure(s) were classified against known shortcuts, but none of them dominate -- the failures don't share one obvious wrong-answer pattern.
Notable failures
- record 7b713cba expected: the photo studioactual: unknown
Trials
Per-trial table
| model | verdict | kind | difficulty | cost ($) | tokens in/out | |
|---|---|---|---|---|---|---|
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "supply closet" | belief | - | 0.000809 | 175/328 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "cafeteria" | belief | - | 0.000856 | 176/349 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "mail room" | belief | - | 0.000766 | 154/313 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "utility closet" | belief | - | 0.000835 | 153/345 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "north stairwell" | belief | - | 0.001225 | 161/520 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "break room" | belief | - | 0.000747 | 205/293 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "observation deck" | belief | - | 0.001031 | 236/415 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "north stairwell" | belief | - | 0.001517 | 223/639 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "north stairwell" | belief | - | 0.002912 | 209/1276 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "break room" | belief | - | 0.001148 | 290/456 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "observation deck" | belief | - | 0.002303 | 276/984 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "library" | belief | - | 0.001757 | 298/731 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "security office" | belief | - | 0.004330 | 269/1907 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "east wing" | belief | - | 0.002067 | 355/859 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "server room" | belief | - | 0.002756 | 341/1175 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "storage room" | belief | - | 0.002624 | 338/1116 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "library" | belief | - | 0.002156 | 317/908 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "break room" | belief | - | 0.007182 | 222/3214 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "lobby" | belief | - | 0.001935 | 393/790 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "archive room" | belief | - | 0.007761 | 289/3462 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "supply closet" | belief | - | 0.001904 | 433/767 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "library" | belief | - | 0.003385 | 329/1464 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "wine cellar" | belief | - | 0.007150 | 395/3160 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "lobby" | belief | - | 0.006793 | 425/2991 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "south stairwell" | belief | - | 0.010888 | 387/4861 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "utility closet" | belief | - | 0.000532 | 154/207 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "observation deck" | belief | - | 0.000750 | 153/306 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "photo studio" | belief | - | 0.000795 | 165/324 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "front office" | belief | - | 0.000890 | 170/366 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "mail room" | belief | - | 0.000878 | 177/359 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "north stairwell" | belief | - | 0.001221 | 171/516 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "print room" | belief | - | 0.000728 | 158/295 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "conference room" | belief | - | 0.000799 | 176/323 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "basement" | belief | - | 0.000724 | 154/294 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "greenhouse" | belief | - | 0.001454 | 171/622 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "basement" | belief | - | 0.001482 | 170/635 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "break room" | belief | - | 0.002331 | 170/1021 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "reception area" | belief | - | 0.001016 | 184/420 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "mail room" | belief | - | 0.001788 | 171/774 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "library" | belief | - | 0.001323 | 152/567 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "south stairwell" | belief | - | 0.002492 | 167/1095 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "supply closet" | belief | - | 0.004522 | 155/2020 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "boiler room" | belief | - | 0.002888 | 171/1274 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "archive room" | belief | - | 0.002038 | 156/891 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "observation deck" | belief | - | 0.002981 | 176/1315 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "break room" | belief | - | 0.006062 | 174/2716 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "garage" | belief | - | 0.013379 | 164/6044 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "gym" | belief | - | 0.010213 | 155/4607 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✕FAILanswered "unknown" (world seeded for: final_true_state, last_mentioned_location, last_witnessed_by_anyone)verified-stable | belief | - | 0.011275 | 175/5085 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✕ERRORlength | belief | - | 0.017681 | 161/8000 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "photo studio" | belief | - | 0.000727 | 155/295 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "atrium" | belief | - | 0.000816 | 171/332 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "courtyard" | belief | - | 0.000897 | 180/367 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "atrium" | belief | - | 0.000928 | 166/384 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "conference room" | belief | - | 0.001013 | 177/420 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "loading dock" | belief | - | 0.000783 | 189/313 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "loading dock" | belief | - | 0.000762 | 191/303 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "supply closet" | belief | - | 0.002783 | 168/1227 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "storage room" | belief | - | 0.001610 | 205/685 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "south stairwell" | belief | - | 0.000596 | 201/225 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "utility closet" | belief | - | 0.000801 | 198/319 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "atrium" | belief | - | 0.002278 | 169/997 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "mail room" | belief | - | 0.003262 | 184/1441 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "archive room" | belief | - | 0.001629 | 195/696 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "utility closet" | belief | - | 0.001422 | 196/602 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "lobby" | belief | - | 0.000744 | 207/291 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "reception area" | belief | - | 0.001133 | 228/463 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "storage room" | belief | - | 0.000759 | 224/294 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "east wing" | belief | - | 0.000597 | 226/220 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "greenhouse" | belief | - | 0.000769 | 244/294 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "photo studio" | belief | - | 0.001057 | 230/428 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "wine cellar" | belief | - | 0.000688 | 240/258 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "storage room" | belief | - | 0.001049 | 236/423 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "loading dock" | belief | - | 0.001692 | 247/713 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "meditation room" | belief | - | 0.001197 | 243/489 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "wine cellar" | belief | - | 0.000796 | 171/323 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "supply closet" | belief | - | 0.000875 | 175/358 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "south stairwell" | belief | - | 0.000888 | 157/368 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "loading dock" | belief | - | 0.002133 | 240/915 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "photo studio" | belief | - | 0.002250 | 170/984 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "boiler room" | belief | - | 0.000633 | 233/235 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "mail room" | belief | - | 0.001851 | 248/785 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "courtyard" | belief | - | 0.000820 | 342/295 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "storage room" | belief | - | 0.001030 | 357/387 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "conference room" | belief | - | 0.004258 | 183/1894 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "cafeteria" | belief | - | 0.002756 | 240/1198 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "observation deck" | belief | - | 0.001851 | 235/788 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "archive room" | belief | - | 0.001606 | 343/652 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "garage" | belief | - | 0.000766 | 401/257 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "atrium" | belief | - | 0.001159 | 337/450 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "server room" | belief | - | 0.000793 | 412/267 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "courtyard" | belief | - | 0.001903 | 431/767 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "lobby" | belief | - | 0.000783 | 528/236 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "east wing" | belief | - | 0.000856 | 494/277 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "gym" | belief | - | 0.001925 | 427/778 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "conference room" | belief | - | 0.001633 | 530/622 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "west wing" | belief | - | 0.004646 | 426/2015 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "kitchen" | belief | - | 0.003100 | 480/1300 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "security office" | belief | - | 0.006458 | 310/2865 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "utility closet" | belief | - | 0.004755 | 512/2045 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "greenhouse" | belief | - | 0.000631 | 153/252 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "break room" | belief | - | 0.000738 | 178/295 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "workshop" | belief | - | 0.000724 | 159/293 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "wine cellar" | belief | - | 0.000861 | 159/355 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "kitchen" | belief | - | 0.000881 | 169/362 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "wine cellar" | belief | - | 0.000938 | 159/390 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "archive room" | belief | - | 0.000643 | 173/253 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "server room" | belief | - | 0.000856 | 159/353 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "security office" | belief | - | 0.001224 | 156/521 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "loading dock" | belief | - | 0.001627 | 174/700 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "cafeteria" | belief | - | 0.001788 | 171/774 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "wine cellar" | belief | - | 0.000847 | 168/347 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "basement" | belief | - | 0.000582 | 161/228 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "courtyard" | belief | - | 0.000761 | 157/310 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "observation deck" | belief | - | 0.000761 | 172/307 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "kitchen" | belief | - | 0.000983 | 162/410 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "boiler room" | belief | - | 0.001023 | 162/428 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "workshop" | belief | - | 0.003918 | 175/1741 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "print room" | belief | - | 0.000582 | 165/227 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "wine cellar" | belief | - | 0.000776 | 166/315 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "rooftop terrace" | belief | - | 0.001674 | 176/721 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "conference room" | belief | - | 0.000641 | 168/253 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "east wing" | belief | - | 0.000687 | 177/272 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "west wing" | belief | - | 0.001325 | 172/563 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "rooftop terrace" | belief | - | 0.002691 | 164/1186 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "conference room" | belief | - | 0.000914 | 768/241 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "observation deck" | belief | - | 0.001309 | 757/423 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "security office" | belief | - | 0.005530 | 763/2340 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "greenhouse" | belief | - | 0.006540 | 738/2805 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✕ERRORlength | belief | - | 0.018000 | 800/8000 | full transcript → |