Run camp-full_map-ultra-002/breaking_point
98% PASSWhat this run revealed, then every individual trial as evidence.
56
1 error
Trials
98%
54/55 scored
over trials
Pass rate
1
Failures
$0.2055
Run cost
Run camp-full_map-ultra-002/breaking_point: passed essentially everything (98%) -- this test did not challenge the model.
Of 55 scored trials (belief query kind), the model passed 98%. That's saturation: the difficulty range probed here isn't finding this model's limits. Consider a harder preset (breaking_point or adversarial_belief) or a wider dimension range.
1 additional trial(s) ended in ERROR (infra failures, cost cap, or cancellation) and are excluded from the pass rate above.
1 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).
1 belief failure(s) were classified against known shortcuts, but none of them dominate -- the failures don't share one obvious wrong-answer pattern.
Notable failures
- record d392ecae expected: the server roomactual: meditation room
Trials
Per-trial table
| model | verdict | kind | difficulty | cost ($) | tokens in/out | |
|---|---|---|---|---|---|---|
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "garage" | belief | - | 0.000902 | 167/372 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "meditation room" | belief | - | 0.001003 | 176/416 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "rooftop terrace" | belief | - | 0.000527 | 174/200 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "east wing" | belief | - | 0.000813 | 178/329 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "east wing" | belief | - | 0.000590 | 163/231 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "utility closet" | belief | - | 0.000805 | 171/327 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "atrium" | belief | - | 0.000898 | 164/371 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "conference room" | belief | - | 0.010885 | 514/4831 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "storage room" | belief | - | 0.005596 | 517/2426 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "utility closet" | belief | - | 0.003613 | 521/1524 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "workshop" | belief | - | 0.010193 | 493/4521 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "basement" | belief | - | 0.008821 | 521/3891 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✕ERRORlength | belief | - | 0.017848 | 496/8000 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "storage room" | belief | - | 0.008451 | 477/3733 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "rooftop terrace" | belief | - | 0.001586 | 870/523 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "observation deck" | belief | - | 0.001768 | 869/606 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "kitchen" | belief | - | 0.001807 | 837/631 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "garage" | belief | - | 0.002784 | 852/1072 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "lobby" | belief | - | 0.012251 | 830/5380 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "archive room" | belief | - | 0.001678 | 835/573 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "boiler room" | belief | - | 0.003233 | 855/1275 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "front office" | belief | - | 0.002762 | 1225/977 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "gym" | belief | - | 0.002610 | 1211/911 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "print room" | belief | - | 0.002886 | 1227/1033 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "reception area" | belief | - | 0.002290 | 1201/768 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "archive room" | belief | - | 0.002283 | 1243/755 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "wine cellar" | belief | - | 0.002875 | 1192/1036 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "gym" | belief | - | 0.002127 | 1188/697 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "supply closet" | belief | - | 0.002474 | 1533/776 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "archive room" | belief | - | 0.002676 | 1573/859 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "archive room" | belief | - | 0.002334 | 1496/721 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "greenhouse" | belief | - | 0.005771 | 1540/2273 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "break room" | belief | - | 0.002286 | 1514/695 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "rooftop terrace" | belief | - | 0.002589 | 1525/830 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "west wing" | belief | - | 0.002221 | 1520/664 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "rooftop terrace" | belief | - | 0.002013 | 1800/506 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "workshop" | belief | - | 0.004644 | 1799/1702 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "courtyard" | belief | - | 0.003996 | 1770/1414 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "photo studio" | belief | - | 0.002586 | 1770/773 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "wine cellar" | belief | - | 0.001802 | 1765/418 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "greenhouse" | belief | - | 0.003224 | 1754/1067 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "server room" | belief | - | 0.002869 | 1796/896 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "wine cellar" | belief | - | 0.002189 | 2054/528 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "loading dock" | belief | - | 0.002971 | 2110/871 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "meditation room" | belief | - | 0.002801 | 2051/807 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "south stairwell" | belief | - | 0.001707 | 2089/301 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "workshop" | belief | - | 0.002058 | 2039/472 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "archive room" | belief | - | 0.009771 | 2035/3979 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "conference room" | belief | - | 0.002598 | 2121/699 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "reception area" | belief | - | 0.001707 | 2323/248 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "supply closet" | belief | - | 0.003569 | 2236/1114 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "supply closet" | belief | - | 0.009113 | 2267/3627 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "server room" | belief | - | 0.003256 | 2274/963 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ?FAILanswered "meditation room"unverified | belief | - | 0.002922 | 2201/828 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "wine cellar" | belief | - | 0.003093 | 2291/885 | full transcript → |
| nvidia/Nemotron-3-Ultra-550b-a55b | ✓PASSanswered "front office" | belief | - | 0.002388 | 2308/561 | full transcript → |