Data to …… trials

Run camp-full_map-ultra-002/breaking_point

98% PASS

What this run revealed, then every individual trial as evidence.

56
1 error
Trials
98%
54/55 scored
over trials
Pass rate
1
2%
Failures
$0.2055
Run cost

Run camp-full_map-ultra-002/breaking_point: passed essentially everything (98%) -- this test did not challenge the model.

Of 55 scored trials (belief query kind), the model passed 98%. That's saturation: the difficulty range probed here isn't finding this model's limits. Consider a harder preset (breaking_point or adversarial_belief) or a wider dimension range.

1 additional trial(s) ended in ERROR (infra failures, cost cap, or cancellation) and are excluded from the pass rate above.

1 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

1 belief failure(s) were classified against known shortcuts, but none of them dominate -- the failures don't share one obvious wrong-answer pattern.

Notable failures

  • record d392ecae — asked "Where does Juno believe the narrow folder is? Answer with only the location name (or 'unknown' if they have no belief)."
    expected: the server room
    actual: meditation room

    the model's answer matches no known shortcut -- an unclassified miss

Trials56 trials
Per-trial table (56 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "garage"belief-0.000902167/372full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "meditation room"belief-0.001003176/416full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "rooftop terrace"belief-0.000527174/200full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "east wing"belief-0.000813178/329full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "east wing"belief-0.000590163/231full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "utility closet"belief-0.000805171/327full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "atrium"belief-0.000898164/371full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "conference room"belief-0.010885514/4831full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "storage room"belief-0.005596517/2426full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "utility closet"belief-0.003613521/1524full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "workshop"belief-0.010193493/4521full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "basement"belief-0.008821521/3891full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✕ERRORlengthbelief-0.017848496/8000full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "storage room"belief-0.008451477/3733full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "rooftop terrace"belief-0.001586870/523full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "observation deck"belief-0.001768869/606full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "kitchen"belief-0.001807837/631full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "garage"belief-0.002784852/1072full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "lobby"belief-0.012251830/5380full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "archive room"belief-0.001678835/573full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "boiler room"belief-0.003233855/1275full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "front office"belief-0.0027621225/977full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "gym"belief-0.0026101211/911full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "print room"belief-0.0028861227/1033full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "reception area"belief-0.0022901201/768full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "archive room"belief-0.0022831243/755full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "wine cellar"belief-0.0028751192/1036full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "gym"belief-0.0021271188/697full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "supply closet"belief-0.0024741533/776full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "archive room"belief-0.0026761573/859full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "archive room"belief-0.0023341496/721full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "greenhouse"belief-0.0057711540/2273full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "break room"belief-0.0022861514/695full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "rooftop terrace"belief-0.0025891525/830full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "west wing"belief-0.0022211520/664full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "rooftop terrace"belief-0.0020131800/506full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "workshop"belief-0.0046441799/1702full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "courtyard"belief-0.0039961770/1414full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "photo studio"belief-0.0025861770/773full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "wine cellar"belief-0.0018021765/418full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "greenhouse"belief-0.0032241754/1067full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "server room"belief-0.0028691796/896full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "wine cellar"belief-0.0021892054/528full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "loading dock"belief-0.0029712110/871full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "meditation room"belief-0.0028012051/807full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "south stairwell"belief-0.0017072089/301full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "workshop"belief-0.0020582039/472full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "archive room"belief-0.0097712035/3979full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "conference room"belief-0.0025982121/699full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "reception area"belief-0.0017072323/248full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "supply closet"belief-0.0035692236/1114full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "supply closet"belief-0.0091132267/3627full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "server room"belief-0.0032562274/963full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b?FAILanswered "meditation room"unverifiedbelief-0.0029222201/828full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "wine cellar"belief-0.0030932291/885full transcript →
nvidia/Nemotron-3-Ultra-550b-a55b✓PASSanswered "front office"belief-0.0023882308/561full transcript →