Data to …… trials

Run camp-belief-nemotron-nano-001

93% PASS

What this run revealed, then every individual trial as evidence.

105
Trials
93%
98/105 scored
over trials
Pass rate
7
7%
Failures
$0.0139
Run cost

Run camp-belief-nemotron-nano-001: 93% pass rate on belief queries.

Of 105 scored trials (belief query kind), the model passed 93% and failed 7%.

7 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Diffuse weakness (below 90% but no outright flip) on: entity_state (pass rate 84% at the high end).

7 belief FAIL(s) were recorded, but they predate the generator-hardening fix and can't be safely re-classified against regenerated ground truth.

Notable failures

  • record 464c525b — asked "Where does Victor believe the heavy box is? Answer with only the location name (or 'unknown' if they have no belief)."
    expected: (unknown)
    actual: loading dock

    answer did not match the verified ground truth

  • record d3b6c6ce — asked "Where does Omar believe the shiny folder is? Answer with only the location name (or 'unknown' if they have no belief)."
    expected: (unknown)
    actual: archive room

    answer did not match the verified ground truth

  • record 3ef94581 — asked "Where does Juno believe the worn tablet is? Answer with only the location name (or 'unknown' if they have no belief)."
    expected: (unknown)
    actual: lobby

    answer did not match the verified ground truth

Trials105 trials
Per-trial table (105 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "front office"belief-0.000168158/802full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "storage room"belief-0.000096165/441full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "break room"belief-0.000128143/604full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "kitchen"belief-0.000034140/137full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "photo studio"belief-0.000033152/127full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "storage room"belief-0.000074225/314full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "loading dock"unverifiedbelief-0.000089223/388full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "front office"belief-0.000234226/1112full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "utility closet"belief-0.000113224/509full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "library"belief-0.000071192/305full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "archive room"unverifiedbelief-0.000181280/835full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "reception area"belief-0.000218237/1033full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "south stairwell"belief-0.000276281/1309full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "lobby"unverifiedbelief-0.000202268/944full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "west wing"belief-0.000215281/1007full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "atrium"belief-0.000198323/911full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "archive room"belief-0.000199311/917full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "supply closet"belief-0.000124337/538full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "courtyard"belief-0.000174327/790full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "east wing"belief-0.000245347/1136full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "mail room"belief-0.000635389/3078full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "front office"belief-0.000268396/1243full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "library"unverifiedbelief-0.000236409/1080full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "basement"belief-0.000201401/906full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "south stairwell"belief-0.000253399/1167full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "kitchen"belief-0.000072153/323full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "greenhouse"belief-0.000090169/409full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "greenhouse"belief-0.000063145/279full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "supply closet"belief-0.000067163/295full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "server room"belief-0.000162165/768full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "workshop"belief-0.000087158/397full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "gym"belief-0.000136155/639full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "utility closet"belief-0.000073150/330full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "rooftop terrace"belief-0.000123160/577full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "utility closet"belief-0.000238161/1151full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "lobby"belief-0.000154157/732full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "utility closet"belief-0.000125164/583full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "photo studio"belief-0.000232161/1122full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "kitchen"belief-0.000067162/293full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "supply closet"belief-0.000123148/579full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "west wing"belief-0.000123145/580full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "front office"belief-0.000140158/661full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "storage room"belief-0.000046165/188full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "photo studio"belief-0.000070165/307full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "reception area"belief-0.000037165/143full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "garage"belief-0.000085140/388full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "north stairwell"belief-0.000201144/967full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "west wing"unverifiedbelief-0.000227162/1092full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "north stairwell"belief-0.000072144/324full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "greenhouse"belief-0.000094148/432full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "observation deck"belief-0.000031143/121full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "gym"belief-0.000088148/405full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "lobby"belief-0.000044146/182full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "rooftop terrace"belief-0.000129162/603full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "observation deck"unverifiedbelief-0.000150158/708full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "east wing"belief-0.000062156/273full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "utility closet"belief-0.000164175/775full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B?FAILanswered "security office"unverifiedbelief-0.000289174/1402full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "garage"belief-0.000118167/549full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "supply closet"belief-0.000192176/915full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "meditation room"belief-0.000073182/319full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "mail room"belief-0.000065180/280full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "reception area"belief-0.000062183/264full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "reception area"belief-0.000088185/395full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "mail room"belief-0.000054176/227full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "north stairwell"belief-0.000067218/282full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "basement"belief-0.000094213/415full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "archive room"belief-0.000098204/440full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "west wing"belief-0.000064211/265full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "north stairwell"belief-0.000065223/267full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "garage"belief-0.000076225/326full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "lobby"belief-0.000142233/651full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "courtyard"belief-0.000057229/227full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "library"belief-0.000076228/325full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "supply closet"belief-0.000084224/363full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "courtyard"belief-0.000065162/285full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "front office"belief-0.000087160/397full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "south stairwell"belief-0.000053173/221full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "utility closet"belief-0.000064153/281full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "west wing"belief-0.000086159/390full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "courtyard"belief-0.000105232/466full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "workshop"belief-0.000047232/179full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "courtyard"belief-0.000348216/1684full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "cafeteria"belief-0.000063218/260full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "meditation room"belief-0.000062221/253full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "gym"belief-0.000087312/359full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "east wing"belief-0.000150310/673full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "west wing"belief-0.000047316/155full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "cafeteria"belief-0.000121342/517full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "workshop"belief-0.000126327/548full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "break room"belief-0.000101423/400full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "rooftop terrace"belief-0.000190427/844full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "basement"belief-0.000049380/151full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "utility closet"belief-0.000058432/180full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "west wing"belief-0.000125400/524full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "kitchen"belief-0.000080470/284full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "greenhouse"belief-0.000079498/271full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "utility closet"belief-0.000056492/159full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "boiler room"belief-0.000224479/998full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "rooftop terrace"belief-0.000065490/204full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "security office"belief-0.000260823/1094full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "greenhouse"belief-0.000248797/1041full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "workshop"belief-0.000218766/897full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "utility closet"belief-0.000300834/1294full transcript →
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B✓PASSanswered "front office"belief-0.000324785/1426full transcript →