Data to …… trials

Run camp-state-nemotron-super-001

97% PASS

What this run revealed, then every individual trial as evidence.

watch live →compare across models →
30
Trials
97%
29/30 scored
over trials
Pass rate
1
3%
Failures
$0.0122
Run cost

Run camp-state-nemotron-super-001: 97% pass rate on state queries.

Of 30 scored trials (state query kind), the model passed 97% and failed 3%.

1 failure(s) recorded; failures not yet verified -- use "Verify failures" on this page (or scripts/verify_past_failures.py).

Notable failures

  • record 8ac17ceb — asked "Where is the heavy wrench now? Answer with only the location name."
    expected: (unknown)
    actual: wine cellar

    answer did not match the verified ground truth

Trials30 trials
Per-trial table (30 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "storage room"state-0.000268133/572full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "courtyard"state-0.000130134/265full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "courtyard"state-0.000066128/124full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "photo studio"state-0.000110131/222full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "utility closet"state-0.000140140/286full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "boiler room"state-0.000148265/282full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "cafeteria"state-0.000236256/480full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "storage room"state-0.000209272/417full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "observation deck"state-0.000361268/754full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "boiler room"state-0.000210255/421full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "loading dock"state-0.000616383/1300full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "east wing"state-0.000403406/824full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "cafeteria"state-0.000484405/1004full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "mail room"state-0.000432401/889full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "south stairwell"state-0.000088398/125full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "archive room"state-0.000358504/705full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "west wing"state-0.000569525/1171full transcript →
nvidia/nemotron-3-super-120b-a12b?FAILanswered "wine cellar"unverifiedstate-0.000105519/141full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "mail room"state-0.000423518/847full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "print room"state-0.000487519/991full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "storage room"state-0.000697641/1435full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "rooftop terrace"state-0.000592663/1198full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "photo studio"state-0.000605629/1233full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "photo studio"state-0.000556657/1119full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "east wing"state-0.000599649/1216full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "cafeteria"state-0.000712679/1461full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "photo studio"state-0.000550682/1102full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "print room"state-0.000582657/1176full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "supply closet"state-0.000861667/1795full transcript →
nvidia/nemotron-3-super-120b-a12b✓PASSanswered "south stairwell"state-0.000610625/1245full transcript →