Data to …… trials

Run camp-full_map-gemma-001/state

56% PASS

What this run revealed, then every individual trial as evidence.

45
Trials
56%
25/45 scored
over trials
Pass rate
20
20 stable / 0 noise
44%
Failures
$0.0021
Run cost

Run camp-full_map-gemma-001/state: 56% pass rate on state queries.

Of 45 scored trials (state query kind), the model passed 56% and failed 44%.

20 failures recorded; all 20 survived verification.

Notable failures

  • record d55ca622 — asked "Where is the small box now? Answer with only the location name."
    expected: the boiler room
    actual: West wing

    the model's tracked object state diverged from the simulated ground truth

  • record 9bc3cedf — asked "Where is the small wrench now? Answer with only the location name."
    expected: the east wing
    actual: mail room

    the model's tracked object state diverged from the simulated ground truth

  • record eacfcd43 — asked "Where is the heavy basket now? Answer with only the location name."
    expected: the south stairwell
    actual: Server room

    the model's tracked object state diverged from the simulated ground truth

Trials45 trials
Per-trial table (45 trials -- prompt, reasoning, verdict, answer key)
modelverdictkinddifficultycost ($)tokens in/out
google/gemma-3-27b-it✓PASSanswered "Utility closet"state-0.000014128/3full transcript →
google/gemma-3-27b-it✓PASSanswered "Courtyard"state-0.000013125/3full transcript →
google/gemma-3-27b-it✕FAILanswered "West wing"verified-stablestate-0.000027256/3full transcript →
google/gemma-3-27b-it✓PASSanswered "Photo studio"state-0.000013125/3full transcript →
google/gemma-3-27b-it✓PASSanswered "Storage room"state-0.000013125/3full transcript →
google/gemma-3-27b-it✓PASSanswered "Courtyard"state-0.000013122/3full transcript →
google/gemma-3-27b-it✓PASSanswered "Cafeteria"state-0.000026247/4full transcript →
google/gemma-3-27b-it✓PASSanswered "Storage room"state-0.000025244/3full transcript →
google/gemma-3-27b-it✓PASSanswered "Observation deck"state-0.000026249/3full transcript →
google/gemma-3-27b-it✓PASSanswered "loading dock"state-0.000038371/3full transcript →
google/gemma-3-27b-it✓PASSanswered "Boiler room"state-0.000026249/4full transcript →
google/gemma-3-27b-it✕FAILanswered "mail room"verified-stablestate-0.000039376/3full transcript →
google/gemma-3-27b-it✓PASSanswered "Cafeteria"state-0.000039380/4full transcript →
google/gemma-3-27b-it✓PASSanswered "mail room"state-0.000038371/3full transcript →
google/gemma-3-27b-it✕FAILanswered "Server room"verified-stablestate-0.000040388/3full transcript →
google/gemma-3-27b-it✓PASSanswered "West wing"state-0.000050493/3full transcript →
google/gemma-3-27b-it✓PASSanswered "The archive room."state-0.000051492/5full transcript →
google/gemma-3-27b-it✕FAILanswered "Wine cellar"verified-stablestate-0.000050488/3full transcript →
google/gemma-3-27b-it✕FAILanswered "Security office"verified-stablestate-0.000052511/3full transcript →
google/gemma-3-27b-it✓PASSanswered "Print room"state-0.000051504/3full transcript →
google/gemma-3-27b-it✓PASSanswered "Storage room"state-0.000064627/3full transcript →
google/gemma-3-27b-it✕FAILanswered "atrium"verified-stablestate-0.000063617/3full transcript →
google/gemma-3-27b-it✓PASSanswered "rooftop terrace"state-0.000063617/5full transcript →
google/gemma-3-27b-it✕FAILanswered "wine cellar"verified-stablestate-0.000065641/3full transcript →
google/gemma-3-27b-it✕FAILanswered "Archive room"verified-stablestate-0.000064630/3full transcript →
google/gemma-3-27b-it✓PASSanswered "Photo studio"state-0.000057562/3full transcript →
google/gemma-3-27b-it✕FAILanswered "north stairwell"verified-stablestate-0.000056546/4full transcript →
google/gemma-3-27b-it✓PASSanswered "print room"state-0.000056554/3full transcript →
google/gemma-3-27b-it✓PASSanswered "library"state-0.000056555/2full transcript →
google/gemma-3-27b-it✕FAILanswered "Courtyard"verified-stablestate-0.000057559/3full transcript →
google/gemma-3-27b-it✕FAILanswered "Garage"verified-stablestate-0.000060589/2full transcript →
google/gemma-3-27b-it✕FAILanswered "photo studio"verified-stablestate-0.000060586/3full transcript →
google/gemma-3-27b-it✕FAILanswered "Kitchen"verified-stablestate-0.000059583/2full transcript →
google/gemma-3-27b-it✓PASSanswered "West wing"state-0.000060591/3full transcript →
google/gemma-3-27b-it✓PASSanswered "mail room"state-0.000061598/3full transcript →
google/gemma-3-27b-it✓PASSanswered "Workshop"state-0.000058573/2full transcript →
google/gemma-3-27b-it✕FAILanswered "Storage room"verified-stablestate-0.000058569/3full transcript →
google/gemma-3-27b-it✕FAILanswered "library"verified-stablestate-0.000058576/2full transcript →
google/gemma-3-27b-it✕FAILanswered "server room"verified-stablestate-0.000057561/3full transcript →
google/gemma-3-27b-it✕FAILanswered "break room"verified-stablestate-0.000057561/3full transcript →
google/gemma-3-27b-it✕FAILanswered "Library"verified-stablestate-0.000062610/2full transcript →
google/gemma-3-27b-it✓PASSanswered "workshop"state-0.000062616/2full transcript →
google/gemma-3-27b-it✓PASSanswered "Storage room"state-0.000064627/3full transcript →
google/gemma-3-27b-it✕FAILanswered "boiler room"verified-stablestate-0.000064634/3full transcript →
google/gemma-3-27b-it✕FAILanswered "Gym"verified-stablestate-0.000065646/2full transcript →