Run camp-full_map-gemma-001/state
56% PASSWhat this run revealed, then every individual trial as evidence.
45
Trials
56%
25/45 scored
over trials
Pass rate
20
20 stable / 0 noise
Failures
$0.0021
Run cost
Run camp-full_map-gemma-001/state: 56% pass rate on state queries.
Of 45 scored trials (state query kind), the model passed 56% and failed 44%.
20 failures recorded; all 20 survived verification.
Notable failures
- record d55ca622 expected: the boiler roomactual: West wing
- record 9bc3cedf expected: the east wingactual: mail room
- record eacfcd43 expected: the south stairwellactual: Server room
Trials
Per-trial table
| model | verdict | kind | difficulty | cost ($) | tokens in/out | |
|---|---|---|---|---|---|---|
| google/gemma-3-27b-it | ✓PASSanswered "Utility closet" | state | - | 0.000014 | 128/3 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "Courtyard" | state | - | 0.000013 | 125/3 | full transcript → |
| google/gemma-3-27b-it | ✕FAILanswered "West wing"verified-stable | state | - | 0.000027 | 256/3 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "Photo studio" | state | - | 0.000013 | 125/3 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "Storage room" | state | - | 0.000013 | 125/3 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "Courtyard" | state | - | 0.000013 | 122/3 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "Cafeteria" | state | - | 0.000026 | 247/4 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "Storage room" | state | - | 0.000025 | 244/3 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "Observation deck" | state | - | 0.000026 | 249/3 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "loading dock" | state | - | 0.000038 | 371/3 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "Boiler room" | state | - | 0.000026 | 249/4 | full transcript → |
| google/gemma-3-27b-it | ✕FAILanswered "mail room"verified-stable | state | - | 0.000039 | 376/3 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "Cafeteria" | state | - | 0.000039 | 380/4 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "mail room" | state | - | 0.000038 | 371/3 | full transcript → |
| google/gemma-3-27b-it | ✕FAILanswered "Server room"verified-stable | state | - | 0.000040 | 388/3 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "West wing" | state | - | 0.000050 | 493/3 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "The archive room." | state | - | 0.000051 | 492/5 | full transcript → |
| google/gemma-3-27b-it | ✕FAILanswered "Wine cellar"verified-stable | state | - | 0.000050 | 488/3 | full transcript → |
| google/gemma-3-27b-it | ✕FAILanswered "Security office"verified-stable | state | - | 0.000052 | 511/3 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "Print room" | state | - | 0.000051 | 504/3 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "Storage room" | state | - | 0.000064 | 627/3 | full transcript → |
| google/gemma-3-27b-it | ✕FAILanswered "atrium"verified-stable | state | - | 0.000063 | 617/3 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "rooftop terrace" | state | - | 0.000063 | 617/5 | full transcript → |
| google/gemma-3-27b-it | ✕FAILanswered "wine cellar"verified-stable | state | - | 0.000065 | 641/3 | full transcript → |
| google/gemma-3-27b-it | ✕FAILanswered "Archive room"verified-stable | state | - | 0.000064 | 630/3 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "Photo studio" | state | - | 0.000057 | 562/3 | full transcript → |
| google/gemma-3-27b-it | ✕FAILanswered "north stairwell"verified-stable | state | - | 0.000056 | 546/4 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "print room" | state | - | 0.000056 | 554/3 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "library" | state | - | 0.000056 | 555/2 | full transcript → |
| google/gemma-3-27b-it | ✕FAILanswered "Courtyard"verified-stable | state | - | 0.000057 | 559/3 | full transcript → |
| google/gemma-3-27b-it | ✕FAILanswered "Garage"verified-stable | state | - | 0.000060 | 589/2 | full transcript → |
| google/gemma-3-27b-it | ✕FAILanswered "photo studio"verified-stable | state | - | 0.000060 | 586/3 | full transcript → |
| google/gemma-3-27b-it | ✕FAILanswered "Kitchen"verified-stable | state | - | 0.000059 | 583/2 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "West wing" | state | - | 0.000060 | 591/3 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "mail room" | state | - | 0.000061 | 598/3 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "Workshop" | state | - | 0.000058 | 573/2 | full transcript → |
| google/gemma-3-27b-it | ✕FAILanswered "Storage room"verified-stable | state | - | 0.000058 | 569/3 | full transcript → |
| google/gemma-3-27b-it | ✕FAILanswered "library"verified-stable | state | - | 0.000058 | 576/2 | full transcript → |
| google/gemma-3-27b-it | ✕FAILanswered "server room"verified-stable | state | - | 0.000057 | 561/3 | full transcript → |
| google/gemma-3-27b-it | ✕FAILanswered "break room"verified-stable | state | - | 0.000057 | 561/3 | full transcript → |
| google/gemma-3-27b-it | ✕FAILanswered "Library"verified-stable | state | - | 0.000062 | 610/2 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "workshop" | state | - | 0.000062 | 616/2 | full transcript → |
| google/gemma-3-27b-it | ✓PASSanswered "Storage room" | state | - | 0.000064 | 627/3 | full transcript → |
| google/gemma-3-27b-it | ✕FAILanswered "boiler room"verified-stable | state | - | 0.000064 | 634/3 | full transcript → |
| google/gemma-3-27b-it | ✕FAILanswered "Gym"verified-stable | state | - | 0.000065 | 646/2 | full transcript → |