Coverage
What this instrument tests, and what it honestly doesn't yet: every province and query kind behind the Map, the narrative dials and traps, the analysis machinery, and the gaps.
6
Provinces
10
9 measured
Query kinds
5
Narrative foil traps
4
Known gaps
Provinces and query kinds
| province | kind | what it asks |
|---|---|---|
| Narrative | state | After a long chain of moves, where is each object really? Pure event-log replay, no perspective involved. |
| Narrative | order | When a story is told out of order, which of two events really happened first? |
| Narrative | belief | Where does a person think an object is, based only on what that person saw or was told? |
| Narrative | adversarial_belief | Does belief tracking survive stories searched so the tempting wrong answer is planted right there? |
| Tool planning | tool_plan | Can the model write a whole plan of tool calls that reaches the goal, through precondition chains, decoys and look-alike tools -- and say so when no plan exists? |
| Interactive | interactive | Calling tools one at a time and seeing each result, can the model carry returned values forward, avoid look-alikes and retry a failed call? |
| Code tracing | code_trace | Given a generated program, can the model predict a variable's final value through aliasing, closures, look-alike names and declared dialects? |
| Code repair | code_repair | Given a function with injected bugs, its wrong output and the right one, can the model fix the cause without breaking hidden inputs? |
| Code repair | code_write | From a plain-English description and a few examples, can the model write a function that also gets unseen edge cases right? (Built, not yet run: the spec under-determines some answers.) |
| Repository | repo_fix | In a multi-file repository, can the model follow the calls to the file that holds the cause and fix it there, using coding-harness tools? |
Narrative difficulty dials
| dial | what it varies |
|---|---|
| chain_length | Number of steps in the causal event chain (object moves/transfers) the model must track. |
| scramble | 0.0-1.0: how much the narration order is shuffled relative to true chronology (every sentence still carries its real timestamp). |
| coref_pressure | Number of extra coreference-stressing mentions (pronouns, repeated near-identical entities) injected into the story. |
| distractor_events | Number of plausible-looking but query-irrelevant events involving other entities/objects mixed into the narration. |
| filler_events | Number of narratively neutral padding events that lengthen the story without touching the query's answer. |
| witness_cutoff | 0.0-1.0: how early an entity's last witnessed exposure to the object is cut off relative to the object's later moves -- stresses whether a model over-generalizes 'last witnessed' into 'still watching'. |
Narrative foil traps
| trap | mechanism |
|---|---|
| final_true_state | The object's actual final location -- the most damaging trap for belief queries: what a model says if it collapses belief into reality. |
| last_mentioned_location | The last location named in narration order, regardless of true chronology or who witnessed it. |
| last_witnessed_by_anyone | The destination of the last move ANYONE witnessed, not necessarily the query agent. |
| query_agent_own_last_move | If the query agent themselves moved the object, their own last such move's destination (only applies when the agent moved it). |
| most_frequent_location | The most-mentioned destination across the whole story, trusting mention frequency over recency or truth. |
Analysis machinery
| tool | what it does |
|---|---|
| escalation | Walks a single ray through the full joint difficulty space (all dims at once, base to generator ceiling) to find where a model actually breaks -- see cartographer/schedule/escalate.py. |
| resample-gate | The scheduler's explore/refine waves re-probe weak/ambiguous signals with additional resamples before committing to a boundary, instead of trusting a single trial. |
| ablation | Deterministic, answer-key-preserving prompt transforms over the same latent world, used to causally isolate which surface factor a failure actually depends on -- see cartographer/ablate/ablations.py. |
| fresh-seed confirmation | Re-tests a decompose hypothesis on seeds that were never used during the search that produced it (seed_base >= 10,000 by convention), reporting an effect size + p-value rather than a bare pass/fail -- see cartographer/confirm/confirmation.py. |
Not yet covered
- Instruction-following under explicit constraints
- Quantitative/numeric reasoning
- Long-context retrieval (needle-in-haystack style)
- Image or other non-text inputs