Data to …… trials

Coverage

What this instrument tests, and what it honestly doesn't yet: every province and query kind behind the Map, the narrative dials and traps, the analysis machinery, and the gaps.

6
Provinces
10
9 measured
Query kinds
5
Narrative foil traps
4
Known gaps
Provinces and query kinds 10 kinds in 6 provinces
provincekindwhat it asks
NarrativestateAfter a long chain of moves, where is each object really? Pure event-log replay, no perspective involved.
NarrativeorderWhen a story is told out of order, which of two events really happened first?
NarrativebeliefWhere does a person think an object is, based only on what that person saw or was told?
Narrativeadversarial_beliefDoes belief tracking survive stories searched so the tempting wrong answer is planted right there?
Tool planningtool_planCan the model write a whole plan of tool calls that reaches the goal, through precondition chains, decoys and look-alike tools -- and say so when no plan exists?
InteractiveinteractiveCalling tools one at a time and seeing each result, can the model carry returned values forward, avoid look-alikes and retry a failed call?
Code tracingcode_traceGiven a generated program, can the model predict a variable's final value through aliasing, closures, look-alike names and declared dialects?
Code repaircode_repairGiven a function with injected bugs, its wrong output and the right one, can the model fix the cause without breaking hidden inputs?
Code repaircode_writeFrom a plain-English description and a few examples, can the model write a function that also gets unseen edge cases right? (Built, not yet run: the spec under-determines some answers.)
Repositoryrepo_fixIn a multi-file repository, can the model follow the calls to the file that holds the cause and fix it there, using coding-harness tools?
Narrative difficulty dials 6
dialwhat it varies
chain_lengthNumber of steps in the causal event chain (object moves/transfers) the model must track.
scramble0.0-1.0: how much the narration order is shuffled relative to true chronology (every sentence still carries its real timestamp).
coref_pressureNumber of extra coreference-stressing mentions (pronouns, repeated near-identical entities) injected into the story.
distractor_eventsNumber of plausible-looking but query-irrelevant events involving other entities/objects mixed into the narration.
filler_eventsNumber of narratively neutral padding events that lengthen the story without touching the query's answer.
witness_cutoff0.0-1.0: how early an entity's last witnessed exposure to the object is cut off relative to the object's later moves -- stresses whether a model over-generalizes 'last witnessed' into 'still watching'.
Narrative foil traps 5
trapmechanism
final_true_stateThe object's actual final location -- the most damaging trap for belief queries: what a model says if it collapses belief into reality.
last_mentioned_locationThe last location named in narration order, regardless of true chronology or who witnessed it.
last_witnessed_by_anyoneThe destination of the last move ANYONE witnessed, not necessarily the query agent.
query_agent_own_last_moveIf the query agent themselves moved the object, their own last such move's destination (only applies when the agent moved it).
most_frequent_locationThe most-mentioned destination across the whole story, trusting mention frequency over recency or truth.

The plausible-but-wrong answers a shallow reader gives on a belief query; adversarial seed search picks the worlds that pit a trap most sharply against the truth. Every other province has its own dials and traps, listed per campaign in docs/campaign-catalog.md.

Analysis machinery 4
toolwhat it does
escalationWalks a single ray through the full joint difficulty space (all dims at once, base to generator ceiling) to find where a model actually breaks -- see cartographer/schedule/escalate.py.
resample-gateThe scheduler's explore/refine waves re-probe weak/ambiguous signals with additional resamples before committing to a boundary, instead of trusting a single trial.
ablationDeterministic, answer-key-preserving prompt transforms over the same latent world, used to causally isolate which surface factor a failure actually depends on -- see cartographer/ablate/ablations.py.
fresh-seed confirmationRe-tests a decompose hypothesis on seeds that were never used during the search that produced it (seed_base >= 10,000 by convention), reporting an effect size + p-value rather than a bare pass/fail -- see cartographer/confirm/confirmation.py.
Not yet covered 4 known gaps

Capabilities this instrument does not probe today.

  • Instruction-following under explicit constraints
  • Quantitative/numeric reasoning
  • Long-context retrieval (needle-in-haystack style)
  • Image or other non-text inputs