Data to …… trials

Cross-model atlas

Where each model breaks. Pick a query kind and a difficulty dimension; every model with campaign data is binned into the same difficulty buckets so the cells are directly comparable.

Map another model →
6
Models compared
8
Query kinds
79%
7,285/9,195
Campaign pass rate
143
Boundaries fitted
Query kind BeliefCode repairCode traceInteractiveOrderRepository fixStateTool plan
Difficulty dial context lengthcoreferencedistractorsentity statetemporalwitness gapfoil divergencenoop distractorsrelational timeconditionevidence spanevidence turns
Pass rate by difficulty · Belief / coreference6 models × 4 buckets · 3,101 trials

Gemma 3 27B breaks first: its pass rate crosses 50% at coreference = 0. Darker red = lower pass rate. Click any cell for the trials behind it.

0–11–33–44–37COREFERENCE → HARDERGemma 3 27Boverall 46% · n=307google/gemma-3-27b-it - coreference 0–1: 46.4% pass (n=196)46%n=196google/gemma-3-27b-it - coreference 1–3: 48.6% pass (n=72)49%n=72google/gemma-3-27b-it - coreference 3–4: 46.2% pass (n=13)46%n=13google/gemma-3-27b-it - coreference 4–37: 38.5% pass (n=26)38%n=26google/gemma-3-27b-it: fitted frontier at coreference=0Gemma 4 31B QAToverall 61% · n=898gemma-4-31b-it-qat-w4a16-ct - coreference 0–1: 59.8% pass (n=522)60%n=522gemma-4-31b-it-qat-w4a16-ct - coreference 1–3: 59.2% pass (n=228)59%n=228gemma-4-31b-it-qat-w4a16-ct - coreference 3–4: 72.2% pass (n=18)72%n=18gemma-4-31b-it-qat-w4a16-ct - coreference 4–37: 64.6% pass (n=130)65%n=130gemma-4-31b-it-qat-w4a16-ct: fitted frontier at coreference=7.50Nemotron 3 Nano 30Boverall 89% · n=895NVIDIA-Nemotron-3-Nano-30B-A3B - coreference 0–1: 89.3% pass (n=531)89%n=531NVIDIA-Nemotron-3-Nano-30B-A3B - coreference 1–3: 88.0% pass (n=208)88%n=208NVIDIA-Nemotron-3-Nano-30B-A3B - coreference 3–4: 91.3% pass (n=23)91%n=23NVIDIA-Nemotron-3-Nano-30B-A3B - coreference 4–37: 91.7% pass (n=133)92%n=133NVIDIA-Nemotron-3-Nano-30B-A3B: fitted frontier at coreference=27.75Nemotron 3 Super 120Boverall 96% · n=451nemotron-3-super-120b-a12b - coreference 0–1: 96.5% pass (n=259)97%n=259nemotron-3-super-120b-a12b - coreference 1–3: 94.9% pass (n=98)95%n=98nemotron-3-super-120b-a12b - coreference 3–4: 92.3% pass (n=13)92%n=13nemotron-3-super-120b-a12b - coreference 4–37: 98.8% pass (n=81)99%n=81gpt-oss-120boverall 97% · n=201gpt-oss-120b - coreference 0–1: 97.0% pass (n=132)97%n=132gpt-oss-120b - coreference 1–3: 98.1% pass (n=52)98%n=52gpt-oss-120b - coreference 3–4: 100.0% pass (n=2)100%n=2gpt-oss-120b - coreference 4–37: 93.3% pass (n=15)93%n=15gpt-oss-120b: fitted frontier at coreference=21.88Nemotron 3 Ultra 550Boverall 99% · n=311Nemotron-3-Ultra-550b-a55b - coreference 0–1: 99.4% pass (n=168)99%n=168Nemotron-3-Ultra-550b-a55b - coreference 1–3: 98.5% pass (n=67)99%n=67Nemotron-3-Ultra-550b-a55b - coreference 3–4: 100.0% pass (n=8)100%n=8Nemotron-3-Ultra-550b-a55b - coreference 4–37: 98.5% pass (n=68)99%n=68

24 populated cells · 4 fitted frontier(s) in range · hover a cell for exact counts

Degradation curve · model vs. modelsame buckets

The same cells as slopes, for reading two models against each other. Toggle a model off to narrow the comparison.

100%50%0%break0–11–33–44–37COREFERENCEgoogle/gemma-3-27b-it - coreference 0–1: 46.4% pass (n=196)google/gemma-3-27b-it - coreference 1–3: 48.6% pass (n=72)google/gemma-3-27b-it - coreference 3–4: 46.2% pass (n=13)google/gemma-3-27b-it - coreference 4–37: 38.5% pass (n=26)Gemma 3 27Bfrontier 0gemma-4-31b-it-qat-w4a16-ct - coreference 0–1: 59.8% pass (n=522)gemma-4-31b-it-qat-w4a16-ct - coreference 1–3: 59.2% pass (n=228)gemma-4-31b-it-qat-w4a16-ct - coreference 3–4: 72.2% pass (n=18)gemma-4-31b-it-qat-w4a16-ct - coreference 4–37: 64.6% pass (n=130)Gemma 4 31B QATNVIDIA-Nemotron-3-Nano-30B-A3B - coreference 0–1: 89.3% pass (n=531)NVIDIA-Nemotron-3-Nano-30B-A3B - coreference 1–3: 88.0% pass (n=208)NVIDIA-Nemotron-3-Nano-30B-A3B - coreference 3–4: 91.3% pass (n=23)NVIDIA-Nemotron-3-Nano-30B-A3B - coreference 4–37: 91.7% pass (n=133)Nemotron 3 Nano 3nemotron-3-super-120b-a12b - coreference 0–1: 96.5% pass (n=259)nemotron-3-super-120b-a12b - coreference 1–3: 94.9% pass (n=98)nemotron-3-super-120b-a12b - coreference 3–4: 92.3% pass (n=13)nemotron-3-super-120b-a12b - coreference 4–37: 98.8% pass (n=81)Nemotron 3 Super gpt-oss-120b - coreference 0–1: 97.0% pass (n=132)gpt-oss-120b - coreference 1–3: 98.1% pass (n=52)gpt-oss-120b - coreference 3–4: 100.0% pass (n=2)gpt-oss-120b - coreference 4–37: 93.3% pass (n=15)gpt-oss-120bNemotron-3-Ultra-550b-a55b - coreference 0–1: 99.4% pass (n=168)Nemotron-3-Ultra-550b-a55b - coreference 1–3: 98.5% pass (n=67)Nemotron-3-Ultra-550b-a55b - coreference 3–4: 100.0% pass (n=8)Nemotron-3-Ultra-550b-a55b - coreference 4–37: 98.5% pass (n=68)Nemotron 3 Ultra
Summary · model x query kind32 pairs
modelquery_kindpass ratentotal cost
Nemotron 3 Nano 30Bbelief800/89589.4%927$0.189034
Nemotron 3 Nano 30Binteractive92/11977.3%133$0.774886
Nemotron 3 Nano 30Border156/15898.7%160$0.022909
Nemotron 3 Nano 30Bstate90/90100.0%90$0.014016
Nemotron 3 Nano 30Btool_plan402/61565.4%637$0.348336
Nemotron 3 Ultra 550Bbelief308/31199.0%316$0.815803
Nemotron 3 Ultra 550Binteractive0/00.0%10$0.432233
Nemotron 3 Ultra 550Border80/80100.0%80$0.085632
Nemotron 3 Ultra 550Bstate30/30100.0%30$0.043059
Nemotron 3 Ultra 550Btool_plan260/29189.3%291$1.094840
Gemma 4 31B QATbelief544/89860.6%898$0.028576
Gemma 4 31B QATcode_repair21/6532.3%72$0.036091
Gemma 4 31B QATcode_trace792/81996.7%823$0.201881
Gemma 4 31B QATinteractive352/61857.0%675$1.073585
Gemma 4 31B QATorder199/20995.2%209$0.008847
Gemma 4 31B QATrepo_fix52/18228.6%204$1.966217
Gemma 4 31B QATstate124/124100.0%124$0.004818
Gemma 4 31B QATtool_plan699/86480.9%867$0.170132
Gemma 3 27Bbelief142/30746.3%307$0.007658
Gemma 3 27Bcode_trace183/24574.7%245$0.062598
Gemma 3 27Binteractive104/11292.9%115$0.913870
Gemma 3 27Border79/8098.8%80$0.004680
Gemma 3 27Bstate25/4555.6%45$0.002148
Gemma 3 27Btool_plan219/39755.2%397$0.069666
gpt-oss-120bbelief195/20197.0%202$0.046658
gpt-oss-120border79/8098.8%80$0.015004
gpt-oss-120bstate60/60100.0%60$0.025043
Nemotron 3 Super 120Bbelief435/45196.5%451$0.157160
Nemotron 3 Super 120Binteractive133/13797.1%148$1.080437
Nemotron 3 Super 120Border79/8098.8%80$0.018572
Nemotron 3 Super 120Bstate91/9595.8%95$0.033380
Nemotron 3 Super 120Btool_plan460/53785.7%551$0.515123