Data to …… trials

Cross-model atlas

Where each model breaks. Pick a query kind and a difficulty dimension; every model with campaign data is binned into the same difficulty buckets so the cells are directly comparable.

Map another model →
6
Models compared
8
Query kinds
79%
7,285/9,195
Campaign pass rate
143
Boundaries fitted
Query kind BeliefCode repairCode traceInteractiveOrderRepository fixStateTool plan
Difficulty dial context lengthcoreferencedistractorsentity statetemporalwitness gapnoop distractorsrelational time
Pass rate by difficulty · State / coreference6 models × 2 buckets · 444 trials

Gemma 3 27B breaks first: its pass rate crosses 50% at coreference = 2.42. Darker red = lower pass rate. Click any cell for the trials behind it.

13COREFERENCE → HARDERGemma 3 27Boverall 56% · n=45google/gemma-3-27b-it - coreference 1: 59.4% pass (n=32)59%n=32google/gemma-3-27b-it - coreference 3: 46.2% pass (n=13)46%n=13Nemotron 3 Super 120Boverall 96% · n=95nemotron-3-super-120b-a12b - coreference 1: 95.9% pass (n=73)96%n=73nemotron-3-super-120b-a12b - coreference 3: 95.5% pass (n=22)95%n=22Nemotron 3 Nano 30Boverall 100% · n=90NVIDIA-Nemotron-3-Nano-30B-A3B - coreference 1: 100.0% pass (n=69)100%n=69NVIDIA-Nemotron-3-Nano-30B-A3B - coreference 3: 100.0% pass (n=21)100%n=21Nemotron 3 Ultra 550Boverall 100% · n=30Nemotron-3-Ultra-550b-a55b - coreference 1: 100.0% pass (n=23)100%n=23Nemotron-3-Ultra-550b-a55b - coreference 3: 100.0% pass (n=7)100%n=7Gemma 4 31B QAToverall 100% · n=124gemma-4-31b-it-qat-w4a16-ct - coreference 1: 100.0% pass (n=97)100%n=97gemma-4-31b-it-qat-w4a16-ct - coreference 3: 100.0% pass (n=27)100%n=27gpt-oss-120boverall 100% · n=60gpt-oss-120b - coreference 1: 100.0% pass (n=46)100%n=46gpt-oss-120b - coreference 3: 100.0% pass (n=14)100%n=14

12 populated cells · 1 fitted frontier(s) in range · hover a cell for exact counts

Degradation curve · model vs. modelsame buckets

The same cells as slopes, for reading two models against each other. Toggle a model off to narrow the comparison.

100%50%0%break13COREFERENCEgoogle/gemma-3-27b-it - coreference 1: 59.4% pass (n=32)google/gemma-3-27b-it - coreference 3: 46.2% pass (n=13)Gemma 3 27Bnemotron-3-super-120b-a12b - coreference 1: 95.9% pass (n=73)nemotron-3-super-120b-a12b - coreference 3: 95.5% pass (n=22)Nemotron 3 Super NVIDIA-Nemotron-3-Nano-30B-A3B - coreference 1: 100.0% pass (n=69)NVIDIA-Nemotron-3-Nano-30B-A3B - coreference 3: 100.0% pass (n=21)Nemotron 3 Nano 3Nemotron-3-Ultra-550b-a55b - coreference 1: 100.0% pass (n=23)Nemotron-3-Ultra-550b-a55b - coreference 3: 100.0% pass (n=7)Nemotron 3 Ultra gemma-4-31b-it-qat-w4a16-ct - coreference 1: 100.0% pass (n=97)gemma-4-31b-it-qat-w4a16-ct - coreference 3: 100.0% pass (n=27)Gemma 4 31B QATgpt-oss-120b - coreference 1: 100.0% pass (n=46)gpt-oss-120b - coreference 3: 100.0% pass (n=14)gpt-oss-120b
Summary · model x query kind32 pairs
modelquery_kindpass ratentotal cost
Nemotron 3 Nano 30Bbelief800/89589.4%927$0.189034
Nemotron 3 Nano 30Binteractive92/11977.3%133$0.774886
Nemotron 3 Nano 30Border156/15898.7%160$0.022909
Nemotron 3 Nano 30Bstate90/90100.0%90$0.014016
Nemotron 3 Nano 30Btool_plan402/61565.4%637$0.348336
Nemotron 3 Ultra 550Bbelief308/31199.0%316$0.815803
Nemotron 3 Ultra 550Binteractive0/00.0%10$0.432233
Nemotron 3 Ultra 550Border80/80100.0%80$0.085632
Nemotron 3 Ultra 550Bstate30/30100.0%30$0.043059
Nemotron 3 Ultra 550Btool_plan260/29189.3%291$1.094840
Gemma 4 31B QATbelief544/89860.6%898$0.028576
Gemma 4 31B QATcode_repair21/6532.3%72$0.036091
Gemma 4 31B QATcode_trace792/81996.7%823$0.201881
Gemma 4 31B QATinteractive352/61857.0%675$1.073585
Gemma 4 31B QATorder199/20995.2%209$0.008847
Gemma 4 31B QATrepo_fix52/18228.6%204$1.966217
Gemma 4 31B QATstate124/124100.0%124$0.004818
Gemma 4 31B QATtool_plan699/86480.9%867$0.170132
Gemma 3 27Bbelief142/30746.3%307$0.007658
Gemma 3 27Bcode_trace183/24574.7%245$0.062598
Gemma 3 27Binteractive104/11292.9%115$0.913870
Gemma 3 27Border79/8098.8%80$0.004680
Gemma 3 27Bstate25/4555.6%45$0.002148
Gemma 3 27Btool_plan219/39755.2%397$0.069666
gpt-oss-120bbelief195/20197.0%202$0.046658
gpt-oss-120border79/8098.8%80$0.015004
gpt-oss-120bstate60/60100.0%60$0.025043
Nemotron 3 Super 120Bbelief435/45196.5%451$0.157160
Nemotron 3 Super 120Binteractive133/13797.1%148$1.080437
Nemotron 3 Super 120Border79/8098.8%80$0.018572
Nemotron 3 Super 120Bstate91/9595.8%95$0.033380
Nemotron 3 Super 120Btool_plan460/53785.7%551$0.515123