Data to …… trials

Cross-model atlas

Where each model breaks. Pick a query kind and a difficulty dimension; every model with campaign data is binned into the same difficulty buckets so the cells are directly comparable.

Map another model →
6
Models compared
8
Query kinds
79%
7,285/9,195
Campaign pass rate
143
Boundaries fitted
Query kind BeliefCode repairCode traceInteractiveOrderRepository fixStateTool plan
Difficulty dial context lengthcoreferencedistractorsentity statetemporalwitness gapnoop distractorsrelational time
Pass rate by difficulty · State / context length6 models × 5 buckets · 444 trials

Gemma 3 27B breaks first: its pass rate crosses 50% at context length = 28.40. Darker red = lower pass rate. Click any cell for the trials behind it.

6–1313–1414–2020–2727–34CONTEXT LENGTH → HARDERGemma 3 27Boverall 56% · n=45google/gemma-3-27b-it - context_length 6–13: 90.0% pass (n=10)90%n=10—google/gemma-3-27b-it - context_length 14–20: 60.0% pass (n=5)60%n=5google/gemma-3-27b-it - context_length 20–27: 60.0% pass (n=5)60%n=5google/gemma-3-27b-it - context_length 27–34: 40.0% pass (n=25)40%n=25google/gemma-3-27b-it: fitted frontier at context_length=28.40Nemotron 3 Super 120Boverall 96% · n=95nemotron-3-super-120b-a12b - context_length 6–13: 100.0% pass (n=38)100%n=38—nemotron-3-super-120b-a12b - context_length 14–20: 93.3% pass (n=15)93%n=15nemotron-3-super-120b-a12b - context_length 20–27: 83.3% pass (n=18)83%n=18nemotron-3-super-120b-a12b - context_length 27–34: 100.0% pass (n=24)100%n=24Nemotron 3 Nano 30Boverall 100% · n=90NVIDIA-Nemotron-3-Nano-30B-A3B - context_length 6–13: 100.0% pass (n=30)100%n=30—NVIDIA-Nemotron-3-Nano-30B-A3B - context_length 14–20: 100.0% pass (n=15)100%n=15NVIDIA-Nemotron-3-Nano-30B-A3B - context_length 20–27: 100.0% pass (n=15)100%n=15NVIDIA-Nemotron-3-Nano-30B-A3B - context_length 27–34: 100.0% pass (n=30)100%n=30Nemotron 3 Ultra 550Boverall 100% · n=30Nemotron-3-Ultra-550b-a55b - context_length 6–13: 100.0% pass (n=10)100%n=10—Nemotron-3-Ultra-550b-a55b - context_length 14–20: 100.0% pass (n=5)100%n=5Nemotron-3-Ultra-550b-a55b - context_length 20–27: 100.0% pass (n=5)100%n=5Nemotron-3-Ultra-550b-a55b - context_length 27–34: 100.0% pass (n=10)100%n=10Gemma 4 31B QAToverall 100% · n=124gemma-4-31b-it-qat-w4a16-ct - context_length 6–13: 100.0% pass (n=36)100%n=36gemma-4-31b-it-qat-w4a16-ct - context_length 13–14: 100.0% pass (n=8)100%n=8gemma-4-31b-it-qat-w4a16-ct - context_length 14–20: 100.0% pass (n=50)100%n=50gemma-4-31b-it-qat-w4a16-ct - context_length 20–27: 100.0% pass (n=10)100%n=10gemma-4-31b-it-qat-w4a16-ct - context_length 27–34: 100.0% pass (n=20)100%n=20gpt-oss-120boverall 100% · n=60gpt-oss-120b - context_length 6–13: 100.0% pass (n=20)100%n=20—gpt-oss-120b - context_length 14–20: 100.0% pass (n=10)100%n=10gpt-oss-120b - context_length 20–27: 100.0% pass (n=10)100%n=10gpt-oss-120b - context_length 27–34: 100.0% pass (n=20)100%n=20

25 populated cells · 1 fitted frontier(s) in range · hover a cell for exact counts

Degradation curve · model vs. modelsame buckets

The same cells as slopes, for reading two models against each other. Toggle a model off to narrow the comparison.

100%50%0%break6–1313–1414–2020–2727–34CONTEXT LENGTHgoogle/gemma-3-27b-it - context_length 6–13: 90.0% pass (n=10)google/gemma-3-27b-it - context_length 14–20: 60.0% pass (n=5)google/gemma-3-27b-it - context_length 20–27: 60.0% pass (n=5)google/gemma-3-27b-it - context_length 27–34: 40.0% pass (n=25)Gemma 3 27Bfrontier 28.40nemotron-3-super-120b-a12b - context_length 6–13: 100.0% pass (n=38)nemotron-3-super-120b-a12b - context_length 14–20: 93.3% pass (n=15)nemotron-3-super-120b-a12b - context_length 20–27: 83.3% pass (n=18)nemotron-3-super-120b-a12b - context_length 27–34: 100.0% pass (n=24)Nemotron 3 Super NVIDIA-Nemotron-3-Nano-30B-A3B - context_length 6–13: 100.0% pass (n=30)NVIDIA-Nemotron-3-Nano-30B-A3B - context_length 14–20: 100.0% pass (n=15)NVIDIA-Nemotron-3-Nano-30B-A3B - context_length 20–27: 100.0% pass (n=15)NVIDIA-Nemotron-3-Nano-30B-A3B - context_length 27–34: 100.0% pass (n=30)Nemotron 3 Nano 3Nemotron-3-Ultra-550b-a55b - context_length 6–13: 100.0% pass (n=10)Nemotron-3-Ultra-550b-a55b - context_length 14–20: 100.0% pass (n=5)Nemotron-3-Ultra-550b-a55b - context_length 20–27: 100.0% pass (n=5)Nemotron-3-Ultra-550b-a55b - context_length 27–34: 100.0% pass (n=10)Nemotron 3 Ultra gemma-4-31b-it-qat-w4a16-ct - context_length 6–13: 100.0% pass (n=36)gemma-4-31b-it-qat-w4a16-ct - context_length 13–14: 100.0% pass (n=8)gemma-4-31b-it-qat-w4a16-ct - context_length 14–20: 100.0% pass (n=50)gemma-4-31b-it-qat-w4a16-ct - context_length 20–27: 100.0% pass (n=10)gemma-4-31b-it-qat-w4a16-ct - context_length 27–34: 100.0% pass (n=20)Gemma 4 31B QATgpt-oss-120b - context_length 6–13: 100.0% pass (n=20)gpt-oss-120b - context_length 14–20: 100.0% pass (n=10)gpt-oss-120b - context_length 20–27: 100.0% pass (n=10)gpt-oss-120b - context_length 27–34: 100.0% pass (n=20)gpt-oss-120b
Summary · model x query kind32 pairs
modelquery_kindpass ratentotal cost
Nemotron 3 Nano 30Bbelief800/89589.4%927$0.189034
Nemotron 3 Nano 30Binteractive92/11977.3%133$0.774886
Nemotron 3 Nano 30Border156/15898.7%160$0.022909
Nemotron 3 Nano 30Bstate90/90100.0%90$0.014016
Nemotron 3 Nano 30Btool_plan402/61565.4%637$0.348336
Nemotron 3 Ultra 550Bbelief308/31199.0%316$0.815803
Nemotron 3 Ultra 550Binteractive0/00.0%10$0.432233
Nemotron 3 Ultra 550Border80/80100.0%80$0.085632
Nemotron 3 Ultra 550Bstate30/30100.0%30$0.043059
Nemotron 3 Ultra 550Btool_plan260/29189.3%291$1.094840
Gemma 4 31B QATbelief544/89860.6%898$0.028576
Gemma 4 31B QATcode_repair21/6532.3%72$0.036091
Gemma 4 31B QATcode_trace792/81996.7%823$0.201881
Gemma 4 31B QATinteractive352/61857.0%675$1.073585
Gemma 4 31B QATorder199/20995.2%209$0.008847
Gemma 4 31B QATrepo_fix52/18228.6%204$1.966217
Gemma 4 31B QATstate124/124100.0%124$0.004818
Gemma 4 31B QATtool_plan699/86480.9%867$0.170132
Gemma 3 27Bbelief142/30746.3%307$0.007658
Gemma 3 27Bcode_trace183/24574.7%245$0.062598
Gemma 3 27Binteractive104/11292.9%115$0.913870
Gemma 3 27Border79/8098.8%80$0.004680
Gemma 3 27Bstate25/4555.6%45$0.002148
Gemma 3 27Btool_plan219/39755.2%397$0.069666
gpt-oss-120bbelief195/20197.0%202$0.046658
gpt-oss-120border79/8098.8%80$0.015004
gpt-oss-120bstate60/60100.0%60$0.025043
Nemotron 3 Super 120Bbelief435/45196.5%451$0.157160
Nemotron 3 Super 120Binteractive133/13797.1%148$1.080437
Nemotron 3 Super 120Border79/8098.8%80$0.018572
Nemotron 3 Super 120Bstate91/9595.8%95$0.033380
Nemotron 3 Super 120Btool_plan460/53785.7%551$0.515123