Data to …… trials

Cross-model atlas

Where each model breaks. Pick a query kind and a difficulty dimension; every model with campaign data is binned into the same difficulty buckets so the cells are directly comparable.

Map another model →
6
Models compared
8
Query kinds
79%
7,285/9,195
Campaign pass rate
143
Boundaries fitted
Query kind BeliefCode repairCode traceInteractiveOrderRepository fixStateTool plan
Difficulty dial context lengthcoreferencedistractorsentity statetemporalwitness gapnoop distractorsrelational time
Pass rate by difficulty · Order / context length6 models × 4 buckets · 689 trials

Nemotron 3 Nano 30B breaks first: its pass rate crosses 50% at context length = 34.23. Darker red = lower pass rate. Click any cell for the trials behind it.

6–1010–1515–2424–90CONTEXT LENGTH → HARDERGemma 4 31B QAToverall 95% · n=209gemma-4-31b-it-qat-w4a16-ct - context_length 6–10: 97.8% pass (n=90)98%n=90gemma-4-31b-it-qat-w4a16-ct - context_length 10–15: 90.9% pass (n=44)91%n=44gemma-4-31b-it-qat-w4a16-ct - context_length 15–24: 100.0% pass (n=35)100%n=35gemma-4-31b-it-qat-w4a16-ct - context_length 24–90: 90.0% pass (n=40)90%n=40gemma-4-31b-it-qat-w4a16-ct: fitted frontier at context_length=62Nemotron 3 Nano 30Boverall 99% · n=158NVIDIA-Nemotron-3-Nano-30B-A3B - context_length 6–10: 100.0% pass (n=89)100%n=89NVIDIA-Nemotron-3-Nano-30B-A3B - context_length 10–15: 100.0% pass (n=20)100%n=20NVIDIA-Nemotron-3-Nano-30B-A3B - context_length 15–24: 100.0% pass (n=30)100%n=30NVIDIA-Nemotron-3-Nano-30B-A3B - context_length 24–90: 89.5% pass (n=19)89%n=19NVIDIA-Nemotron-3-Nano-30B-A3B: fitted frontier at context_length=34.23Gemma 3 27Boverall 99% · n=80google/gemma-3-27b-it - context_length 6–10: 97.8% pass (n=45)98%n=45google/gemma-3-27b-it - context_length 10–15: 100.0% pass (n=10)100%n=10google/gemma-3-27b-it - context_length 15–24: 100.0% pass (n=15)100%n=15google/gemma-3-27b-it - context_length 24–90: 100.0% pass (n=10)100%n=10google/gemma-3-27b-it: fitted frontier at context_length=35.17gpt-oss-120boverall 99% · n=80gpt-oss-120b - context_length 6–10: 97.8% pass (n=45)98%n=45gpt-oss-120b - context_length 10–15: 100.0% pass (n=10)100%n=10gpt-oss-120b - context_length 15–24: 100.0% pass (n=15)100%n=15gpt-oss-120b - context_length 24–90: 100.0% pass (n=10)100%n=10gpt-oss-120b: fitted frontier at context_length=35.17Nemotron 3 Super 120Boverall 99% · n=80nemotron-3-super-120b-a12b - context_length 6–10: 100.0% pass (n=45)100%n=45nemotron-3-super-120b-a12b - context_length 10–15: 100.0% pass (n=10)100%n=10nemotron-3-super-120b-a12b - context_length 15–24: 93.3% pass (n=15)93%n=15nemotron-3-super-120b-a12b - context_length 24–90: 100.0% pass (n=10)100%n=10nemotron-3-super-120b-a12b: fitted frontier at context_length=35.17Nemotron 3 Ultra 550Boverall 100% · n=80Nemotron-3-Ultra-550b-a55b - context_length 6–10: 100.0% pass (n=45)100%n=45Nemotron-3-Ultra-550b-a55b - context_length 10–15: 100.0% pass (n=10)100%n=10Nemotron-3-Ultra-550b-a55b - context_length 15–24: 100.0% pass (n=15)100%n=15Nemotron-3-Ultra-550b-a55b - context_length 24–90: 100.0% pass (n=10)100%n=10

24 populated cells · 5 fitted frontier(s) in range · hover a cell for exact counts

Degradation curve · model vs. modelsame buckets

The same cells as slopes, for reading two models against each other. Toggle a model off to narrow the comparison.

100%50%0%break6–1010–1515–2424–90CONTEXT LENGTHgemma-4-31b-it-qat-w4a16-ct - context_length 6–10: 97.8% pass (n=90)gemma-4-31b-it-qat-w4a16-ct - context_length 10–15: 90.9% pass (n=44)gemma-4-31b-it-qat-w4a16-ct - context_length 15–24: 100.0% pass (n=35)gemma-4-31b-it-qat-w4a16-ct - context_length 24–90: 90.0% pass (n=40)Gemma 4 31B QATNVIDIA-Nemotron-3-Nano-30B-A3B - context_length 6–10: 100.0% pass (n=89)NVIDIA-Nemotron-3-Nano-30B-A3B - context_length 10–15: 100.0% pass (n=20)NVIDIA-Nemotron-3-Nano-30B-A3B - context_length 15–24: 100.0% pass (n=30)NVIDIA-Nemotron-3-Nano-30B-A3B - context_length 24–90: 89.5% pass (n=19)Nemotron 3 Nano 3frontier 34.23google/gemma-3-27b-it - context_length 6–10: 97.8% pass (n=45)google/gemma-3-27b-it - context_length 10–15: 100.0% pass (n=10)google/gemma-3-27b-it - context_length 15–24: 100.0% pass (n=15)google/gemma-3-27b-it - context_length 24–90: 100.0% pass (n=10)Gemma 3 27Bgpt-oss-120b - context_length 6–10: 97.8% pass (n=45)gpt-oss-120b - context_length 10–15: 100.0% pass (n=10)gpt-oss-120b - context_length 15–24: 100.0% pass (n=15)gpt-oss-120b - context_length 24–90: 100.0% pass (n=10)gpt-oss-120bnemotron-3-super-120b-a12b - context_length 6–10: 100.0% pass (n=45)nemotron-3-super-120b-a12b - context_length 10–15: 100.0% pass (n=10)nemotron-3-super-120b-a12b - context_length 15–24: 93.3% pass (n=15)nemotron-3-super-120b-a12b - context_length 24–90: 100.0% pass (n=10)Nemotron 3 Super Nemotron-3-Ultra-550b-a55b - context_length 6–10: 100.0% pass (n=45)Nemotron-3-Ultra-550b-a55b - context_length 10–15: 100.0% pass (n=10)Nemotron-3-Ultra-550b-a55b - context_length 15–24: 100.0% pass (n=15)Nemotron-3-Ultra-550b-a55b - context_length 24–90: 100.0% pass (n=10)Nemotron 3 Ultra
Summary · model x query kind32 pairs
modelquery_kindpass ratentotal cost
Nemotron 3 Nano 30Bbelief800/89589.4%927$0.189034
Nemotron 3 Nano 30Binteractive92/11977.3%133$0.774886
Nemotron 3 Nano 30Border156/15898.7%160$0.022909
Nemotron 3 Nano 30Bstate90/90100.0%90$0.014016
Nemotron 3 Nano 30Btool_plan402/61565.4%637$0.348336
Nemotron 3 Ultra 550Bbelief308/31199.0%316$0.815803
Nemotron 3 Ultra 550Binteractive0/00.0%10$0.432233
Nemotron 3 Ultra 550Border80/80100.0%80$0.085632
Nemotron 3 Ultra 550Bstate30/30100.0%30$0.043059
Nemotron 3 Ultra 550Btool_plan260/29189.3%291$1.094840
Gemma 4 31B QATbelief544/89860.6%898$0.028576
Gemma 4 31B QATcode_repair21/6532.3%72$0.036091
Gemma 4 31B QATcode_trace792/81996.7%823$0.201881
Gemma 4 31B QATinteractive352/61857.0%675$1.073585
Gemma 4 31B QATorder199/20995.2%209$0.008847
Gemma 4 31B QATrepo_fix52/18228.6%204$1.966217
Gemma 4 31B QATstate124/124100.0%124$0.004818
Gemma 4 31B QATtool_plan699/86480.9%867$0.170132
Gemma 3 27Bbelief142/30746.3%307$0.007658
Gemma 3 27Bcode_trace183/24574.7%245$0.062598
Gemma 3 27Binteractive104/11292.9%115$0.913870
Gemma 3 27Border79/8098.8%80$0.004680
Gemma 3 27Bstate25/4555.6%45$0.002148
Gemma 3 27Btool_plan219/39755.2%397$0.069666
gpt-oss-120bbelief195/20197.0%202$0.046658
gpt-oss-120border79/8098.8%80$0.015004
gpt-oss-120bstate60/60100.0%60$0.025043
Nemotron 3 Super 120Bbelief435/45196.5%451$0.157160
Nemotron 3 Super 120Binteractive133/13797.1%148$1.080437
Nemotron 3 Super 120Border79/8098.8%80$0.018572
Nemotron 3 Super 120Bstate91/9595.8%95$0.033380
Nemotron 3 Super 120Btool_plan460/53785.7%551$0.515123