Data to …… trials

Cross-model atlas

Where each model breaks. Pick a query kind and a difficulty dimension; every model with campaign data is binned into the same difficulty buckets so the cells are directly comparable.

Map another model →
6
Models compared
8
Query kinds
79%
7,285/9,195
Campaign pass rate
143
Boundaries fitted
Query kind BeliefCode repairCode traceInteractiveOrderRepository fixStateTool plan
Difficulty dial catalog sizechain lengthdecoy countirreversibilityparallel goalsrealized plan depthstate obscurityplan overheaddishonest decoys onid collisions onimplicit preconditions onpoison trapszero slack onunsolvablefoil divergenceconditionevidence spanevidence turns
Pass rate by difficulty · Tool plan / realized plan depth5 models × 6 buckets · 2,743 trials

Gemma 3 27B breaks first: its pass rate crosses 50% at realized plan depth = 10.50. Darker red = lower pass rate. Click any cell for the trials behind it.

0–44–66–99–1111–1414–31REALIZED PLAN DEPTH → HARDERGemma 3 27Boverall 55% · n=397google/gemma-3-27b-it - realized_plan_depth 0–4: 70.7% pass (n=58)71%n=58google/gemma-3-27b-it - realized_plan_depth 4–6: 72.9% pass (n=129)73%n=129google/gemma-3-27b-it - realized_plan_depth 6–9: 28.6% pass (n=98)29%n=98google/gemma-3-27b-it - realized_plan_depth 9–11: 78.6% pass (n=14)79%n=14google/gemma-3-27b-it - realized_plan_depth 11–14: 57.1% pass (n=56)57%n=56google/gemma-3-27b-it - realized_plan_depth 14–31: 31.0% pass (n=42)31%n=42google/gemma-3-27b-it: fitted frontier at realized_plan_depth=10.50Nemotron 3 Nano 30Boverall 65% · n=615NVIDIA-Nemotron-3-Nano-30B-A3B - realized_plan_depth 0–4: 92.5% pass (n=106)92%n=106NVIDIA-Nemotron-3-Nano-30B-A3B - realized_plan_depth 4–6: 83.1% pass (n=77)83%n=77NVIDIA-Nemotron-3-Nano-30B-A3B - realized_plan_depth 6–9: 64.4% pass (n=194)64%n=194NVIDIA-Nemotron-3-Nano-30B-A3B - realized_plan_depth 9–11: 50.4% pass (n=115)50%n=115NVIDIA-Nemotron-3-Nano-30B-A3B - realized_plan_depth 11–14: 46.3% pass (n=123)46%n=123—NVIDIA-Nemotron-3-Nano-30B-A3B: fitted frontier at realized_plan_depth=11.20Gemma 4 31B QAToverall 81% · n=864gemma-4-31b-it-qat-w4a16-ct - realized_plan_depth 0–4: 97.6% pass (n=169)98%n=169gemma-4-31b-it-qat-w4a16-ct - realized_plan_depth 4–6: 99.4% pass (n=180)99%n=180gemma-4-31b-it-qat-w4a16-ct - realized_plan_depth 6–9: 93.0% pass (n=115)93%n=115gemma-4-31b-it-qat-w4a16-ct - realized_plan_depth 9–11: 39.4% pass (n=109)39%n=109gemma-4-31b-it-qat-w4a16-ct - realized_plan_depth 11–14: 79.2% pass (n=144)79%n=144gemma-4-31b-it-qat-w4a16-ct - realized_plan_depth 14–31: 61.9% pass (n=147)62%n=147gemma-4-31b-it-qat-w4a16-ct: fitted frontier at realized_plan_depth=24.58Nemotron 3 Super 120Boverall 86% · n=537nemotron-3-super-120b-a12b - realized_plan_depth 0–4: 98.1% pass (n=106)98%n=106nemotron-3-super-120b-a12b - realized_plan_depth 4–6: 99.3% pass (n=144)99%n=144nemotron-3-super-120b-a12b - realized_plan_depth 6–9: 98.4% pass (n=62)98%n=62nemotron-3-super-120b-a12b - realized_plan_depth 9–11: 97.4% pass (n=39)97%n=39nemotron-3-super-120b-a12b - realized_plan_depth 11–14: 83.7% pass (n=43)84%n=43nemotron-3-super-120b-a12b - realized_plan_depth 14–31: 54.5% pass (n=143)55%n=143nemotron-3-super-120b-a12b: fitted frontier at realized_plan_depth=20.83Nemotron 3 Ultra 550Boverall 89% · n=291Nemotron-3-Ultra-550b-a55b - realized_plan_depth 0–4: 95.2% pass (n=62)95%n=62Nemotron-3-Ultra-550b-a55b - realized_plan_depth 4–6: 100.0% pass (n=26)100%n=26Nemotron-3-Ultra-550b-a55b - realized_plan_depth 6–9: 92.9% pass (n=42)93%n=42Nemotron-3-Ultra-550b-a55b - realized_plan_depth 9–11: 92.9% pass (n=42)93%n=42Nemotron-3-Ultra-550b-a55b - realized_plan_depth 11–14: 91.4% pass (n=35)91%n=35Nemotron-3-Ultra-550b-a55b - realized_plan_depth 14–31: 77.4% pass (n=84)77%n=84Nemotron-3-Ultra-550b-a55b: fitted frontier at realized_plan_depth=30.22

29 populated cells · 5 fitted frontier(s) in range · hover a cell for exact counts

Degradation curve · model vs. modelsame buckets

The same cells as slopes, for reading two models against each other. Toggle a model off to narrow the comparison.

100%50%0%break0–44–66–99–1111–1414–31REALIZED PLAN DEPTHgoogle/gemma-3-27b-it - realized_plan_depth 0–4: 70.7% pass (n=58)google/gemma-3-27b-it - realized_plan_depth 4–6: 72.9% pass (n=129)google/gemma-3-27b-it - realized_plan_depth 6–9: 28.6% pass (n=98)google/gemma-3-27b-it - realized_plan_depth 9–11: 78.6% pass (n=14)google/gemma-3-27b-it - realized_plan_depth 11–14: 57.1% pass (n=56)google/gemma-3-27b-it - realized_plan_depth 14–31: 31.0% pass (n=42)Gemma 3 27Bfrontier 10.50NVIDIA-Nemotron-3-Nano-30B-A3B - realized_plan_depth 0–4: 92.5% pass (n=106)NVIDIA-Nemotron-3-Nano-30B-A3B - realized_plan_depth 4–6: 83.1% pass (n=77)NVIDIA-Nemotron-3-Nano-30B-A3B - realized_plan_depth 6–9: 64.4% pass (n=194)NVIDIA-Nemotron-3-Nano-30B-A3B - realized_plan_depth 9–11: 50.4% pass (n=115)NVIDIA-Nemotron-3-Nano-30B-A3B - realized_plan_depth 11–14: 46.3% pass (n=123)Nemotron 3 Nano 3gemma-4-31b-it-qat-w4a16-ct - realized_plan_depth 0–4: 97.6% pass (n=169)gemma-4-31b-it-qat-w4a16-ct - realized_plan_depth 4–6: 99.4% pass (n=180)gemma-4-31b-it-qat-w4a16-ct - realized_plan_depth 6–9: 93.0% pass (n=115)gemma-4-31b-it-qat-w4a16-ct - realized_plan_depth 9–11: 39.4% pass (n=109)gemma-4-31b-it-qat-w4a16-ct - realized_plan_depth 11–14: 79.2% pass (n=144)gemma-4-31b-it-qat-w4a16-ct - realized_plan_depth 14–31: 61.9% pass (n=147)Gemma 4 31B QATnemotron-3-super-120b-a12b - realized_plan_depth 0–4: 98.1% pass (n=106)nemotron-3-super-120b-a12b - realized_plan_depth 4–6: 99.3% pass (n=144)nemotron-3-super-120b-a12b - realized_plan_depth 6–9: 98.4% pass (n=62)nemotron-3-super-120b-a12b - realized_plan_depth 9–11: 97.4% pass (n=39)nemotron-3-super-120b-a12b - realized_plan_depth 11–14: 83.7% pass (n=43)nemotron-3-super-120b-a12b - realized_plan_depth 14–31: 54.5% pass (n=143)Nemotron 3 Super Nemotron-3-Ultra-550b-a55b - realized_plan_depth 0–4: 95.2% pass (n=62)Nemotron-3-Ultra-550b-a55b - realized_plan_depth 4–6: 100.0% pass (n=26)Nemotron-3-Ultra-550b-a55b - realized_plan_depth 6–9: 92.9% pass (n=42)Nemotron-3-Ultra-550b-a55b - realized_plan_depth 9–11: 92.9% pass (n=42)Nemotron-3-Ultra-550b-a55b - realized_plan_depth 11–14: 91.4% pass (n=35)Nemotron-3-Ultra-550b-a55b - realized_plan_depth 14–31: 77.4% pass (n=84)Nemotron 3 Ultra
Summary · model x query kind32 pairs
modelquery_kindpass ratentotal cost
Nemotron 3 Nano 30Bbelief800/89589.4%927$0.189034
Nemotron 3 Nano 30Binteractive92/11977.3%133$0.774886
Nemotron 3 Nano 30Border156/15898.7%160$0.022909
Nemotron 3 Nano 30Bstate90/90100.0%90$0.014016
Nemotron 3 Nano 30Btool_plan402/61565.4%637$0.348336
Nemotron 3 Ultra 550Bbelief308/31199.0%316$0.815803
Nemotron 3 Ultra 550Binteractive0/00.0%10$0.432233
Nemotron 3 Ultra 550Border80/80100.0%80$0.085632
Nemotron 3 Ultra 550Bstate30/30100.0%30$0.043059
Nemotron 3 Ultra 550Btool_plan260/29189.3%291$1.094840
Gemma 4 31B QATbelief544/89860.6%898$0.028576
Gemma 4 31B QATcode_repair21/6532.3%72$0.036091
Gemma 4 31B QATcode_trace792/81996.7%823$0.201881
Gemma 4 31B QATinteractive352/61857.0%675$1.073585
Gemma 4 31B QATorder199/20995.2%209$0.008847
Gemma 4 31B QATrepo_fix52/18228.6%204$1.966217
Gemma 4 31B QATstate124/124100.0%124$0.004818
Gemma 4 31B QATtool_plan699/86480.9%867$0.170132
Gemma 3 27Bbelief142/30746.3%307$0.007658
Gemma 3 27Bcode_trace183/24574.7%245$0.062598
Gemma 3 27Binteractive104/11292.9%115$0.913870
Gemma 3 27Border79/8098.8%80$0.004680
Gemma 3 27Bstate25/4555.6%45$0.002148
Gemma 3 27Btool_plan219/39755.2%397$0.069666
gpt-oss-120bbelief195/20197.0%202$0.046658
gpt-oss-120border79/8098.8%80$0.015004
gpt-oss-120bstate60/60100.0%60$0.025043
Nemotron 3 Super 120Bbelief435/45196.5%451$0.157160
Nemotron 3 Super 120Binteractive133/13797.1%148$1.080437
Nemotron 3 Super 120Border79/8098.8%80$0.018572
Nemotron 3 Super 120Bstate91/9595.8%95$0.033380
Nemotron 3 Super 120Btool_plan460/53785.7%551$0.515123