Experiment

PWS-Decisions: the site's models on dated web facts

Decision models

Perplexity Decider 27B v1.2 · API · with web

87.0%

with web search

The same model without web

50.7%

Perplexity Decider 27B v1.2 · API

Best decision model without search

60.0%

Metask-Jev-4B

Accuracy

Share of the 100 public questions answered right, the mean of our 3 runs per setup · higher is better; beside it the cost (per 1,000 questionsper 1M questions) and the median and 95th percentile request time

Always the first option: 68%
  1. With web search
  2. GPT-6.1 Sol high · chat · CLI · with web100.0% | $55.689$55,689 | 12,860 · 29,712
  3. Gemini 4 Argon low · chat · with web99.7% | ≥ $6.088≥ $6,088 | 12,216 · 88,370
  4. GPT-6 Astra high · chat · CLI · with web99.7% | $177.498$177,498 | 9,482 · 22,548
  5. Claude Fable 5.1 high · chat · CLI · with web · 2 requests handed to Claude Opus 4.8, counted wrong99.0% | $121.016$121,016 | 17,589 · 52,579
  6. GPT-6.1 Terra high · chat · CLI · with web99.0% | $24.072$24,072 | 9,727 · 17,746
  7. Gemini 3.8 Flash high · chat · with web98.3% | ≥ $4.781≥ $4,781 | 11,615 · 46,567
  8. Grok 4.7 high · chat · with web98.3% | $210.897$210,897 | 16,629 · 67,866
  9. Claude Opus 5.5 high · chat · CLI · with web98.0% | $50.805$50,805 | 16,231 · 38,925
  10. Claude Sonnet 5.5 high · chat · CLI · with web96.3% | $32.381$32,381 | 13,948 · 35,296
  11. Perplexity Decider 27B v1.2 API · with web87.0% | ≤ $1.041≤ $1,041 | 483 · 728
  12. GPT-6 Luna low · chat · CLI · with web63.7% | $4.886$4,886 | 5,905 · 13,315
  13. Without search
  14. Claude Sonnet 5.5 high · chat · CLI80.0% | $5.782$5,782 | 5,368 · 10,007
  15. Claude Opus 5.5 high · chat · CLI78.7% | $10.940$10,940 | 5,190 · 9,312
  16. GPT-6.1 Sol high · chat · CLI75.7% | $10.985$10,985 | 15,665 · 43,525
  17. GPT-6 Astra high · chat · CLI75.3% | $41.488$41,488 | 10,229 · 26,512
  18. Claude Fable 5.1 high · chat · CLI · 5 requests handed to Claude Opus 4.8, counted wrong73.7% | $36.773$36,773 | 7,880 · 18,796
  19. GPT-6.1 Terra high · chat · CLI73.0% | $3.359$3,359 | 6,217 · 10,543
  20. Gemini 4 Argon low · chat68.7% | $2.887$2,887 | 3,484 · 12,776
  21. Kimi K3.1 Pro low · chat66.3% | $1.994$1,994 | 5,312 · 11,414
  22. Erk-14B reasoning off · chat65.0% | no price | 255 · 260
  23. GLM-5.3 low · chat61.0% | $0.299$299 | 1,412 · 2,467
  24. Metask-Jev-4B60.0% | no price | 74 · 79
  25. DeepSeek-V4.1-Flash reasoning on · chat · 101759.3% | $1.368$1,368 | 16,474 · 70,100
  26. Perplexity Decider Flash v1.2picks the first choice in 7 of 10 questions; always picking the first choice scores 68%59.0% | $0.002$2.01 | 27 · 30
  27. Qwen3-14B reasoning off · chat59.0% | no price | 366 · 391
  28. d358.0% | no price | 63 · 77
  29. d1-ultra57.7% | $0.006$6.17 | 220 · 560
  30. Qwen3.8-27B low · chat57.7% | no price | 5,636 · 16,380
  31. Laya57.0% | no price | 11 · 12
  32. Perplexity Decider 27B v1.2 DGX Spark57.0% | $0.004$4.03 | 74 · 98
  33. d3-ultra55.0% | no price | 125 · 171
  34. REAX 1.355.0% | no price | 215 · 355
  35. Strands Decider 2Bpicks the first choice in 7 of 10 questions; always picking the first choice scores 68%55.0% | no price | 24 · 28
  36. d1-2B54.0% | no price | 11 · 12
  37. Gemma 4 31B reasoning off · chat54.0% | no price | 321 · 335
  38. Laya Multilingualpicks the first choice in 6 of 10 questions; always picking the first choice scores 68%54.0% | no price | 9 · 11
  39. Mercury Decide53.3% | $0.004$3.57 | 291 · 365
  40. d1.1picks the first choice in 6 of 10 questions; always picking the first choice scores 68%53.0% | $0.003$2.94 | 180 · 415
  41. GLiDE v1.253.0% | $0.007$6.90 | 64 · 71
  42. SubQ Snap 1.153.0% | $0.008$7.94 | 378 · 390
  43. SubQ Snap Preview53.0% | $0.010$9.92 | 380 · 392
  44. Clef52.0% | $0.058$58.40 | 126 · 154
  45. Perplexity Decider 27B v1.2 API50.7% | $0.004$3.97 | 244 · 656
  46. Grok 4.7 high · chat50.3% | $35.746$35,746 | 57,796 · 343,779
  47. Decider 150.0% | $0.007$6.81 | 629 · 690
  48. Microsoft-Decision-149.0% | $0.006$6.11 | 317 · 385
  49. Open-Jev-27B-v1.1picks the first choice in 5 of 10 questions; always picking the first choice scores 68%49.0% | no price | 146 · 272
  50. Clef-Flash48.0% | $0.009$9.25 | 85 · 109
  51. d147.7% | $0.003$2.94 | 185 · 425
  52. Drex Max v1.147.7% | $0.012$11.90 | 331 · 470
  53. Gemini 3.8 Flash high · chat46.0% | $19.985$19,985 | 22,523 · 167,428
  54. GPT-6 Luna low · chat · CLI45.3% | $0.307$307 | 3,981 · 5,971
  55. OneJev-27B45.0% | no price | 94 · 132
  56. DeepSeek-V4.1-Flash reasoning off · chat · 101743.7% | $0.078$77.53 | 944 · 1,270
  57. Drex Max42.0% | $0.012$12.08 | 64 · 78
  58. GPT-6 Luna Decisions API42.0% | $0.022$21.93 | 285 · 522
  59. GLiNER2.5-multi-Decidepicks the first choice in 4 of 10 questions; always picking the first choice scores 68%41.0% | no price | 8 · 9
  60. Mammoth-1 RTX PRO 6000 Blackwell39.0% | $0.020$20.43 | 128 · 135
  61. Winnow-12B39.0% | no price | 48 · 59
  62. Mammoth-1 Baseten37.0% | $0.020$20.42 | 184 · 255
  63. Jev 1.13picks the first choice in 3 of 10 questions; always picking the first choice scores 68%35.0% | $0.018$18.01 | 170 · 242
  64. Jev 1.14picks the first choice in 3 of 10 questions; always picking the first choice scores 68%34.3% | $0.018$18.01 | 165 · 203
  65. celeris-1-decision33.3% | $0.008$8.46 | 384 · 389
  66. Elephant-133.0% | $0.007$7.15 | 55 · 63
  67. Cygnet32.0% | $0.023$22.82 | 27 · 29
  68. Drex 2.0picks the first choice in 2 of 10 questions; always picking the first choice scores 68%31.0% | $0.008$8.11 | 293 · 1,507

GPT Gemini Claude Grok Perplexity Kimi Erk GLM Metask DeepSeek Qwen vLLM-SR Liquid Laya REAX Strands Gemma Inception Fastino SubQ Clef meraGPT Microsoft Open-Jev Drex OneJev Mammoth Winnow Jev Celeris Elephant Cygnet With web: the setup searched the web for each question (striped) “chat” in a tag: a chat model, asked each question in the benchmark's own words, its answer read from its reply; the other rows are decision models. The chance line: Parallel's random-guess baseline, 44.67% (the benchmark's README), uniform guessing over each question's choices, a property of the questions, not a run of ours. The dashed line: always picking the first option scores 68%, as the right answer is the first choice on 68 of the 100 questions. A line under a setup's name: its score is at most 1 point above what its answering habits alone would score (without knowing the facts).

Can a model score here without knowing the facts?

Always choosing the first option scores 68%, above every decision setup without search here. Answer-position and yes/true imbalances complicate the ranking; these checks cannot establish whether a model has seen the questions.

The right answer is the first choice on 68 questions, the second on 26, the third on 5 and the fourth on 1. 60 questions are yes/no (53) or true/false (7); on every one "yes" or "true" is the first choice, and "yes" or "true" is right on 48 of them (42 of 53 and 6 of 7). The checks below use only totals: the answer key's positions, each decision setup's answers over its 3 runs, and public dates.

1. Strategies that never read the question

  1. Never reading the question
  2. Always the first choice68%
  3. Always the last choice19%
  4. Always the position most often right (the first)68%
  5. Always "yes"/"true" on the 60 yes/no and true/false questions80% of 60
  6. Always "yes"/"true" on those 60, uniform guess on the rest62.7%
  7. Best decision setup without search
  8. Metask-Jev-4B60.0%

MetaskA strategy that never reads the question Uniform random guessing on the same questions (44.7% on all 100; 50% on the 60 yes/no and true/false questions)

StrategyScoreon the 100 questionsUniform random guessingthe same questions
Always the first choice68%44.7%
Always the last choice19%44.7%
Always the position most often right in the key (the first)68%44.7%
Always "yes" or "true" on the 60 yes/no and true/false questions48 of 60 right (80%)50% on them
Always "yes" or "true" on those 60, a uniform random guess on the other 4062.7%44.7%

No decision setup without search reaches the 68% of always picking the first choice; the best, Metask-Jev-4B, scores 60.0%. 25 of the 37 score above uniform random guessing (44.7%).

2. Answering habits

  1. With web search
  2. Perplexity Decider 27B v1.2 API · with web87.0%58.6%+28.4
  3. Without search
  4. Metask-Jev-4B60.0%56.1%+3.9
  5. Perplexity Decider Flash v1.2at or below habit baseline + 1 pt59.0%58.2%+0.8
  6. d358.0%51.5%+6.5
  7. d1-ultra57.7%47.6%+10.0
  8. Laya57.0%51.7%+5.3
  9. Perplexity Decider 27B v1.2 DGX Spark57.0%50.0%+7.0
  10. d3-ultra55.0%48.0%+7.0
  11. REAX 1.355.0%48.0%+7.0
  12. Strands Decider 2Bat or below habit baseline + 1 pt55.0%55.7%-0.7
  13. d1-2B54.0%50.1%+3.9
  14. Laya Multilingualat or below habit baseline + 1 pt54.0%55.8%-1.8
  15. Mercury Decide53.3%48.3%+5.0
  16. d1.1at or below habit baseline + 1 pt53.0%53.2%-0.2
  17. GLiDE v1.253.0%46.9%+6.1
  18. SubQ Snap 1.153.0%48.0%+5.0
  19. SubQ Snap Preview53.0%48.0%+5.0
  20. Clef52.0%41.3%+10.7
  21. Perplexity Decider 27B v1.2 API50.7%48.2%+2.4
  22. Decider 150.0%47.2%+2.8
  23. Microsoft-Decision-149.0%44.0%+5.0
  24. Open-Jev-27B-v1.1at or below habit baseline + 1 pt49.0%49.0%-0.0
  25. Clef-Flash48.0%45.3%+2.7
  26. d147.7%46.2%+1.4
  27. Drex Max v1.147.7%43.4%+4.3
  28. OneJev-27B45.0%38.9%+6.1
  29. Drex Max42.0%40.8%+1.2
  30. GPT-6 Luna Decisions API42.0%38.3%+3.7
  31. GLiNER2.5-multi-Decideat or below habit baseline + 1 pt41.0%41.7%-0.7
  32. Mammoth-1 RTX PRO 6000 Blackwell39.0%35.5%+3.5
  33. Winnow-12B39.0%36.7%+2.3
  34. Mammoth-1 Baseten37.0%33.9%+3.1
  35. Jev 1.13at or below habit baseline + 1 pt35.0%37.1%-2.1
  36. Jev 1.14at or below habit baseline + 1 pt34.3%37.1%-2.8
  37. celeris-1-decision33.3%31.4%+1.9
  38. Elephant-133.0%29.8%+3.2
  39. Cygnet32.0%29.2%+2.8
  40. Drex 2.0at or below habit baseline + 1 pt31.0%31.7%-0.7

Perplexity Metask vLLM-SR Liquid Laya REAX Strands Inception Fastino SubQ Clef meraGPT Microsoft Open-Jev Drex OneJev GPT Mammoth Winnow Jev Celeris Elephant CygnetThe upper bar: accuracy, the mean of the runs, in the family's colour The lower bar: the habit baseline, the score of the same habits without reading the question A line under a name: at or below habit baseline + 1 pt

SetupAccuracymean of 3 runsFirst choiceshare of answersSecond choiceshare of answersThird choiceshare of answersFourth choiceshare of answers"Yes" or "true"share on the 60 yes/no and true/false questionsFirst choice, other questionsshare on the other 40Same habits, no readinghabit baselineAbove the habitspointspone-sided, exploratoryp, Holmover the 37 without search
Answer key68%26%5%1%80%50%
Perplexity Decider 27B v1.2 API · with web Open · 27B87.0%68.0%25.0%5.0%2.0%76.7%55.0%58.6%+28.40.000with search: not tested
Metask-Jev-4B Open · 4.5B60.0%66.0%32.0%2.0%0.0%70.0%60.0%56.1%+3.90.1951.000
Perplexity Decider Flash v1.2 Open · 4.0B59.0%66.0%26.0%5.0%3.0%78.3%47.5%58.2%+0.8at or below habit baseline + 1 pt0.4211.000
d3 Open · 26B58.0%55.0%37.0%5.0%3.0%61.7%45.0%51.5%+6.50.0851.000
d1-ultra Closed · Efficient57.7%53.0%39.0%2.0%6.0%57.8%45.8%47.6%+10.00.0180.662
Laya Open · 0.4B57.0%59.0%32.0%8.0%1.0%60.0%57.5%51.7%+5.30.1281.000
Perplexity Decider 27B v1.2 DGX Spark Open · 27B57.0%52.0%37.0%8.0%3.0%65.0%32.5%50.0%+7.00.0661.000
d3-ultra Open · 26B55.0%48.0%44.0%6.0%2.0%55.0%37.5%48.0%+7.00.0751.000
REAX 1.3 Open · 57B55.0%49.0%46.0%4.0%1.0%53.3%42.5%48.0%+7.00.0781.000
Strands Decider 2B Open · 2.0B55.0%69.0%26.0%5.0%0.0%70.0%67.5%55.7%-0.7at or below habit baseline + 1 pt0.5631.000
d1-2B Open · 1.9B54.0%55.0%37.0%7.0%1.0%61.7%45.0%50.1%+3.90.2071.000
Laya Multilingual Open · 0.3B54.0%59.0%30.0%9.0%2.0%70.0%42.5%55.8%-1.8at or below habit baseline + 1 pt0.6571.000
Mercury Decide Closed · Efficient53.3%51.7%38.3%6.0%4.0%56.1%45.0%48.3%+5.00.1501.000
d1.1 Closed · Efficient53.0%62.7%28.7%6.0%2.7%66.1%57.5%53.2%-0.2at or below habit baseline + 1 pt0.5131.000
GLiDE v1.2 Open · 27B53.0%47.0%45.0%5.0%3.0%56.7%32.5%46.9%+6.10.1011.000
SubQ Snap 1.1 Closed · Efficient53.0%48.0%41.0%8.0%3.0%60.0%30.0%48.0%+5.00.1471.000
SubQ Snap Preview Closed · Efficient53.0%48.0%41.0%8.0%3.0%60.0%30.0%48.0%+5.00.1471.000
Clef Open · 27B52.0%40.0%52.0%5.0%3.0%38.3%42.5%41.3%+10.70.0130.481
Perplexity Decider 27B v1.2 API Open · 27B50.7%47.7%44.0%5.3%3.0%56.7%34.2%48.2%+2.40.3041.000
Decider 1 Closed · Efficient50.0%46.0%48.0%5.0%1.0%53.3%35.0%47.2%+2.80.2861.000
Microsoft-Decision-1 Closed · Efficient49.0%40.7%54.3%3.0%2.0%41.1%40.0%44.0%+5.00.1501.000
Open-Jev-27B-v1.1 Open · 27B49.0%53.0%37.0%8.0%2.0%58.3%45.0%49.0%-0.0at or below habit baseline + 1 pt0.5011.000
Clef-Flash Open · 9.0B48.0%47.0%45.0%5.0%3.0%51.7%40.0%45.3%+2.70.2871.000
d1 Closed · Efficient47.7%48.0%43.3%5.7%3.0%50.6%44.2%46.2%+1.40.3851.000
Drex Max v1.1 Closed · Efficient (provisional)47.7%38.3%54.7%4.0%3.0%45.6%27.5%43.4%+4.30.1801.000
OneJev-27B Open · 27B45.0%32.0%61.0%5.0%2.0%31.7%32.5%38.9%+6.10.0861.000
Drex Max Open · 27B42.0%34.0%59.0%4.0%3.0%38.3%27.5%40.8%+1.20.3941.000
GPT-6 Luna Decisions API Closed · Efficient42.0%33.0%58.0%2.0%7.0%45.0%15.0%38.3%+3.70.2011.000
GLiNER2.5-multi-Decide Open · 0.3B41.0%39.0%51.0%9.0%1.0%36.7%42.5%41.7%-0.7at or below habit baseline + 1 pt0.5591.000
Mammoth-1 RTX PRO 6000 Blackwell Open · 27B39.0%24.0%71.0%2.0%3.0%25.0%22.5%35.5%+3.50.2021.000
Winnow-12B Open · 12B39.0%28.0%61.0%3.0%8.0%35.0%17.5%36.7%+2.30.2991.000
Mammoth-1 Baseten Open · 27B37.0%23.0%72.0%2.0%3.0%23.3%22.5%33.9%+3.10.2301.000
Jev 1.13 Closed · Efficient35.0%30.3%57.7%4.7%7.3%35.6%22.5%37.1%-2.1at or below habit baseline + 1 pt0.6781.000
Jev 1.14 Closed · Efficient34.3%28.3%58.3%6.0%7.3%33.3%20.8%37.1%-2.8at or below habit baseline + 1 pt0.7311.000
celeris-1-decision Closed · Efficient33.3%18.0%72.0%3.0%7.0%16.7%20.0%31.4%+1.90.3131.000
Elephant-1 Open · 9.0B33.0%17.0%76.0%4.0%3.0%13.3%22.5%29.8%+3.20.1941.000
Cygnet Open · 12B32.0%15.0%72.0%4.0%9.0%18.3%10.0%29.2%+2.80.2261.000
Drex 2.0 Open · 35B MoE · 3.0B active31.0%19.0%72.0%5.0%4.0%18.3%20.0%31.7%-0.7at or below habit baseline + 1 pt0.5761.000

3. Hard questions

  1. With web search
  2. Perplexity Decider 27B v1.2 API · with web8.00 of 10
  3. Without search
  4. Metask-Jev-4B3.00 of 10
  5. Perplexity Decider Flash v1.22.00 of 10
  6. d30.00 of 10
  7. d1-ultra1.00 of 10
  8. Laya2.00 of 10
  9. Perplexity Decider 27B v1.2 DGX Spark1.00 of 10
  10. d3-ultra1.00 of 10
  11. REAX 1.31.00 of 10
  12. Strands Decider 2B4.00 of 10
  13. d1-2B1.00 of 10
  14. Laya Multilingual3.00 of 10
  15. Mercury Decide2.33 of 10
  16. d1.11.33 of 10
  17. GLiDE v1.21.00 of 10
  18. SubQ Snap 1.10.00 of 10
  19. SubQ Snap Preview0.00 of 10
  20. Clef3.00 of 10
  21. Perplexity Decider 27B v1.2 API0.67 of 10
  22. Decider 15.00 of 10
  23. Microsoft-Decision-10.67 of 10
  24. Open-Jev-27B-v1.11.00 of 10
  25. Clef-Flash0.00 of 10
  26. d10.00 of 10
  27. Drex Max v1.11.00 of 10
  28. OneJev-27B0.00 of 10
  29. Drex Max0.00 of 10
  30. GPT-6 Luna Decisions API1.00 of 10
  31. GLiNER2.5-multi-Decide2.00 of 10
  32. Mammoth-1 RTX PRO 6000 Blackwell0.00 of 10
  33. Winnow-12B1.00 of 10
  34. Mammoth-1 Baseten0.00 of 10
  35. Jev 1.130.00 of 10
  36. Jev 1.140.00 of 10
  37. celeris-1-decision0.00 of 10
  38. Elephant-10.00 of 10
  39. Cygnet1.00 of 10
  40. Drex 2.00.00 of 10

Perplexity Metask vLLM-SR Liquid Laya REAX Strands Inception Fastino SubQ Clef meraGPT Microsoft Open-Jev Drex OneJev GPT Mammoth Winnow Jev Celeris Elephant CygnetUniform random guessing: 3.75 of the 10 expected right

SetupRight, runs 1 / 2 / 3of 10Meanof the runsRight in all 3 runs
Perplexity Decider 27B v1.2 API · with web Open · 27B8 / 8 / 88.008
Metask-Jev-4B Open · 4.5B3 / 3 / 33.003
Perplexity Decider Flash v1.2 Open · 4.0B2 / 2 / 22.002
d3 Open · 26B0 / 0 / 00.000
d1-ultra Closed · Efficient1 / 1 / 11.001
Laya Open · 0.4B2 / 2 / 22.002
Perplexity Decider 27B v1.2 DGX Spark Open · 27B1 / 1 / 11.001
d3-ultra Open · 26B1 / 1 / 11.001
REAX 1.3 Open · 57B1 / 1 / 11.001
Strands Decider 2B Open · 2.0B4 / 4 / 44.004
d1-2B Open · 1.9B1 / 1 / 11.001
Laya Multilingual Open · 0.3B3 / 3 / 33.003
Mercury Decide Closed · Efficient3 / 2 / 22.331
d1.1 Closed · Efficient2 / 1 / 11.331
GLiDE v1.2 Open · 27B1 / 1 / 11.001
SubQ Snap 1.1 Closed · Efficient0 / 0 / 00.000
SubQ Snap Preview Closed · Efficient0 / 0 / 00.000
Clef Open · 27B3 / 3 / 33.003
Perplexity Decider 27B v1.2 API Open · 27B0 / 1 / 10.670
Decider 1 Closed · Efficient5 / 5 / 55.005
Microsoft-Decision-1 Closed · Efficient1 / 0 / 10.670
Open-Jev-27B-v1.1 Open · 27B1 / 1 / 11.001
Clef-Flash Open · 9.0B0 / 0 / 00.000
d1 Closed · Efficient0 / 0 / 00.000
Drex Max v1.1 Closed · Efficient (provisional)1 / 1 / 11.001
OneJev-27B Open · 27B0 / 0 / 00.000
Drex Max Open · 27B0 / 0 / 00.000
GPT-6 Luna Decisions API Closed · Efficient1 / 1 / 11.001
GLiNER2.5-multi-Decide Open · 0.3B2 / 2 / 22.002
Mammoth-1 RTX PRO 6000 Blackwell Open · 27B0 / 0 / 00.000
Winnow-12B Open · 12B1 / 1 / 11.001
Mammoth-1 Baseten Open · 27B0 / 0 / 00.000
Jev 1.13 Closed · Efficient0 / 0 / 00.000
Jev 1.14 Closed · Efficient0 / 0 / 00.000
celeris-1-decision Closed · Efficient0 / 0 / 00.000
Elephant-1 Open · 9.0B0 / 0 / 00.000
Cygnet Open · 12B1 / 1 / 11.001
Drex 2.0 Open · 35B MoE · 3.0B active0 / 0 / 00.000

Hard: answered right by at most 20% of the 37 decision setups without search (each setup's mean over its runs). 10 questions (5 with 2 choices and 5 with 4 choices); uniform random guessing expects 3.75 of them right, and the right answer is the first choice on 7 of them. The hard set comes from the same setups' answers, so it is no independent contamination test.

4. Probability of the chosen choice

  1. With web search
  2. Perplexity Decider 27B v1.2 API · with web93.4%73.5%169
  3. Without search
  4. Metask-Jev-4B78.2%67.9%0
  5. Perplexity Decider Flash v1.269.9%62.2%0
  6. d368.0%63.3%6
  7. d1-ultra69.8%64.4%0
  8. Laya71.5%68.4%0
  9. Perplexity Decider 27B v1.2 DGX Spark68.2%61.8%3
  10. d3-ultra81.5%75.7%6
  11. REAX 1.368.6%63.3%0
  12. Strands Decider 2B73.4%67.3%0
  13. d1-2B63.4%61.9%0
  14. Laya Multilingual77.3%71.7%6
  15. Mercury Decide68.2%61.4%6
  16. d1.168.3%64.3%0
  17. GLiDE v1.268.9%66.2%0
  18. SubQ Snap 1.175.4%64.4%0
  19. SubQ Snap Preview75.4%64.4%0
  20. Clef66.5%69.7%0
  21. Perplexity Decider 27B v1.2 API70.0%59.0%6
  22. Decider 161.3%59.3%0
  23. Microsoft-Decision-167.3%64.9%0
  24. Open-Jev-27B-v1.154.3%52.2%0
  25. Clef-Flash71.9%67.9%0
  26. d170.3%67.5%4
  27. Drex Max v1.171.9%71.4%3
  28. OneJev-27B59.5%60.7%0
  29. Drex Max83.1%80.8%36
  30. GPT-6 Luna Decisions API71.0%67.2%18
  31. GLiNER2.5-multi-Decide62.3%62.6%3
  32. Mammoth-1 RTX PRO 6000 Blackwell68.3%71.0%0
  33. Winnow-12B76.1%77.0%0
  34. Mammoth-1 Baseten70.1%69.9%0
  35. Jev 1.1366.6%70.5%3
  36. Jev 1.1465.3%66.7%3
  37. celeris-1-decision70.0%68.9%0
  38. Elephant-168.4%68.1%0
  39. Cygnet80.5%80.4%0
  40. Drex 2.067.6%68.9%0

Perplexity Metask vLLM-SR Liquid Laya REAX Strands Inception Fastino SubQ Clef meraGPT Microsoft Open-Jev Drex OneJev GPT Mammoth Winnow Jev Celeris Elephant CygnetThe upper bar: the mean probability of the chosen choice on right answers, in the family's colour The lower bar: the same on wrong answers At ≥99%: answers at that probability or more, over the 3 runs

SetupRight answersmean probability of the chosen choiceWrong answersthe sameAt ≥99%answers over the 3 runsOf them wrong
Perplexity Decider 27B v1.2 API · with web Open · 27B93.4%73.5%169runs 56 / 56 / 570
Metask-Jev-4B Open · 4.5B78.2%67.9%0runs 0 / 0 / 00
Perplexity Decider Flash v1.2 Open · 4.0B69.9%62.2%0runs 0 / 0 / 00
d3 Open · 26B68.0%63.3%6runs 2 / 2 / 20
d1-ultra Closed · Efficient69.8%64.4%0runs 0 / 0 / 00
Laya Open · 0.4B71.5%68.4%0runs 0 / 0 / 00
Perplexity Decider 27B v1.2 DGX Spark Open · 27B68.2%61.8%3runs 1 / 1 / 10
d3-ultra Open · 26B81.5%75.7%6runs 2 / 2 / 20
REAX 1.3 Open · 57B68.6%63.3%0runs 0 / 0 / 00
Strands Decider 2B Open · 2.0B73.4%67.3%0runs 0 / 0 / 00
d1-2B Open · 1.9B63.4%61.9%0runs 0 / 0 / 00
Laya Multilingual Open · 0.3B77.3%71.7%6runs 2 / 2 / 23
Mercury Decide Closed · Efficient68.2%61.4%6runs 2 / 2 / 23
d1.1 Closed · Efficient68.3%64.3%0runs 0 / 0 / 00
GLiDE v1.2 Open · 27B68.9%66.2%0runs 0 / 0 / 00
SubQ Snap 1.1 Closed · Efficient75.4%64.4%0runs 0 / 0 / 00
SubQ Snap Preview Closed · Efficient75.4%64.4%0runs 0 / 0 / 00
Clef Open · 27B66.5%69.7%0runs 0 / 0 / 00
Perplexity Decider 27B v1.2 API Open · 27B70.0%59.0%6runs 2 / 2 / 20
Decider 1 Closed · Efficient61.3%59.3%0runs 0 / 0 / 00
Microsoft-Decision-1 Closed · Efficient67.3%64.9%0runs 0 / 0 / 00
Open-Jev-27B-v1.1 Open · 27B54.3%52.2%0runs 0 / 0 / 00
Clef-Flash Open · 9.0B71.9%67.9%0runs 0 / 0 / 00
d1 Closed · Efficient70.3%67.5%4runs 2 / 1 / 13
Drex Max v1.1 Closed · Efficient (provisional)71.9%71.4%3runs 1 / 1 / 10
OneJev-27B Open · 27B59.5%60.7%0runs 0 / 0 / 00
Drex Max Open · 27B83.1%80.8%36runs 12 / 12 / 1221
GPT-6 Luna Decisions API Closed · Efficient71.0%67.2%18runs 6 / 6 / 69
GLiNER2.5-multi-Decide Open · 0.3B62.3%62.6%3runs 1 / 1 / 13
Mammoth-1 RTX PRO 6000 Blackwell Open · 27B68.3%71.0%0runs 0 / 0 / 00
Winnow-12B Open · 12B76.1%77.0%0runs 0 / 0 / 00
Mammoth-1 Baseten Open · 27B70.1%69.9%0runs 0 / 0 / 00
Jev 1.13 Closed · Efficient66.6%70.5%3runs 1 / 1 / 13
Jev 1.14 Closed · Efficient65.3%66.7%3runs 1 / 1 / 13
celeris-1-decision Closed · Efficient70.0%68.9%0runs 0 / 0 / 00
Elephant-1 Open · 9.0B68.4%68.1%0runs 0 / 0 / 00
Cygnet Open · 12B80.5%80.4%0runs 0 / 0 / 00
Drex 2.0 Open · 35B MoE · 3.0B active67.6%68.9%0runs 0 / 0 / 00

The probability each setup's output gave its chosen choice, from the per-choice probabilities recorded in its runs (not the providers' own "confidence" field, which differs from setup to setup). The answers at ≥99% are the “sure” answers of the table above.

5. Reversed choices

12 decision setups without search answered the 100 questions once more with each question's choices in reverse order, each answer mapped back to the original choice before scoring; the main scores and the ranking are unchanged. 9 of the 12 scored within 3 points of their run 1; Strands Decider 2B (-10.0), REAX 1.3 (-8.0) and d1-ultra (-4.0) scored more than 3 points lower, consistent with sensitivity to the position of the choices.

  1. Strands Decider 2B55%45%-10.072%
  2. REAX 1.355%47%-8.087%
  3. d1-ultra58%54%-4.086%
  4. Metask-Jev-4B60%57%-3.079%
  5. Perplexity Decider 27B v1.2 DGX Spark57%55%-2.093%
  6. d358%57%-1.095%
  7. Laya57%56%-1.071%
  8. Clef52%52%+0.0100%
  9. d3-ultra55%55%+0.093%
  10. Laya Multilingual54%54%+0.078%
  11. d1-2B54%56%+2.091%
  12. Perplexity Decider Flash v1.259%62%+3.095%

Strands REAX Liquid Metask Perplexity vLLM-SR Laya ClefThe upper bar: run 1 in the normal order, in the family's colour The lower bar: the run with the choices reversed Change: reversed minus run 1, in points; same choice: the share of the questions where both chose the same original choice

SetupNormal orderrun 1Reversedone runChangepointsSame choice as run 1shareRuns 1 and 2 agreesame choice, shareMean of 3 normal runsthe published scoreErrors / missingreversed run
Strands Decider 2B Open · 2.0B55%45%-10.072%100%55.0%0 / 0
REAX 1.3 Open · 57B55%47%-8.087%100%55.0%0 / 0
d1-ultra Closed · Efficient58%54%-4.086%89%57.7%0 / 0
Metask-Jev-4B Open · 4.5B60%57%-3.079%100%60.0%0 / 0
Perplexity Decider 27B v1.2 DGX Spark Open · 27B57%55%-2.093%100%57.0%0 / 0
d3 Open · 26B58%57%-1.095%100%58.0%0 / 0
Laya Open · 0.4B57%56%-1.071%100%57.0%0 / 0
Clef Open · 27B52%52%+0.0100%100%52.0%0 / 0
d3-ultra Open · 26B55%55%+0.093%100%55.0%0 / 0
Laya Multilingual Open · 0.3B54%54%+0.078%100%54.0%0 / 0
d1-2B Open · 1.9B54%56%+2.091%100%54.0%0 / 0
Perplexity Decider Flash v1.2 Open · 4.0B59%62%+3.095%100%59.0%0 / 0
Perplexity Decider 27B v1.2 API · with web Open · 27BNot run: no API credit
Perplexity Decider 27B v1.2 API Open · 27BNot run: no API credit

Each figure is a total over the 100 questions. The reversed order is a single run; "Runs 1 and 2 agree" is the share of the questions where the published runs 1 and 2 chose the same choice, shown in its own column. The mean of the 3 normal runs is the published score, also in its own column. Errors: reversed requests without a successful answer; missing: questions with no answer in the reversed run.

6. Weights and question dates

  1. Cygnet105
  2. Laya44
  3. Laya Multilingual44
  4. Winnow-12B42
  5. Metask-Jev-4B41
  6. Open-Jev-27B-v1.140
  7. GLiNER2.5-multi-Decide35
  8. Clef32
  9. Strands Decider 2B28
  10. Clef-Flash26
  11. OneJev-27B17
  12. Perplexity Decider 27B v1.2 DGX Spark17
  13. Perplexity Decider Flash v1.217
  14. d311
  15. Elephant-111
  16. GLiDE v1.211
  17. Mammoth-1 RTX PRO 6000 Blackwell11
  18. d3-ultra7
  19. d1-2B4
  20. Drex Max3
  21. REAX 1.31 day after

Cygnet Laya Winnow Metask Open-Jev Fastino Clef Strands OneJev Perplexity vLLM-SR Elephant Mammoth Liquid Drex REAXDays from each open setup's weights revision to the questions' earliest public record A revision committed after that record: the days after it, with no bar

SetupHugging Face repositoryRevision we ranRevision committedUTCDays before the questions were public
Cygnet Open · 12Bgoogle/gemma-4-12B-itCygnet runs it unchanged, with its own sampling recipe707f0a32026-07-20105
Laya Open · 0.4Bconvaiinnovations/layac5d78732026-09-1944
Laya Multilingual Open · 0.3Bconvaiinnovations/layaits multilingual/ folderc5d78732026-09-1944
Winnow-12B Open · 12BEldanRing/Winnow-12BQ8_0 GGUF; revision from its manifest, file hash not checkedb2b14212026-09-2142
Metask-Jev-4B Open · 4.5Bwayfind/metask-jev-4b-policy-mixea20fe82026-09-2241
Open-Jev-27B-v1.1 Open · 27BZefanCai/Open-Jev-27B-v1.1on Qwen/Qwen3.8-27B at 1d4bf0f, 2026-08-1428cf7302026-09-2340
GLiNER2.5-multi-Decide Open · 0.3Bfastino/GLiNER2.5-multi-Decidea35a0cd2026-09-2835
Clef Open · 27BCloudflare/clef2f3de3d2026-10-0132
Strands Decider 2B Open · 2.0BStrandsAgents/strands-decider-2B-hobson-v212b52a622026-10-0528
Clef-Flash Open · 9.0BCloudflare/clef-flashweights 2026-09-30fde727a2026-10-0726
OneJev-27B Open · 27BOmniJev/OneJev-27Ba5e5f552026-10-1617
Perplexity Decider 27B v1.2 DGX Spark Open · 27Bperplexity-ai/pplx-decider-v1.2225c9132026-10-1617
Perplexity Decider Flash v1.2 Open · 4.0Bperplexity-ai/pplx-decider-flash-v1.26d771ad2026-10-1617
d3 Open · 26Bvllm-sr/d320f37b92026-10-2211
Elephant-1 Open · 9.0Bbaselabs/elephant-1b3f8a932026-10-2211
GLiDE v1.2 Open · 27Bfastino/GLiDEa7e45892026-10-2211
Mammoth-1 RTX PRO 6000 Blackwell Open · 27Bbaselabs/mammoth-161cec442026-10-2211
d3-ultra Open · 26Bvllm-sr/d3-ultra52ba9e72026-10-267
d1-2B Open · 1.9BLiquidAI/d1-2B10e76712026-10-294
Drex Max Open · 27Bnace-ai/drex-max-v1cac65e12026-10-303
REAX 1.3 Open · 57Bdecisionmodels/REAX-1.3fa4bf7e2026-11-031 day after

Earliest public records of the questions we can observe: the Hugging Face dataset parallel-web/pws-decisions-benchmark was created on 1 November 2026 at 23:49 UTC; its only commit, 64da0d8, of 2 November 2026 at 01:37 UTC, holds the questions. The GitHub repository's only commit, 0f403cf, of 2 November 2026 at 17:33 UTC, holds none. All but one of the 21 open setups' revisions were committed 3 to 105 days before; the revision of REAX 1.3 was committed 1 day after (commit dates as the Hugging Face API gave them, read 6 November 2026; files can be older). A revision's commit date is not the end of its training, which no maker here states, so the order of these dates shows neither use nor absence of use of the questions. These dates do not rule out earlier exposure to the questions or their sources; the API setups' training dates are unknown.

Cost

Estimated dollars per 1,000 questionsper 1M questions, the mean of 3 runs, log scale; with web search, then without, each cheapest first · lower is better

  1. With web search
  2. Perplexity Decider 27B v1.2 API · with web≤ $1.041≤ $1,041 incl. search
  3. Gemini 3.8 Flash high · chat · with web≥ $4.781≥ $4,781 tokens only
  4. GPT-6 Luna low · chat · CLI · with web$4.886$4,886 incl. search
  5. Gemini 4 Argon low · chat · with web≥ $6.088≥ $6,088 tokens only
  6. GPT-6.1 Terra high · chat · CLI · with web$24.072$24,072 incl. search
  7. Claude Sonnet 5.5 high · chat · CLI · with web$32.381$32,381 incl. search
  8. Claude Opus 5.5 high · chat · CLI · with web$50.805$50,805 incl. search
  9. GPT-6.1 Sol high · chat · CLI · with web$55.689$55,689 incl. search
  10. Claude Fable 5.1 high · chat · CLI · with web · 2 requests handed to Claude Opus 4.8, counted wrong$121.016$121,016 incl. search
  11. GPT-6 Astra high · chat · CLI · with web$177.498$177,498 incl. search
  12. Grok 4.7 high · chat · with web$210.897$210,897 incl. search
  13. Without search
  14. Perplexity Decider Flash v1.2$0.002$2.01
  15. d1$0.003$2.94
  16. d1.1$0.003$2.94
  17. Mercury Decide$0.004$3.57
  18. Perplexity Decider 27B v1.2 API$0.004$3.97
  19. Perplexity Decider 27B v1.2 DGX Spark$0.004$4.03
  20. Microsoft-Decision-1$0.006$6.11
  21. d1-ultra$0.006$6.17
  22. Decider 1$0.007$6.81
  23. GLiDE v1.2$0.007$6.90
  24. Elephant-1$0.007$7.15
  25. SubQ Snap 1.1$0.008$7.94
  26. Drex 2.0$0.008$8.11
  27. celeris-1-decision$0.008$8.46
  28. Clef-Flash$0.009$9.25
  29. SubQ Snap Preview$0.010$9.92
  30. Drex Max v1.1$0.012$11.90
  31. Drex Max$0.012$12.08
  32. Jev 1.13$0.018$18.01
  33. Jev 1.14$0.018$18.01
  34. Mammoth-1 Baseten$0.020$20.42
  35. Mammoth-1 RTX PRO 6000 Blackwell$0.020$20.43
  36. GPT-6 Luna Decisions API$0.022$21.93
  37. Cygnet$0.023$22.82
  38. Clef$0.058$58.40
  39. DeepSeek-V4.1-Flash reasoning off · chat · 1017$0.078$77.53
  40. GLM-5.3 low · chat$0.299$299
  41. GPT-6 Luna low · chat · CLI$0.307$307
  42. DeepSeek-V4.1-Flash reasoning on · chat · 1017$1.368$1,368
  43. Kimi K3.1 Pro low · chat$1.994$1,994
  44. Gemini 4 Argon low · chat$2.887$2,887
  45. GPT-6.1 Terra high · chat · CLI$3.359$3,359
  46. Claude Sonnet 5.5 high · chat · CLI$5.782$5,782
  47. Claude Opus 5.5 high · chat · CLI$10.940$10,940
  48. GPT-6.1 Sol high · chat · CLI$10.985$10,985
  49. Gemini 3.8 Flash high · chat$19.985$19,985
  50. Grok 4.7 high · chat$35.746$35,746
  51. Claude Fable 5.1 high · chat · CLI · 5 requests handed to Claude Opus 4.8, counted wrong$36.773$36,773
  52. GPT-6 Astra high · chat · CLI$41.488$41,488

Perplexity Liquid Inception Microsoft meraGPT Fastino Elephant SubQ Drex Celeris Clef Jev Mammoth GPT Cygnet DeepSeek GLM Kimi Gemini Claude Grok With web (striped) “chat” in a tag: a chat model, asked each question in the benchmark's own words, its answer read from its reply; the other rows are decision models. Every cost is an estimate: the list price × the tokens and searches our runs recorded, not a provider's bill; for a subscription run, what the same usage would cost on the API (API-equivalent estimate). Without web: each run's tokens at the setup's list price, no search fee. With web: the same, plus the provider's web search fee (“incl. search”). ≤ Upper-bound estimate: one $0.001 web search counted per request; Perplexity bills only completed searches. ≥ Lower bound: the API reports no search count, so the cost counts the tokens only. Each search per question would add Gemini 3.8 Flash · high · chat · with web: $14.000 per 1,000 questions ($14,000 per 1M); Gemini 4 Argon · low · chat · with web: $14.000 per 1,000 questions ($14,000 per 1M). Erk-14B · reasoning off · chat: no price: on our GPU, priced only where an API list price exists. Metask-Jev-4B: no price: on our GPU, priced only where an API list price exists. Qwen3-14B · reasoning off · chat: no price: on our GPU, priced only where an API list price exists. d3: no price: on our GPU, priced only where an API list price exists. Qwen3.8-27B · low · chat: no price: on our GPU, priced only where an API list price exists. Laya: no price: on our GPU, priced only where an API list price exists. d3-ultra: no price: on our GPU, priced only where an API list price exists. REAX 1.3: no price: on our GPU, priced only where an API list price exists. Strands Decider 2B: no price: on our GPU, priced only where an API list price exists. d1-2B: no price: on our GPU, priced only where an API list price exists. Gemma 4 31B · reasoning off · chat: no price: on our GPU, priced only where an API list price exists. Laya Multilingual: no price: on our GPU, priced only where an API list price exists. Open-Jev-27B-v1.1: no price: on our GPU, priced only where an API list price exists. OneJev-27B: no price: on our GPU, priced only where an API list price exists. GLiNER2.5-multi-Decide: no price: on our GPU, priced only where an API list price exists. Winnow-12B: no price: on our GPU, priced only where an API list price exists.

Latency

Median time per request (the bar) and 95th percentile by nearest rank (the line), one question per request, from our machine, the mean of 3 runs, log scale; with web search, then without, each fastest first · lower is better

  1. With web search
  2. Perplexity Decider 27B v1.2 API · with web483 · 728 ms
  3. GPT-6 Luna low · chat · CLI · with web5,905 · 13,315 ms
  4. GPT-6 Astra high · chat · CLI · with web9,482 · 22,548 ms
  5. GPT-6.1 Terra high · chat · CLI · with web9,727 · 17,746 ms
  6. Gemini 3.8 Flash high · chat · with web11,615 · 46,567 ms
  7. Gemini 4 Argon low · chat · with web12,216 · 88,370 ms
  8. GPT-6.1 Sol high · chat · CLI · with web12,860 · 29,712 ms
  9. Claude Sonnet 5.5 high · chat · CLI · with web13,948 · 35,296 ms
  10. Claude Opus 5.5 high · chat · CLI · with web16,231 · 38,925 ms
  11. Grok 4.7 high · chat · with web16,629 · 67,866 ms
  12. Claude Fable 5.1 high · chat · CLI · with web · 2 requests handed to Claude Opus 4.8, counted wrong17,589 · 52,579 ms
  13. Without search
  14. GLiNER2.5-multi-Decide8 · 9 ms
  15. Laya Multilingual9 · 11 ms
  16. Laya11 · 12 ms
  17. d1-2B11 · 12 ms
  18. Strands Decider 2B24 · 28 ms
  19. Cygnet27 · 29 ms
  20. Perplexity Decider Flash v1.227 · 30 ms
  21. Winnow-12B48 · 59 ms
  22. Elephant-155 · 63 ms
  23. d363 · 77 ms
  24. GLiDE v1.264 · 71 ms
  25. Drex Max64 · 78 ms
  26. Metask-Jev-4B74 · 79 ms
  27. Perplexity Decider 27B v1.2 DGX Spark74 · 98 ms
  28. Clef-Flash85 · 109 ms
  29. OneJev-27B94 · 132 ms
  30. d3-ultra125 · 171 ms
  31. Clef126 · 154 ms
  32. Mammoth-1 RTX PRO 6000 Blackwell128 · 135 ms
  33. Open-Jev-27B-v1.1146 · 272 ms
  34. Jev 1.14165 · 203 ms
  35. Jev 1.13170 · 242 ms
  36. d1.1180 · 415 ms
  37. Mammoth-1 Baseten184 · 255 ms
  38. d1185 · 425 ms
  39. REAX 1.3215 · 355 ms
  40. d1-ultra220 · 560 ms
  41. Perplexity Decider 27B v1.2 API244 · 656 ms
  42. Erk-14B reasoning off · chat255 · 260 ms
  43. GPT-6 Luna Decisions API285 · 522 ms
  44. Mercury Decide291 · 365 ms
  45. Drex 2.0293 · 1,507 ms
  46. Microsoft-Decision-1317 · 385 ms
  47. Gemma 4 31B reasoning off · chat321 · 335 ms
  48. Drex Max v1.1331 · 470 ms
  49. Qwen3-14B reasoning off · chat366 · 391 ms
  50. SubQ Snap 1.1378 · 390 ms
  51. SubQ Snap Preview380 · 392 ms
  52. celeris-1-decision384 · 389 ms
  53. Decider 1629 · 690 ms
  54. DeepSeek-V4.1-Flash reasoning off · chat · 1017944 · 1,270 ms
  55. GLM-5.3 low · chat1,412 · 2,467 ms
  56. Gemini 4 Argon low · chat3,484 · 12,776 ms
  57. GPT-6 Luna low · chat · CLI3,981 · 5,971 ms
  58. Claude Opus 5.5 high · chat · CLI5,190 · 9,312 ms
  59. Kimi K3.1 Pro low · chat5,312 · 11,414 ms
  60. Claude Sonnet 5.5 high · chat · CLI5,368 · 10,007 ms
  61. Qwen3.8-27B low · chat5,636 · 16,380 ms
  62. GPT-6.1 Terra high · chat · CLI6,217 · 10,543 ms
  63. Claude Fable 5.1 high · chat · CLI · 5 requests handed to Claude Opus 4.8, counted wrong7,880 · 18,796 ms
  64. GPT-6 Astra high · chat · CLI10,229 · 26,512 ms
  65. GPT-6.1 Sol high · chat · CLI15,665 · 43,525 ms
  66. DeepSeek-V4.1-Flash reasoning on · chat · 101716,474 · 70,100 ms
  67. Gemini 3.8 Flash high · chat22,523 · 167,428 ms
  68. Grok 4.7 high · chat57,796 · 343,779 ms

Fastino Laya Liquid Strands Cygnet Perplexity Winnow Elephant vLLM-SR Drex Metask Clef OneJev Mammoth Open-Jev Jev REAX Erk GPT Inception Microsoft Gemma Qwen SubQ Celeris meraGPT DeepSeek GLM Gemini Claude Kimi Grok With web (striped) “chat” in a tag: a chat model, asked each question in the benchmark's own words, its answer read from its reply; the other rows are decision models. 95th percentile Request times include the network and any retries.

Parallel Web Systems' 100 public English questions on dated web facts; not part of the Turkish ranking.

More about this page

Every number, setup by setup

Ranked by mean accuracy. Each value is the mean of the 3 runs, with the lowest and highest run under it; under accuracy also the 95% interval from the question sample. Request times are from our machine. Probabilities: where the API returns them, the mean probability it gave the answer it chose, and how many answers given at 99% or more (the site's “sure” threshold) were wrong, over the 3 runs.

SetupAccuracymean of 3 runs; under it the runs' range and the 95% interval from the question sample; higher is betterMedian requestms; lower is better95th percentile requestms, nearest rank; lower is betterCostper 1,000 questionsper 1M questions, estimate at list prices; lower is betterProbability of the chosen answermean; neither better nor worse on its ownWrong when sureanswers at ≥99%, wrong of all, over the 3 runs; lower is better
GPT-6.1 Sol high · chat · CLI · with web Closed · WorkhorseCodex CLI, ChatGPT subscription; gpt-6.1-solThe Codex CLI adds its own context items to every request: 4,278 to 4,284 more input tokens than OpenAI's Responses API took for the same question (257 there; 62 questions compared), counted in the cost.100.0%each run95% interval 100.0% to 100.0% (question sample)12,86012,412 to 13,34029,71228,092 to 32,691$55.689$55,689$52.561$52,561 to $59.243$59,243API-equivalent estimate; includes the web search fee0.9940.993 to 0.994stated by the model0of 282
Gemini 4 Argon low · chat · with web Closed · FlagshipGemini API; gemini-4-argon99.7%runs 99.0% to 100.0%95% interval 99.0% to 100.0% (question sample)12,2169,827 to 14,21488,37065,312 to 112,346≥ $6.088≥ $6,088≥ $5.798≥ $5,798 to ≥ $6.528≥ $6,528API usage cost estimate; tokens only: the API reports no search count; the search fee is missing: the API reports no search count, so the estimate covers the tokens only0.9920.991 to 0.992stated by the model0of 278
GPT-6 Astra high · chat · CLI · with web Closed · FlagshipCodex CLI, ChatGPT subscription; gpt-6-astraThe Codex CLI adds its own context items to every request: 4,278 to 4,284 more input tokens than OpenAI's Responses API took for the same question (257 there; 62 questions compared), counted in the cost.99.7%runs 99.0% to 100.0%95% interval 99.0% to 100.0% (question sample)9,4829,350 to 9,71322,54821,831 to 23,130$177.498$177,498$170.856$170,856 to $188.995$188,995API-equivalent estimate; includes the web search fee0.9920.991 to 0.993stated by the model0of 275
Claude Fable 5.1 high · chat · CLI · with web · 2 requests handed to Claude Opus 4.8, counted wrong Closed · FlagshipClaude Code CLI, Claude subscription; claude-fable-5-1Claude Code writes the prompt to its 1-hour cache, counted in the cost; Claude Code runs each web search through a separate Claude Haiku 5.5 call, whose tokens are counted in the cost at Haiku 5.5's list price.99.0%runs 98.0% to 100.0%95% interval 97.3% to 100.0% (question sample)17,58915,933 to 18,51252,57944,186 to 68,262$121.016$121,016$109.661$109,661 to $140.392$140,392API-equivalent estimate; includes the web search fee0.8690.866 to 0.872stated by the model0of 1
GPT-6.1 Terra high · chat · CLI · with web Closed · WorkhorseCodex CLI, ChatGPT subscription; gpt-6.1-terraThe Codex CLI adds its own context items to every request: 4,278 to 4,284 more input tokens than OpenAI's Responses API took for the same question (257 there; 62 questions compared), counted in the cost.99.0%each run95% interval 97.0% to 100.0% (question sample)9,7279,532 to 9,98417,74616,387 to 18,560$24.072$24,072$23.906$23,906 to $24.294$24,294API-equivalent estimate; includes the web search fee0.9920.991 to 0.992stated by the model3of 282
Gemini 3.8 Flash high · chat · with web Closed · WorkhorseGemini API; gemini-3.8-flash98.3%runs 98.0% to 99.0%95% interval 95.7% to 100.0% (question sample)11,61511,293 to 11,92546,56738,792 to 55,200≥ $4.781≥ $4,781≥ $4.512≥ $4,512 to ≥ $5.113≥ $5,113API usage cost estimate; tokens only: the API reports no search count; the search fee is missing: the API reports no search count, so the estimate covers the tokens only0.9910.990 to 0.992stated by the model0of 245
Grok 4.7 high · chat · with web Closed · WorkhorsexAI Responses API, SuperGrok subscription; grok-4.798.3%runs 97.0% to 100.0%95% interval 96.0% to 100.0% (question sample)16,62915,385 to 18,24767,86657,031 to 75,709$210.897$210,897$204.995$204,995 to $222.148$222,148API-equivalent estimate; includes the web search fee0.9420.940 to 0.945stated by the model0of 6
Claude Opus 5.5 high · chat · CLI · with web Closed · FlagshipClaude Code CLI, Claude subscription; claude-opus-5-5Claude Code writes the prompt to its 1-hour cache, counted in the cost; Claude Code runs each web search through a separate Claude Haiku 5.5 call, whose tokens are counted in the cost at Haiku 5.5's list price.98.0%runs 97.0% to 99.0%95% interval 95.7% to 99.7% (question sample)16,23115,409 to 17,02738,92537,159 to 40,753$50.805$50,805$46.468$46,468 to $55.918$55,918API-equivalent estimate; includes the web search fee0.8460.840 to 0.854stated by the modelNo answers at ≥99%
Claude Sonnet 5.5 high · chat · CLI · with web Closed · WorkhorseClaude Code CLI, Claude subscription; claude-sonnet-5-5Claude Code writes the prompt to its 1-hour cache, counted in the cost; Claude Code runs each web search through a separate Claude Haiku 5.5 call, whose tokens are counted in the cost at Haiku 5.5's list price.96.3%runs 95.0% to 98.0%95% interval 93.3% to 98.7% (question sample)13,94813,705 to 14,11435,29628,917 to 38,923$32.381$32,381$31.559$31,559 to $33.132$33,132API-equivalent estimate; includes the web search fee0.845each runstated by the modelNo answers at ≥99%
Perplexity Decider 27B v1.2 API · with web Open · 27BPerplexity Decisions API; pplx-decider-v1.287.0%each run95% interval 80.0% to 93.0% (question sample)483475 to 491728619 to 876≤ $1.041≤ $1,041each runAPI usage cost estimate; includes the web search fee0.9080.906 to 0.9110of 169
Claude Sonnet 5.5 high · chat · CLI Closed · WorkhorseClaude Code CLI, Claude subscription; claude-sonnet-5-5Claude Code writes the prompt to its 1-hour cache, counted in the cost.80.0%runs 77.0% to 83.0%95% interval 72.7% to 86.7% (question sample)5,3685,225 to 5,65010,0079,277 to 11,018$5.782$5,782$5.053$5,053 to $7.152$7,152API-equivalent estimate; no search fee0.7140.713 to 0.715stated by the modelNo answers at ≥99%
Claude Opus 5.5 high · chat · CLI Closed · FlagshipClaude Code CLI, Claude subscription; claude-opus-5-5Claude Code writes the prompt to its 1-hour cache, counted in the cost.78.7%runs 77.0% to 81.0%95% interval 70.7% to 86.0% (question sample)5,1904,996 to 5,4889,3129,115 to 9,416$10.940$10,940$9.510$9,510 to $13.689$13,689API-equivalent estimate; no search fee0.7090.708 to 0.712stated by the modelNo answers at ≥99%
GPT-6.1 Sol high · chat · CLI Closed · WorkhorseCodex CLI, ChatGPT subscription; gpt-6.1-solThe Codex CLI adds its own context items to every request: 4,278 to 4,284 more input tokens than OpenAI's Responses API took for the same question (257 there; 62 questions compared), counted in the cost.75.7%runs 74.0% to 77.0%95% interval 67.7% to 83.0% (question sample)15,66514,857 to 16,47243,52542,147 to 45,361$10.985$10,985$10.275$10,275 to $12.271$12,271API-equivalent estimate; no search fee0.8830.880 to 0.888stated by the model2of 83
GPT-6 Astra high · chat · CLI Closed · FlagshipCodex CLI, ChatGPT subscription; gpt-6-astraThe Codex CLI adds its own context items to every request: 4,278 to 4,284 more input tokens than OpenAI's Responses API took for the same question (257 there; 62 questions compared), counted in the cost.75.3%runs 74.0% to 77.0%95% interval 67.3% to 83.0% (question sample)10,22910,004 to 10,43026,51225,395 to 27,551$41.488$41,488$39.060$39,060 to $45.523$45,523API-equivalent estimate; no search fee0.8800.873 to 0.886stated by the model1of 74
Claude Fable 5.1 high · chat · CLI · 5 requests handed to Claude Opus 4.8, counted wrong Closed · FlagshipClaude Code CLI, Claude subscription; claude-fable-5-1Claude Code writes the prompt to its 1-hour cache, counted in the cost.73.7%runs 71.0% to 76.0%95% interval 66.0% to 81.0% (question sample)7,8807,278 to 8,66818,79618,218 to 19,320$36.773$36,773$32.015$32,015 to $44.432$44,432API-equivalent estimate; no search fee0.7020.699 to 0.704stated by the modelNo answers at ≥99%
GPT-6.1 Terra high · chat · CLI Closed · WorkhorseCodex CLI, ChatGPT subscription; gpt-6.1-terraThe Codex CLI adds its own context items to every request: 4,278 to 4,284 more input tokens than OpenAI's Responses API took for the same question (257 there; 62 questions compared), counted in the cost.73.0%runs 68.0% to 77.0%95% interval 65.0% to 80.7% (question sample)6,2176,131 to 6,31810,5439,865 to 11,153$3.359$3,359$3.165$3,165 to $3.512$3,512API-equivalent estimate; no search fee0.8290.828 to 0.831stated by the model0of 37
Gemini 4 Argon low · chat Closed · FlagshipGemini API; gemini-4-argon68.7%runs 67.0% to 70.0%95% interval 60.0% to 77.0% (question sample)3,4843,448 to 3,50512,7768,262 to 16,855$2.887$2,887$2.882$2,882 to $2.890$2,890API usage cost estimate; no search fee0.7180.714 to 0.723stated by the model0of 29
Kimi K3.1 Pro low · chat Open · 2.8T MoE · 104B activeOpenRouter chat API, Moonshot AI's endpoint (mxfp4), fallbacks off; moonshotai/kimi-k3.1-pro66.3%runs 66.0% to 67.0%95% interval 58.7% to 73.7% (question sample)5,3125,215 to 5,39611,41410,802 to 11,787$1.994$1,994$1.785$1,785 to $2.410$2,410API usage cost estimate; no search fee0.9170.911 to 0.923stated by the model17of 69
Erk-14B reasoning off · chat Open · 14BvLLM, self-hosted, BF16; erk-14b65.0%each run95% interval 55.0% to 74.0% (question sample)255246 to 271260248 to 284No priceon our GPU, priced only where an API list price exists0.7820.781 to 0.783No answers at ≥99%
GPT-6 Luna low · chat · CLI · with web Closed · EfficientCodex CLI, ChatGPT subscription; gpt-6-lunaThe Codex CLI adds its own context items to every request: 4,278 to 4,284 more input tokens than OpenAI's Responses API took for the same question (257 there; 62 questions compared), counted in the cost.63.7%runs 61.0% to 67.0%95% interval 55.7% to 71.3% (question sample)5,9054,404 to 8,39913,31511,754 to 15,715$4.886$4,886$4.452$4,452 to $5.478$5,478API-equivalent estimate; includes the web search fee0.9380.931 to 0.944stated by the model21of 127
GLM-5.3 low · chat Open · WorkhorseOpenRouter chat API, Fireworks endpoint, fallbacks off; z-ai/glm-5.361.0%runs 58.0% to 64.0%95% interval 53.0% to 68.3% (question sample)1,4121,387 to 1,4512,4672,328 to 2,569$0.299$299$0.270$270 to $0.352$352API usage cost estimate; no search fee0.7220.705 to 0.733stated by the modelNo answers at ≥99%
Metask-Jev-4B Open · 4.5BMetask-Jev server, self-hosted; metask-jev-4b-policy-mix60.0%each run95% interval 50.0% to 69.0% (question sample)7473 to 747979 to 80No priceon our GPU, priced only where an API list price exists0.741each runNo answers at ≥99%
DeepSeek-V4.1-Flash reasoning on · chat · 1017 Open · 552B MoE · 16B activeSGLang, self-hosted, published mixed precision; 1017 weights (revision 1bae0e1); deepseek-v4.1-flash-101759.3%runs 57.0% to 63.0%95% interval 51.0% to 67.3% (question sample)16,47415,086 to 18,74270,10064,277 to 74,066$1.368$1,368$1.288$1,288 to $1.434$1,434API-equivalent estimate; no search fee0.9990.999 to 1.000121of 296
Perplexity Decider Flash v1.2 Open · 4.0BPerplexity Decider server, self-hosted; pplx-decider-flash-v1.2picks the first choice in 7 of 10 questions; always picking the first choice scores 68%59.0%each run95% interval 50.0% to 69.0% (question sample)27each run3029 to 31$0.002$2.01each runlist-price estimate; no search fee0.668each runNo answers at ≥99%
Qwen3-14B reasoning off · chat Open · 15BvLLM, self-hosted, BF16; qwen3-14b59.0%each run95% interval 49.0% to 69.0% (question sample)366359 to 381391379 to 416No priceon our GPU, priced only where an API list price exists0.9560.955 to 0.95873of 202
d3 Open · 26BvLLM System One endpoint (nightly), self-hosted, BF16; vllm-sr/d358.0%each run95% interval 49.0% to 68.0% (question sample)63each run7776 to 77No priceon our GPU, priced only where an API list price exists0.660each run0of 6
d1-ultra Closed · EfficientLiquid AI decisions API; d1-ultra57.7%runs 56.0% to 59.0%95% interval 48.3% to 66.7% (question sample)220216 to 226560550 to 567$0.006$6.17each runAPI usage cost estimate; no search fee0.6750.671 to 0.680No answers at ≥99%
Qwen3.8-27B low · chat Open · 27BvLLM, self-hosted; qwen38-27b-bf1657.7%runs 57.0% to 58.0%95% interval 48.7% to 66.3% (question sample)5,6365,532 to 5,74616,38016,106 to 16,832No priceon our GPU, priced only where an API list price exists0.9980.997 to 0.998126of 291
Laya Open · 0.4BLaya server, self-hosted; laya57.0%each run95% interval 47.0% to 66.0% (question sample)11each run12each runNo priceon our GPU, priced only where an API list price exists0.702each runNo answers at ≥99%
Perplexity Decider 27B v1.2 DGX Spark Open · 27BPerplexity Decider server, self-hosted; pplx-decider-v1.257.0%each run95% interval 47.0% to 66.0% (question sample)74each run9898 to 99$0.004$4.03each runlist-price estimate; no search fee0.655each run0of 3
d3-ultra Open · 26Bd3 package server (transformers, the release code), self-hosted, BF16; vllm-sr/d3-ultra55.0%each run95% interval 45.0% to 65.0% (question sample)125each run171each runNo priceon our GPU, priced only where an API list price exists0.789each run0of 6
REAX 1.3 Open · 57BREAX Engine server (reax13 mode, the maker's release code), self-hosted, FP8; reax1355.0%each run95% interval 45.0% to 65.0% (question sample)215190 to 237355333 to 368No priceon our GPU, priced only where an API list price exists0.662each runNo answers at ≥99%
Strands Decider 2B Open · 2.0BStrands Decider server, self-hosted; strands-decider-2bpicks the first choice in 7 of 10 questions; always picking the first choice scores 68%55.0%each run95% interval 45.0% to 65.0% (question sample)2424 to 252827 to 30No priceon our GPU, priced only where an API list price exists0.707each runNo answers at ≥99%
d1-2B Open · 1.9Bd1 server (Liquid's model code in Transformers, the model card's call), self-hosted, BF16; LiquidAI/d1-2B54.0%each run95% interval 44.0% to 64.0% (question sample)11each run1212 to 13No priceon our GPU, priced only where an API list price exists0.627each runNo answers at ≥99%
Gemma 4 31B reasoning off · chat Open · 31BvLLM, self-hosted, FP8; gemma-4-31B-it54.0%each run95% interval 44.0% to 64.0% (question sample)321306 to 351335308 to 389No priceon our GPU, priced only where an API list price exists0.971each run94of 223
Laya Multilingual Open · 0.3BLaya server, self-hosted; laya-multilingualpicks the first choice in 6 of 10 questions; always picking the first choice scores 68%54.0%each run95% interval 44.0% to 64.0% (question sample)9each run1110 to 11No priceon our GPU, priced only where an API list price exists0.747each run3of 6
Mercury Decide Closed · EfficientOpenRouter System One API; inception/mercury-decide53.3%runs 53.0% to 54.0%95% interval 44.0% to 62.7% (question sample)291284 to 301365355 to 378$0.004$3.57$0.004$3.57 to $0.004$3.58API usage cost estimate; no search fee0.6500.647 to 0.6553of 6
d1.1 Closed · EfficientLiquid AI decisions API; d1.1picks the first choice in 6 of 10 questions; always picking the first choice scores 68%53.0%runs 51.0% to 56.0%95% interval 44.0% to 62.0% (question sample)180177 to 185415313 to 491$0.003$2.94each runAPI usage cost estimate; no search fee0.6640.657 to 0.676No answers at ≥99%
GLiDE v1.2 Open · 27BGLiDE server (Fastino's release code), self-hosted; glide-v1.253.0%each run95% interval 43.0% to 63.0% (question sample)6463 to 647167 to 76$0.007$6.90each runlist-price estimate; no search fee0.676each runNo answers at ≥99%
SubQ Snap 1.1 Closed · EfficientSubQ Decisions API (public preview); subq-snap-1.153.0%each run95% interval 43.0% to 63.0% (question sample)378377 to 379390387 to 393$0.008$7.94each runAPI usage cost estimate; no search fee0.702each runNo answers at ≥99%
SubQ Snap Preview Closed · EfficientSubQ Decisions API (public preview); subq-snap-preview53.0%each run95% interval 43.0% to 63.0% (question sample)380379 to 381392391 to 393$0.010$9.92each runAPI usage cost estimate; no search fee0.702each runNo answers at ≥99%
Clef Open · 27BClef server, self-hosted; clef52.0%each run95% interval 42.0% to 62.0% (question sample)126123 to 128154153 to 154$0.058$58.40each runlist-price estimate; no search fee0.680each runNo answers at ≥99%
Perplexity Decider 27B v1.2 API Open · 27BPerplexity Decisions API; pplx-decider-v1.250.7%runs 50.0% to 51.0%95% interval 41.0% to 60.0% (question sample)244240 to 249656396 to 877$0.004$3.97each runAPI usage cost estimate; no search fee0.6460.645 to 0.6460of 6
Grok 4.7 high · chat Closed · WorkhorsexAI Responses API, SuperGrok subscription; grok-4.750.3%runs 49.0% to 52.0%95% interval 41.7% to 59.3% (question sample)57,79650,653 to 67,045343,779320,800 to 362,754$35.746$35,746$35.185$35,185 to $36.525$36,525API-equivalent estimate; no search fee0.6560.650 to 0.660stated by the modelNo answers at ≥99%
Decider 1 Closed · EfficientmeraGPT System One API; decider-150.0%each run95% interval 40.0% to 60.0% (question sample)629624 to 635690680 to 706$0.007$6.81each runAPI usage cost estimate; no search fee0.603each runNo answers at ≥99%
Microsoft-Decision-1 Closed · EfficientOpenRouter System One API; microsoft/microsoft-decision-149.0%each run95% interval 39.3% to 58.7% (question sample)317314 to 319385379 to 392$0.006$6.11each runAPI usage cost estimate; no search fee0.6610.659 to 0.665No answers at ≥99%
Open-Jev-27B-v1.1 Open · 27BOpen-Jev server, self-hosted; open-jev-27b-v1.1picks the first choice in 5 of 10 questions; always picking the first choice scores 68%49.0%each run95% interval 39.0% to 59.0% (question sample)146145 to 147272271 to 272No priceon our GPU, priced only where an API list price exists0.532each runNo answers at ≥99%
Clef-Flash Open · 9.0BClef server, self-hosted; clef-flash48.0%each run95% interval 38.0% to 58.0% (question sample)8566 to 12110970 to 137$0.009$9.25each runlist-price estimate; no search fee0.698each runNo answers at ≥99%
d1 Closed · EfficientLiquid AI decisions API; d147.7%runs 47.0% to 49.0%95% interval 38.7% to 56.7% (question sample)185183 to 186425350 to 468$0.003$2.94each runAPI usage cost estimate; no search fee0.6880.682 to 0.6933of 4
Drex Max v1.1 Closed · Efficient (provisional)Nace.AI System One API; drex-max-v1.147.7%runs 47.0% to 48.0%95% interval 38.0% to 57.3% (question sample)331330 to 332470419 to 511$0.012$11.90each runAPI usage cost estimate; no search fee0.7170.716 to 0.7170of 3
Gemini 3.8 Flash high · chat Closed · WorkhorseGemini API; gemini-3.8-flash46.0%runs 43.0% to 48.0%95% interval 37.0% to 55.3% (question sample)22,52321,567 to 23,447167,428156,820 to 183,031$19.985$19,985$19.503$19,503 to $20.232$20,232API usage cost estimate; no search fee0.9360.935 to 0.938stated by the model16of 46
GPT-6 Luna low · chat · CLI Closed · EfficientCodex CLI, ChatGPT subscription; gpt-6-lunaThe Codex CLI adds its own context items to every request: 4,278 to 4,284 more input tokens than OpenAI's Responses API took for the same question (257 there; 62 questions compared), counted in the cost.45.3%runs 41.0% to 54.0%95% interval 37.0% to 54.0% (question sample)3,9813,884 to 4,0855,9715,504 to 6,493$0.307$307$0.280$280 to $0.360$360API-equivalent estimate; no search fee0.9040.894 to 0.919stated by the model42of 82
OneJev-27B Open · 27BOneJev server (qev, OmniJev's release code), self-hosted, BF16; OmniJev/OneJev-27B45.0%each run95% interval 35.0% to 55.0% (question sample)94each run132131 to 132No priceon our GPU, priced only where an API list price exists0.602each runNo answers at ≥99%
DeepSeek-V4.1-Flash reasoning off · chat · 1017 Open · 552B MoE · 16B activeSGLang, self-hosted, published mixed precision; 1017 weights (revision 1bae0e1); deepseek-v4.1-flash-101743.7%runs 43.0% to 45.0%95% interval 34.3% to 53.3% (question sample)944735 to 1,3391,270946 to 1,902$0.078$77.53$0.078$77.52 to $0.078$77.54API-equivalent estimate; no search fee0.7820.782 to 0.7838of 21
Drex Max Open · 27BDrex Max server (Nace's loader, drex_max), self-hosted, BF16; drex-max-v142.0%each run95% interval 32.0% to 52.0% (question sample)64each run78each run$0.012$12.08each runlist-price estimate; no search fee0.818each run21of 36
GPT-6 Luna Decisions API Closed · EfficientOpenAI Decisions API (public beta); gpt-6-luna42.0%each run95% interval 32.0% to 52.0% (question sample)285278 to 294522490 to 540$0.022$21.93each runAPI usage cost estimate; no search fee0.688each run9of 18
GLiNER2.5-multi-Decide Open · 0.3BGLiNER2 server, self-hosted; gliner2.5-multi-decidepicks the first choice in 4 of 10 questions; always picking the first choice scores 68%41.0%each run95% interval 32.0% to 51.0% (question sample)87 to 898 to 10No priceon our GPU, priced only where an API list price exists0.625each run3of 3
Mammoth-1 RTX PRO 6000 Blackwell Open · 27BBase Labs server (qev, the model card's loader), self-hosted, BF16; baselabs/mammoth-139.0%each run95% interval 30.0% to 49.0% (question sample)128127 to 128135134 to 135$0.020$20.43each runlist-price estimate; no search fee0.699each runNo answers at ≥99%
Winnow-12B Open · 12BWinnow server (llama.cpp), self-hosted; winnow-12b39.0%each run95% interval 29.0% to 49.0% (question sample)4847 to 485958 to 59No priceon our GPU, priced only where an API list price exists0.767each runNo answers at ≥99%
Mammoth-1 Baseten Open · 27BBase Labs server (qev, the model card's loader), self-hosted, BF16; decision-models/baseten-mammoth-27b-latest37.0%each run95% interval 28.0% to 47.0% (question sample)184184 to 185255198 to 354$0.020$20.42each runAPI usage cost estimate; no search fee0.699each runNo answers at ≥99%
Jev 1.13 Closed · EfficientTypeSafe System One API; jev-1.13.0picks the first choice in 3 of 10 questions; always picking the first choice scores 68%35.0%runs 34.0% to 36.0%95% interval 25.7% to 44.7% (question sample)170167 to 174242221 to 259$0.018$18.01each runAPI usage cost estimate; no search fee0.6920.690 to 0.6933of 3
Jev 1.14 Closed · EfficientTypeSafe System One API; jev-1.14.0picks the first choice in 3 of 10 questions; always picking the first choice scores 68%34.3%runs 34.0% to 35.0%95% interval 25.3% to 43.7% (question sample)165163 to 168203197 to 210$0.018$18.01each runAPI usage cost estimate; no search fee0.6620.659 to 0.6643of 3
celeris-1-decision Closed · EfficientCeleris System One API; celeris-1-decision33.3%runs 33.0% to 34.0%95% interval 24.3% to 42.7% (question sample)384383 to 384389each run$0.008$8.46each runAPI usage cost estimate; no search fee0.6930.692 to 0.693No answers at ≥99%
Elephant-1 Open · 9.0BBase Labs server (qev, the model card's loader), self-hosted, BF16; baselabs/elephant-133.0%each run95% interval 24.0% to 42.0% (question sample)5553 to 566361 to 67$0.007$7.15each runlist-price estimate; no search fee0.682each runNo answers at ≥99%
Cygnet Open · 12BCygnet decision server on vLLM, self-hosted; cygnet32.0%each run95% interval 23.0% to 41.0% (question sample)2724 to 332926 to 36$0.023$22.82each runlist-price estimate; no search fee0.8050.804 to 0.805No answers at ≥99%
Drex 2.0 Open · 35B MoE · 3.0B activeNace.AI System One API; drex-v2.0picks the first choice in 2 of 10 questions; always picking the first choice scores 68%31.0%each run95% interval 22.0% to 40.0% (question sample)293292 to 2941,5071,500 to 1,512$0.008$8.11each runAPI usage cost estimate; no search fee0.685each runNo answers at ≥99%

With “I don't know” as an extra choice

Perplexity Decider 27B v1.2, off and with web, asked the same 100 questions with “I don't know” offered as an extra choice, 3 runs each. Run on 5 November 2026. Non-hallucination: the share of the answers that were not right where the setup chose “I don't know” instead of a wrong answer. Each value is the mean of the runs, with the lowest and highest run under it.

SetupRight · wrong · I don't knowmean of the runsAccuracyright / questions; higher is betterAnswered(right + wrong) / questionsWrong of answerswrong / (right + wrong); lower is betterNon-hallucinationI don't know / (I don't know + wrong); higher is betterErrorsrequests without an answer, scored wrong
Perplexity Decider 27B v1.2 API Open · 27B3 runs47.7 · 36 · 16.347.7%47.0% to 48.0%83.7%83.0% to 84.0%43.0%42.2% to 44.0%31.2%30.2% to 32.7%0each run
Perplexity Decider 27B v1.2 API · with web Open · 27B3 runs84.3 · 8.3 · 7.384.3%84.0% to 85.0%92.7%92.0% to 93.0%9.0%8.6% to 9.7%46.8%43.8% to 50.0%0each run

Not run, and why

Waiting

Our method

Each question went out as PWS-Decisions' own request (its state, and one choice question with its instructions and its choices), with the setup's model name, to the same API and endpoint as the setup's Text tab run. OpenAI's Decisions API takes the same parts under its own names. One question per request, each provider's requests one after another, the providers at the same time. Timeout 60 s; the runner's retries (up to 5, on 408, 425, 429 and 5xx, and transport failures). Scored with PWS-Decisions' own scorer: a refusal, an error or an invalid answer counts as wrong.

Download: pws-decisions.json (every number on this page, per run). The adapter is pws/bench.py in the repository.