Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Provisional: Drex Max v1.1, whose maker publishes only a promotional price. Left out here: the setups only this site ran (Erk-14B, GLiNER2.5 Multi, GLiNER2.5-multi-Decide, Laya, Laya Multilingual and Qwen3-14B); they show under Class: All. On this page a class narrows the rows of the tables and bar charts; the sentences, counts, tests and line plots still cover every model of the type in view.
Experiment
PWS-Decisions: the site's models on dated web facts
Decision models
Perplexity Decider 27B v1.2 · API · with web
87.0%
with web search
The same model without web
50.7%
Perplexity Decider 27B v1.2 · API
Best decision model without search
60.0%
Metask-Jev-4B
Accuracy
Share of the 100 public questions answered right, the mean of our 3 runs per setup · higher is better; beside it the cost (per 1,000 questionsper 1M questions) and the median and 95th percentile request time
GPT Gemini Claude Grok Perplexity Kimi Erk GLM Metask DeepSeek Qwen vLLM-SR Liquid Laya REAX Strands Gemma Inception Fastino SubQ Clef meraGPT Microsoft Open-Jev Drex OneJev Mammoth Winnow Jev Celeris Elephant Cygnet With web: the setup searched the web for each question (striped) “chat” in a tag: a chat model, asked each question in the benchmark's own words, its answer read from its reply; the other rows are decision models. The chance line: Parallel's random-guess baseline, 44.67% (the benchmark's README), uniform guessing over each question's choices, a property of the questions, not a run of ours. The dashed line: always picking the first option scores 68%, as the right answer is the first choice on 68 of the 100 questions. A line under a setup's name: its score is at most 1 point above what its answering habits alone would score (without knowing the facts).
Can a model score here without knowing the facts?
Always choosing the first option scores 68%, above every decision setup without search here. Answer-position and yes/true imbalances complicate the ranking; these checks cannot establish whether a model has seen the questions.
The right answer is the first choice on 68 questions, the second on 26, the third on 5 and the fourth on 1. 60 questions are yes/no (53) or true/false (7); on every one "yes" or "true" is the first choice, and "yes" or "true" is right on 48 of them (42 of 53 and 6 of 7). The checks below use only totals: the answer key's positions, each decision setup's answers over its 3 runs, and public dates.
1. Strategies that never read the question
MetaskA strategy that never reads the question Uniform random guessing on the same questions (44.7% on all 100; 50% on the 60 yes/no and true/false questions)
| Strategy | Scoreon the 100 questions | Uniform random guessingthe same questions |
|---|---|---|
| Always the first choice | 68% | 44.7% |
| Always the last choice | 19% | 44.7% |
| Always the position most often right in the key (the first) | 68% | 44.7% |
| Always "yes" or "true" on the 60 yes/no and true/false questions | 48 of 60 right (80%) | 50% on them |
| Always "yes" or "true" on those 60, a uniform random guess on the other 40 | 62.7% | 44.7% |
No decision setup without search reaches the 68% of always picking the first choice; the best, Metask-Jev-4B, scores 60.0%. 25 of the 37 score above uniform random guessing (44.7%).
2. Answering habits
Perplexity Metask vLLM-SR Liquid Laya REAX Strands Inception Fastino SubQ Clef meraGPT Microsoft Open-Jev Drex OneJev GPT Mammoth Winnow Jev Celeris Elephant CygnetThe upper bar: accuracy, the mean of the runs, in the family's colour The lower bar: the habit baseline, the score of the same habits without reading the question A line under a name: at or below habit baseline + 1 pt
| Setup | Accuracymean of 3 runs | First choiceshare of answers | Second choiceshare of answers | Third choiceshare of answers | Fourth choiceshare of answers | "Yes" or "true"share on the 60 yes/no and true/false questions | First choice, other questionsshare on the other 40 | Same habits, no readinghabit baseline | Above the habitspoints | pone-sided, exploratory | p, Holmover the 37 without search |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Answer key | 68% | 26% | 5% | 1% | 80% | 50% | |||||
| Perplexity Decider 27B v1.2 API · with web Open · 27B | 87.0% | 68.0% | 25.0% | 5.0% | 2.0% | 76.7% | 55.0% | 58.6% | +28.4 | 0.000 | with search: not tested |
| Metask-Jev-4B Open · 4.5B | 60.0% | 66.0% | 32.0% | 2.0% | 0.0% | 70.0% | 60.0% | 56.1% | +3.9 | 0.195 | 1.000 |
| Perplexity Decider Flash v1.2 Open · 4.0B | 59.0% | 66.0% | 26.0% | 5.0% | 3.0% | 78.3% | 47.5% | 58.2% | +0.8at or below habit baseline + 1 pt | 0.421 | 1.000 |
| d3 Open · 26B | 58.0% | 55.0% | 37.0% | 5.0% | 3.0% | 61.7% | 45.0% | 51.5% | +6.5 | 0.085 | 1.000 |
| d1-ultra Closed · Efficient | 57.7% | 53.0% | 39.0% | 2.0% | 6.0% | 57.8% | 45.8% | 47.6% | +10.0 | 0.018 | 0.662 |
| Laya Open · 0.4B | 57.0% | 59.0% | 32.0% | 8.0% | 1.0% | 60.0% | 57.5% | 51.7% | +5.3 | 0.128 | 1.000 |
| Perplexity Decider 27B v1.2 DGX Spark Open · 27B | 57.0% | 52.0% | 37.0% | 8.0% | 3.0% | 65.0% | 32.5% | 50.0% | +7.0 | 0.066 | 1.000 |
| d3-ultra Open · 26B | 55.0% | 48.0% | 44.0% | 6.0% | 2.0% | 55.0% | 37.5% | 48.0% | +7.0 | 0.075 | 1.000 |
| REAX 1.3 Open · 57B | 55.0% | 49.0% | 46.0% | 4.0% | 1.0% | 53.3% | 42.5% | 48.0% | +7.0 | 0.078 | 1.000 |
| Strands Decider 2B Open · 2.0B | 55.0% | 69.0% | 26.0% | 5.0% | 0.0% | 70.0% | 67.5% | 55.7% | -0.7at or below habit baseline + 1 pt | 0.563 | 1.000 |
| d1-2B Open · 1.9B | 54.0% | 55.0% | 37.0% | 7.0% | 1.0% | 61.7% | 45.0% | 50.1% | +3.9 | 0.207 | 1.000 |
| Laya Multilingual Open · 0.3B | 54.0% | 59.0% | 30.0% | 9.0% | 2.0% | 70.0% | 42.5% | 55.8% | -1.8at or below habit baseline + 1 pt | 0.657 | 1.000 |
| Mercury Decide Closed · Efficient | 53.3% | 51.7% | 38.3% | 6.0% | 4.0% | 56.1% | 45.0% | 48.3% | +5.0 | 0.150 | 1.000 |
| d1.1 Closed · Efficient | 53.0% | 62.7% | 28.7% | 6.0% | 2.7% | 66.1% | 57.5% | 53.2% | -0.2at or below habit baseline + 1 pt | 0.513 | 1.000 |
| GLiDE v1.2 Open · 27B | 53.0% | 47.0% | 45.0% | 5.0% | 3.0% | 56.7% | 32.5% | 46.9% | +6.1 | 0.101 | 1.000 |
| SubQ Snap 1.1 Closed · Efficient | 53.0% | 48.0% | 41.0% | 8.0% | 3.0% | 60.0% | 30.0% | 48.0% | +5.0 | 0.147 | 1.000 |
| SubQ Snap Preview Closed · Efficient | 53.0% | 48.0% | 41.0% | 8.0% | 3.0% | 60.0% | 30.0% | 48.0% | +5.0 | 0.147 | 1.000 |
| Clef Open · 27B | 52.0% | 40.0% | 52.0% | 5.0% | 3.0% | 38.3% | 42.5% | 41.3% | +10.7 | 0.013 | 0.481 |
| Perplexity Decider 27B v1.2 API Open · 27B | 50.7% | 47.7% | 44.0% | 5.3% | 3.0% | 56.7% | 34.2% | 48.2% | +2.4 | 0.304 | 1.000 |
| Decider 1 Closed · Efficient | 50.0% | 46.0% | 48.0% | 5.0% | 1.0% | 53.3% | 35.0% | 47.2% | +2.8 | 0.286 | 1.000 |
| Microsoft-Decision-1 Closed · Efficient | 49.0% | 40.7% | 54.3% | 3.0% | 2.0% | 41.1% | 40.0% | 44.0% | +5.0 | 0.150 | 1.000 |
| Open-Jev-27B-v1.1 Open · 27B | 49.0% | 53.0% | 37.0% | 8.0% | 2.0% | 58.3% | 45.0% | 49.0% | -0.0at or below habit baseline + 1 pt | 0.501 | 1.000 |
| Clef-Flash Open · 9.0B | 48.0% | 47.0% | 45.0% | 5.0% | 3.0% | 51.7% | 40.0% | 45.3% | +2.7 | 0.287 | 1.000 |
| d1 Closed · Efficient | 47.7% | 48.0% | 43.3% | 5.7% | 3.0% | 50.6% | 44.2% | 46.2% | +1.4 | 0.385 | 1.000 |
| Drex Max v1.1 Closed · Efficient (provisional) | 47.7% | 38.3% | 54.7% | 4.0% | 3.0% | 45.6% | 27.5% | 43.4% | +4.3 | 0.180 | 1.000 |
| OneJev-27B Open · 27B | 45.0% | 32.0% | 61.0% | 5.0% | 2.0% | 31.7% | 32.5% | 38.9% | +6.1 | 0.086 | 1.000 |
| Drex Max Open · 27B | 42.0% | 34.0% | 59.0% | 4.0% | 3.0% | 38.3% | 27.5% | 40.8% | +1.2 | 0.394 | 1.000 |
| GPT-6 Luna Decisions API Closed · Efficient | 42.0% | 33.0% | 58.0% | 2.0% | 7.0% | 45.0% | 15.0% | 38.3% | +3.7 | 0.201 | 1.000 |
| GLiNER2.5-multi-Decide Open · 0.3B | 41.0% | 39.0% | 51.0% | 9.0% | 1.0% | 36.7% | 42.5% | 41.7% | -0.7at or below habit baseline + 1 pt | 0.559 | 1.000 |
| Mammoth-1 RTX PRO 6000 Blackwell Open · 27B | 39.0% | 24.0% | 71.0% | 2.0% | 3.0% | 25.0% | 22.5% | 35.5% | +3.5 | 0.202 | 1.000 |
| Winnow-12B Open · 12B | 39.0% | 28.0% | 61.0% | 3.0% | 8.0% | 35.0% | 17.5% | 36.7% | +2.3 | 0.299 | 1.000 |
| Mammoth-1 Baseten Open · 27B | 37.0% | 23.0% | 72.0% | 2.0% | 3.0% | 23.3% | 22.5% | 33.9% | +3.1 | 0.230 | 1.000 |
| Jev 1.13 Closed · Efficient | 35.0% | 30.3% | 57.7% | 4.7% | 7.3% | 35.6% | 22.5% | 37.1% | -2.1at or below habit baseline + 1 pt | 0.678 | 1.000 |
| Jev 1.14 Closed · Efficient | 34.3% | 28.3% | 58.3% | 6.0% | 7.3% | 33.3% | 20.8% | 37.1% | -2.8at or below habit baseline + 1 pt | 0.731 | 1.000 |
| celeris-1-decision Closed · Efficient | 33.3% | 18.0% | 72.0% | 3.0% | 7.0% | 16.7% | 20.0% | 31.4% | +1.9 | 0.313 | 1.000 |
| Elephant-1 Open · 9.0B | 33.0% | 17.0% | 76.0% | 4.0% | 3.0% | 13.3% | 22.5% | 29.8% | +3.2 | 0.194 | 1.000 |
| Cygnet Open · 12B | 32.0% | 15.0% | 72.0% | 4.0% | 9.0% | 18.3% | 10.0% | 29.2% | +2.8 | 0.226 | 1.000 |
| Drex 2.0 Open · 35B MoE · 3.0B active | 31.0% | 19.0% | 72.0% | 5.0% | 4.0% | 18.3% | 20.0% | 31.7% | -0.7at or below habit baseline + 1 pt | 0.576 | 1.000 |
- Shares are of each setup's answers over its 3 runs. The habit baseline is the score of a strategy that never reads the question but has the setup's habits, by number of choices: on the 60 yes/no and true/false questions it says "yes" or "true" as often as the setup does there; on the other 17 two-choice questions it picks the first choice as often as the setup does on them; on three- and four-choice questions it picks each position as often as the setup does on questions of that size. "Above the habits" is accuracy minus that baseline.
- The p values are exploratory, not a validated test: the one-sided chance that the habit strategy, answering each question independently, scores at least the setup's mean accuracy, by a normal approximation with the variance of one run of 100 questions (the habits come from the same answers and are treated as known; the 3 runs of the same questions are not independent). Holm's correction over the 37 setups without search: lowest adjusted p 0.48. Across those 37, the share of first-choice answers and accuracy correlate at r = 0.93.
- "At or below habit baseline + 1 pt" marks the 9 setups whose accuracy is below their habit baseline or at most 1 point above it (a one-sided rule: no lower limit). It is a descriptive threshold: a score at or below that line is consistent with the habits, not proof that they cause it. The setup with web search never gets the mark. The ranking's note beside these setups gives their first-choice share over all 100 questions; since "yes" or "true" is always the first choice, that share mixes the "yes" habit with a pure position habit. The two columns before the baseline separate them.
3. Hard questions
Perplexity Metask vLLM-SR Liquid Laya REAX Strands Inception Fastino SubQ Clef meraGPT Microsoft Open-Jev Drex OneJev GPT Mammoth Winnow Jev Celeris Elephant CygnetUniform random guessing: 3.75 of the 10 expected right
| Setup | Right, runs 1 / 2 / 3of 10 | Meanof the runs | Right in all 3 runs |
|---|---|---|---|
| Perplexity Decider 27B v1.2 API · with web Open · 27B | 8 / 8 / 8 | 8.00 | 8 |
| Metask-Jev-4B Open · 4.5B | 3 / 3 / 3 | 3.00 | 3 |
| Perplexity Decider Flash v1.2 Open · 4.0B | 2 / 2 / 2 | 2.00 | 2 |
| d3 Open · 26B | 0 / 0 / 0 | 0.00 | 0 |
| d1-ultra Closed · Efficient | 1 / 1 / 1 | 1.00 | 1 |
| Laya Open · 0.4B | 2 / 2 / 2 | 2.00 | 2 |
| Perplexity Decider 27B v1.2 DGX Spark Open · 27B | 1 / 1 / 1 | 1.00 | 1 |
| d3-ultra Open · 26B | 1 / 1 / 1 | 1.00 | 1 |
| REAX 1.3 Open · 57B | 1 / 1 / 1 | 1.00 | 1 |
| Strands Decider 2B Open · 2.0B | 4 / 4 / 4 | 4.00 | 4 |
| d1-2B Open · 1.9B | 1 / 1 / 1 | 1.00 | 1 |
| Laya Multilingual Open · 0.3B | 3 / 3 / 3 | 3.00 | 3 |
| Mercury Decide Closed · Efficient | 3 / 2 / 2 | 2.33 | 1 |
| d1.1 Closed · Efficient | 2 / 1 / 1 | 1.33 | 1 |
| GLiDE v1.2 Open · 27B | 1 / 1 / 1 | 1.00 | 1 |
| SubQ Snap 1.1 Closed · Efficient | 0 / 0 / 0 | 0.00 | 0 |
| SubQ Snap Preview Closed · Efficient | 0 / 0 / 0 | 0.00 | 0 |
| Clef Open · 27B | 3 / 3 / 3 | 3.00 | 3 |
| Perplexity Decider 27B v1.2 API Open · 27B | 0 / 1 / 1 | 0.67 | 0 |
| Decider 1 Closed · Efficient | 5 / 5 / 5 | 5.00 | 5 |
| Microsoft-Decision-1 Closed · Efficient | 1 / 0 / 1 | 0.67 | 0 |
| Open-Jev-27B-v1.1 Open · 27B | 1 / 1 / 1 | 1.00 | 1 |
| Clef-Flash Open · 9.0B | 0 / 0 / 0 | 0.00 | 0 |
| d1 Closed · Efficient | 0 / 0 / 0 | 0.00 | 0 |
| Drex Max v1.1 Closed · Efficient (provisional) | 1 / 1 / 1 | 1.00 | 1 |
| OneJev-27B Open · 27B | 0 / 0 / 0 | 0.00 | 0 |
| Drex Max Open · 27B | 0 / 0 / 0 | 0.00 | 0 |
| GPT-6 Luna Decisions API Closed · Efficient | 1 / 1 / 1 | 1.00 | 1 |
| GLiNER2.5-multi-Decide Open · 0.3B | 2 / 2 / 2 | 2.00 | 2 |
| Mammoth-1 RTX PRO 6000 Blackwell Open · 27B | 0 / 0 / 0 | 0.00 | 0 |
| Winnow-12B Open · 12B | 1 / 1 / 1 | 1.00 | 1 |
| Mammoth-1 Baseten Open · 27B | 0 / 0 / 0 | 0.00 | 0 |
| Jev 1.13 Closed · Efficient | 0 / 0 / 0 | 0.00 | 0 |
| Jev 1.14 Closed · Efficient | 0 / 0 / 0 | 0.00 | 0 |
| celeris-1-decision Closed · Efficient | 0 / 0 / 0 | 0.00 | 0 |
| Elephant-1 Open · 9.0B | 0 / 0 / 0 | 0.00 | 0 |
| Cygnet Open · 12B | 1 / 1 / 1 | 1.00 | 1 |
| Drex 2.0 Open · 35B MoE · 3.0B active | 0 / 0 / 0 | 0.00 | 0 |
Hard: answered right by at most 20% of the 37 decision setups without search (each setup's mean over its runs). 10 questions (5 with 2 choices and 5 with 4 choices); uniform random guessing expects 3.75 of them right, and the right answer is the first choice on 7 of them. The hard set comes from the same setups' answers, so it is no independent contamination test.
4. Probability of the chosen choice
Perplexity Metask vLLM-SR Liquid Laya REAX Strands Inception Fastino SubQ Clef meraGPT Microsoft Open-Jev Drex OneJev GPT Mammoth Winnow Jev Celeris Elephant CygnetThe upper bar: the mean probability of the chosen choice on right answers, in the family's colour The lower bar: the same on wrong answers At ≥99%: answers at that probability or more, over the 3 runs
| Setup | Right answersmean probability of the chosen choice | Wrong answersthe same | At ≥99%answers over the 3 runs | Of them wrong |
|---|---|---|---|---|
| Perplexity Decider 27B v1.2 API · with web Open · 27B | 93.4% | 73.5% | 169runs 56 / 56 / 57 | 0 |
| Metask-Jev-4B Open · 4.5B | 78.2% | 67.9% | 0runs 0 / 0 / 0 | 0 |
| Perplexity Decider Flash v1.2 Open · 4.0B | 69.9% | 62.2% | 0runs 0 / 0 / 0 | 0 |
| d3 Open · 26B | 68.0% | 63.3% | 6runs 2 / 2 / 2 | 0 |
| d1-ultra Closed · Efficient | 69.8% | 64.4% | 0runs 0 / 0 / 0 | 0 |
| Laya Open · 0.4B | 71.5% | 68.4% | 0runs 0 / 0 / 0 | 0 |
| Perplexity Decider 27B v1.2 DGX Spark Open · 27B | 68.2% | 61.8% | 3runs 1 / 1 / 1 | 0 |
| d3-ultra Open · 26B | 81.5% | 75.7% | 6runs 2 / 2 / 2 | 0 |
| REAX 1.3 Open · 57B | 68.6% | 63.3% | 0runs 0 / 0 / 0 | 0 |
| Strands Decider 2B Open · 2.0B | 73.4% | 67.3% | 0runs 0 / 0 / 0 | 0 |
| d1-2B Open · 1.9B | 63.4% | 61.9% | 0runs 0 / 0 / 0 | 0 |
| Laya Multilingual Open · 0.3B | 77.3% | 71.7% | 6runs 2 / 2 / 2 | 3 |
| Mercury Decide Closed · Efficient | 68.2% | 61.4% | 6runs 2 / 2 / 2 | 3 |
| d1.1 Closed · Efficient | 68.3% | 64.3% | 0runs 0 / 0 / 0 | 0 |
| GLiDE v1.2 Open · 27B | 68.9% | 66.2% | 0runs 0 / 0 / 0 | 0 |
| SubQ Snap 1.1 Closed · Efficient | 75.4% | 64.4% | 0runs 0 / 0 / 0 | 0 |
| SubQ Snap Preview Closed · Efficient | 75.4% | 64.4% | 0runs 0 / 0 / 0 | 0 |
| Clef Open · 27B | 66.5% | 69.7% | 0runs 0 / 0 / 0 | 0 |
| Perplexity Decider 27B v1.2 API Open · 27B | 70.0% | 59.0% | 6runs 2 / 2 / 2 | 0 |
| Decider 1 Closed · Efficient | 61.3% | 59.3% | 0runs 0 / 0 / 0 | 0 |
| Microsoft-Decision-1 Closed · Efficient | 67.3% | 64.9% | 0runs 0 / 0 / 0 | 0 |
| Open-Jev-27B-v1.1 Open · 27B | 54.3% | 52.2% | 0runs 0 / 0 / 0 | 0 |
| Clef-Flash Open · 9.0B | 71.9% | 67.9% | 0runs 0 / 0 / 0 | 0 |
| d1 Closed · Efficient | 70.3% | 67.5% | 4runs 2 / 1 / 1 | 3 |
| Drex Max v1.1 Closed · Efficient (provisional) | 71.9% | 71.4% | 3runs 1 / 1 / 1 | 0 |
| OneJev-27B Open · 27B | 59.5% | 60.7% | 0runs 0 / 0 / 0 | 0 |
| Drex Max Open · 27B | 83.1% | 80.8% | 36runs 12 / 12 / 12 | 21 |
| GPT-6 Luna Decisions API Closed · Efficient | 71.0% | 67.2% | 18runs 6 / 6 / 6 | 9 |
| GLiNER2.5-multi-Decide Open · 0.3B | 62.3% | 62.6% | 3runs 1 / 1 / 1 | 3 |
| Mammoth-1 RTX PRO 6000 Blackwell Open · 27B | 68.3% | 71.0% | 0runs 0 / 0 / 0 | 0 |
| Winnow-12B Open · 12B | 76.1% | 77.0% | 0runs 0 / 0 / 0 | 0 |
| Mammoth-1 Baseten Open · 27B | 70.1% | 69.9% | 0runs 0 / 0 / 0 | 0 |
| Jev 1.13 Closed · Efficient | 66.6% | 70.5% | 3runs 1 / 1 / 1 | 3 |
| Jev 1.14 Closed · Efficient | 65.3% | 66.7% | 3runs 1 / 1 / 1 | 3 |
| celeris-1-decision Closed · Efficient | 70.0% | 68.9% | 0runs 0 / 0 / 0 | 0 |
| Elephant-1 Open · 9.0B | 68.4% | 68.1% | 0runs 0 / 0 / 0 | 0 |
| Cygnet Open · 12B | 80.5% | 80.4% | 0runs 0 / 0 / 0 | 0 |
| Drex 2.0 Open · 35B MoE · 3.0B active | 67.6% | 68.9% | 0runs 0 / 0 / 0 | 0 |
The probability each setup's output gave its chosen choice, from the per-choice probabilities recorded in its runs (not the providers' own "confidence" field, which differs from setup to setup). The answers at ≥99% are the “sure” answers of the table above.
5. Reversed choices
12 decision setups without search answered the 100 questions once more with each question's choices in reverse order, each answer mapped back to the original choice before scoring; the main scores and the ranking are unchanged. 9 of the 12 scored within 3 points of their run 1; Strands Decider 2B (-10.0), REAX 1.3 (-8.0) and d1-ultra (-4.0) scored more than 3 points lower, consistent with sensitivity to the position of the choices.
Strands REAX Liquid Metask Perplexity vLLM-SR Laya ClefThe upper bar: run 1 in the normal order, in the family's colour The lower bar: the run with the choices reversed Change: reversed minus run 1, in points; same choice: the share of the questions where both chose the same original choice
| Setup | Normal orderrun 1 | Reversedone run | Changepoints | Same choice as run 1share | Runs 1 and 2 agreesame choice, share | Mean of 3 normal runsthe published score | Errors / missingreversed run |
|---|---|---|---|---|---|---|---|
| Strands Decider 2B Open · 2.0B | 55% | 45% | -10.0 | 72% | 100% | 55.0% | 0 / 0 |
| REAX 1.3 Open · 57B | 55% | 47% | -8.0 | 87% | 100% | 55.0% | 0 / 0 |
| d1-ultra Closed · Efficient | 58% | 54% | -4.0 | 86% | 89% | 57.7% | 0 / 0 |
| Metask-Jev-4B Open · 4.5B | 60% | 57% | -3.0 | 79% | 100% | 60.0% | 0 / 0 |
| Perplexity Decider 27B v1.2 DGX Spark Open · 27B | 57% | 55% | -2.0 | 93% | 100% | 57.0% | 0 / 0 |
| d3 Open · 26B | 58% | 57% | -1.0 | 95% | 100% | 58.0% | 0 / 0 |
| Laya Open · 0.4B | 57% | 56% | -1.0 | 71% | 100% | 57.0% | 0 / 0 |
| Clef Open · 27B | 52% | 52% | +0.0 | 100% | 100% | 52.0% | 0 / 0 |
| d3-ultra Open · 26B | 55% | 55% | +0.0 | 93% | 100% | 55.0% | 0 / 0 |
| Laya Multilingual Open · 0.3B | 54% | 54% | +0.0 | 78% | 100% | 54.0% | 0 / 0 |
| d1-2B Open · 1.9B | 54% | 56% | +2.0 | 91% | 100% | 54.0% | 0 / 0 |
| Perplexity Decider Flash v1.2 Open · 4.0B | 59% | 62% | +3.0 | 95% | 100% | 59.0% | 0 / 0 |
| Perplexity Decider 27B v1.2 API · with web Open · 27B | Not run: no API credit | ||||||
| Perplexity Decider 27B v1.2 API Open · 27B | Not run: no API credit |
Each figure is a total over the 100 questions. The reversed order is a single run; "Runs 1 and 2 agree" is the share of the questions where the published runs 1 and 2 chose the same choice, shown in its own column. The mean of the 3 normal runs is the published score, also in its own column. Errors: reversed requests without a successful answer; missing: questions with no answer in the reversed run.
6. Weights and question dates
Cygnet Laya Winnow Metask Open-Jev Fastino Clef Strands OneJev Perplexity vLLM-SR Elephant Mammoth Liquid Drex REAXDays from each open setup's weights revision to the questions' earliest public record A revision committed after that record: the days after it, with no bar
| Setup | Hugging Face repository | Revision we ran | Revision committedUTC | Days before the questions were public |
|---|---|---|---|---|
| Cygnet Open · 12B | google/gemma-4-12B-itCygnet runs it unchanged, with its own sampling recipe | 707f0a3 | 2026-07-20 | 105 |
| Laya Open · 0.4B | convaiinnovations/laya | c5d7873 | 2026-09-19 | 44 |
| Laya Multilingual Open · 0.3B | convaiinnovations/layaits multilingual/ folder | c5d7873 | 2026-09-19 | 44 |
| Winnow-12B Open · 12B | EldanRing/Winnow-12BQ8_0 GGUF; revision from its manifest, file hash not checked | b2b1421 | 2026-09-21 | 42 |
| Metask-Jev-4B Open · 4.5B | wayfind/metask-jev-4b-policy-mix | ea20fe8 | 2026-09-22 | 41 |
| Open-Jev-27B-v1.1 Open · 27B | ZefanCai/Open-Jev-27B-v1.1on Qwen/Qwen3.8-27B at 1d4bf0f, 2026-08-14 | 28cf730 | 2026-09-23 | 40 |
| GLiNER2.5-multi-Decide Open · 0.3B | fastino/GLiNER2.5-multi-Decide | a35a0cd | 2026-09-28 | 35 |
| Clef Open · 27B | Cloudflare/clef | 2f3de3d | 2026-10-01 | 32 |
| Strands Decider 2B Open · 2.0B | StrandsAgents/strands-decider-2B-hobson-v21 | 2b52a62 | 2026-10-05 | 28 |
| Clef-Flash Open · 9.0B | Cloudflare/clef-flashweights 2026-09-30 | fde727a | 2026-10-07 | 26 |
| OneJev-27B Open · 27B | OmniJev/OneJev-27B | a5e5f55 | 2026-10-16 | 17 |
| Perplexity Decider 27B v1.2 DGX Spark Open · 27B | perplexity-ai/pplx-decider-v1.2 | 225c913 | 2026-10-16 | 17 |
| Perplexity Decider Flash v1.2 Open · 4.0B | perplexity-ai/pplx-decider-flash-v1.2 | 6d771ad | 2026-10-16 | 17 |
| d3 Open · 26B | vllm-sr/d3 | 20f37b9 | 2026-10-22 | 11 |
| Elephant-1 Open · 9.0B | baselabs/elephant-1 | b3f8a93 | 2026-10-22 | 11 |
| GLiDE v1.2 Open · 27B | fastino/GLiDE | a7e4589 | 2026-10-22 | 11 |
| Mammoth-1 RTX PRO 6000 Blackwell Open · 27B | baselabs/mammoth-1 | 61cec44 | 2026-10-22 | 11 |
| d3-ultra Open · 26B | vllm-sr/d3-ultra | 52ba9e7 | 2026-10-26 | 7 |
| d1-2B Open · 1.9B | LiquidAI/d1-2B | 10e7671 | 2026-10-29 | 4 |
| Drex Max Open · 27B | nace-ai/drex-max-v1 | cac65e1 | 2026-10-30 | 3 |
| REAX 1.3 Open · 57B | decisionmodels/REAX-1.3 | fa4bf7e | 2026-11-03 | 1 day after |
Earliest public records of the questions we can observe: the Hugging Face dataset parallel-web/pws-decisions-benchmark was created on 1 November 2026 at 23:49 UTC; its only commit, 64da0d8, of 2 November 2026 at 01:37 UTC, holds the questions. The GitHub repository's only commit, 0f403cf, of 2 November 2026 at 17:33 UTC, holds none. All but one of the 21 open setups' revisions were committed 3 to 105 days before; the revision of REAX 1.3 was committed 1 day after (commit dates as the Hugging Face API gave them, read 6 November 2026; files can be older). A revision's commit date is not the end of its training, which no maker here states, so the order of these dates shows neither use nor absence of use of the questions. These dates do not rule out earlier exposure to the questions or their sources; the API setups' training dates are unknown.
Can a model score here without knowing the facts? That section covers the decision models: see All models or Decision models.
Cost
Estimated dollars per 1,000 questionsper 1M questions, the mean of 3 runs, log scale; with web search, then without, each cheapest first · lower is better
Perplexity Liquid Inception Microsoft meraGPT Fastino Elephant SubQ Drex Celeris Clef Jev Mammoth GPT Cygnet DeepSeek GLM Kimi Gemini Claude Grok With web (striped) “chat” in a tag: a chat model, asked each question in the benchmark's own words, its answer read from its reply; the other rows are decision models. Every cost is an estimate: the list price × the tokens and searches our runs recorded, not a provider's bill; for a subscription run, what the same usage would cost on the API (API-equivalent estimate). Without web: each run's tokens at the setup's list price, no search fee. With web: the same, plus the provider's web search fee (“incl. search”). ≤ Upper-bound estimate: one $0.001 web search counted per request; Perplexity bills only completed searches. ≥ Lower bound: the API reports no search count, so the cost counts the tokens only. Each search per question would add Gemini 3.8 Flash · high · chat · with web: $14.000 per 1,000 questions ($14,000 per 1M); Gemini 4 Argon · low · chat · with web: $14.000 per 1,000 questions ($14,000 per 1M). Erk-14B · reasoning off · chat: no price: on our GPU, priced only where an API list price exists. Metask-Jev-4B: no price: on our GPU, priced only where an API list price exists. Qwen3-14B · reasoning off · chat: no price: on our GPU, priced only where an API list price exists. d3: no price: on our GPU, priced only where an API list price exists. Qwen3.8-27B · low · chat: no price: on our GPU, priced only where an API list price exists. Laya: no price: on our GPU, priced only where an API list price exists. d3-ultra: no price: on our GPU, priced only where an API list price exists. REAX 1.3: no price: on our GPU, priced only where an API list price exists. Strands Decider 2B: no price: on our GPU, priced only where an API list price exists. d1-2B: no price: on our GPU, priced only where an API list price exists. Gemma 4 31B · reasoning off · chat: no price: on our GPU, priced only where an API list price exists. Laya Multilingual: no price: on our GPU, priced only where an API list price exists. Open-Jev-27B-v1.1: no price: on our GPU, priced only where an API list price exists. OneJev-27B: no price: on our GPU, priced only where an API list price exists. GLiNER2.5-multi-Decide: no price: on our GPU, priced only where an API list price exists. Winnow-12B: no price: on our GPU, priced only where an API list price exists.
Latency
Median time per request (the bar) and 95th percentile by nearest rank (the line), one question per request, from our machine, the mean of 3 runs, log scale; with web search, then without, each fastest first · lower is better
Fastino Laya Liquid Strands Cygnet Perplexity Winnow Elephant vLLM-SR Drex Metask Clef OneJev Mammoth Open-Jev Jev REAX Erk GPT Inception Microsoft Gemma Qwen SubQ Celeris meraGPT DeepSeek GLM Gemini Claude Kimi Grok With web (striped) “chat” in a tag: a chat model, asked each question in the benchmark's own words, its answer read from its reply; the other rows are decision models. 95th percentile Request times include the network and any retries.
Parallel Web Systems' 100 public English questions on dated web facts; not part of the Turkish ranking.
Every number, setup by setup
Ranked by mean accuracy. Each value is the mean of the 3 runs, with the lowest and highest run under it; under accuracy also the 95% interval from the question sample. Request times are from our machine. Probabilities: where the API returns them, the mean probability it gave the answer it chose, and how many answers given at 99% or more (the site's “sure” threshold) were wrong, over the 3 runs.
| Setup | Accuracymean of 3 runs; under it the runs' range and the 95% interval from the question sample; higher is better | Median requestms; lower is better | 95th percentile requestms, nearest rank; lower is better | Costper 1,000 questionsper 1M questions, estimate at list prices; lower is better | Probability of the chosen answermean; neither better nor worse on its own | Wrong when sureanswers at ≥99%, wrong of all, over the 3 runs; lower is better |
|---|---|---|---|---|---|---|
GPT-6.1 Sol high · chat · CLI · with web Closed · WorkhorseCodex CLI, ChatGPT subscription; gpt-6.1-solThe Codex CLI adds its own context items to every request: 4,278 to 4,284 more input tokens than OpenAI's Responses API took for the same question (257 there; 62 questions compared), counted in the cost. | 100.0%each run95% interval 100.0% to 100.0% (question sample) | 12,86012,412 to 13,340 | 29,71228,092 to 32,691 | $55.689$55,689$52.561$52,561 to $59.243$59,243API-equivalent estimate; includes the web search fee | 0.9940.993 to 0.994stated by the model | 0of 282 |
Gemini 4 Argon low · chat · with web Closed · FlagshipGemini API; gemini-4-argon | 99.7%runs 99.0% to 100.0%95% interval 99.0% to 100.0% (question sample) | 12,2169,827 to 14,214 | 88,37065,312 to 112,346 | ≥ $6.088≥ $6,088≥ $5.798≥ $5,798 to ≥ $6.528≥ $6,528API usage cost estimate; tokens only: the API reports no search count; the search fee is missing: the API reports no search count, so the estimate covers the tokens only | 0.9920.991 to 0.992stated by the model | 0of 278 |
GPT-6 Astra high · chat · CLI · with web Closed · FlagshipCodex CLI, ChatGPT subscription; gpt-6-astraThe Codex CLI adds its own context items to every request: 4,278 to 4,284 more input tokens than OpenAI's Responses API took for the same question (257 there; 62 questions compared), counted in the cost. | 99.7%runs 99.0% to 100.0%95% interval 99.0% to 100.0% (question sample) | 9,4829,350 to 9,713 | 22,54821,831 to 23,130 | $177.498$177,498$170.856$170,856 to $188.995$188,995API-equivalent estimate; includes the web search fee | 0.9920.991 to 0.993stated by the model | 0of 275 |
Claude Fable 5.1 high · chat · CLI · with web · 2 requests handed to Claude Opus 4.8, counted wrong Closed · FlagshipClaude Code CLI, Claude subscription; claude-fable-5-1Claude Code writes the prompt to its 1-hour cache, counted in the cost; Claude Code runs each web search through a separate Claude Haiku 5.5 call, whose tokens are counted in the cost at Haiku 5.5's list price. | 99.0%runs 98.0% to 100.0%95% interval 97.3% to 100.0% (question sample) | 17,58915,933 to 18,512 | 52,57944,186 to 68,262 | $121.016$121,016$109.661$109,661 to $140.392$140,392API-equivalent estimate; includes the web search fee | 0.8690.866 to 0.872stated by the model | 0of 1 |
GPT-6.1 Terra high · chat · CLI · with web Closed · WorkhorseCodex CLI, ChatGPT subscription; gpt-6.1-terraThe Codex CLI adds its own context items to every request: 4,278 to 4,284 more input tokens than OpenAI's Responses API took for the same question (257 there; 62 questions compared), counted in the cost. | 99.0%each run95% interval 97.0% to 100.0% (question sample) | 9,7279,532 to 9,984 | 17,74616,387 to 18,560 | $24.072$24,072$23.906$23,906 to $24.294$24,294API-equivalent estimate; includes the web search fee | 0.9920.991 to 0.992stated by the model | 3of 282 |
Gemini 3.8 Flash high · chat · with web Closed · WorkhorseGemini API; gemini-3.8-flash | 98.3%runs 98.0% to 99.0%95% interval 95.7% to 100.0% (question sample) | 11,61511,293 to 11,925 | 46,56738,792 to 55,200 | ≥ $4.781≥ $4,781≥ $4.512≥ $4,512 to ≥ $5.113≥ $5,113API usage cost estimate; tokens only: the API reports no search count; the search fee is missing: the API reports no search count, so the estimate covers the tokens only | 0.9910.990 to 0.992stated by the model | 0of 245 |
Grok 4.7 high · chat · with web Closed · WorkhorsexAI Responses API, SuperGrok subscription; grok-4.7 | 98.3%runs 97.0% to 100.0%95% interval 96.0% to 100.0% (question sample) | 16,62915,385 to 18,247 | 67,86657,031 to 75,709 | $210.897$210,897$204.995$204,995 to $222.148$222,148API-equivalent estimate; includes the web search fee | 0.9420.940 to 0.945stated by the model | 0of 6 |
Claude Opus 5.5 high · chat · CLI · with web Closed · FlagshipClaude Code CLI, Claude subscription; claude-opus-5-5Claude Code writes the prompt to its 1-hour cache, counted in the cost; Claude Code runs each web search through a separate Claude Haiku 5.5 call, whose tokens are counted in the cost at Haiku 5.5's list price. | 98.0%runs 97.0% to 99.0%95% interval 95.7% to 99.7% (question sample) | 16,23115,409 to 17,027 | 38,92537,159 to 40,753 | $50.805$50,805$46.468$46,468 to $55.918$55,918API-equivalent estimate; includes the web search fee | 0.8460.840 to 0.854stated by the model | No answers at ≥99% |
Claude Sonnet 5.5 high · chat · CLI · with web Closed · WorkhorseClaude Code CLI, Claude subscription; claude-sonnet-5-5Claude Code writes the prompt to its 1-hour cache, counted in the cost; Claude Code runs each web search through a separate Claude Haiku 5.5 call, whose tokens are counted in the cost at Haiku 5.5's list price. | 96.3%runs 95.0% to 98.0%95% interval 93.3% to 98.7% (question sample) | 13,94813,705 to 14,114 | 35,29628,917 to 38,923 | $32.381$32,381$31.559$31,559 to $33.132$33,132API-equivalent estimate; includes the web search fee | 0.845each runstated by the model | No answers at ≥99% |
Perplexity Decider 27B v1.2 API · with web Open · 27BPerplexity Decisions API; pplx-decider-v1.2 | 87.0%each run95% interval 80.0% to 93.0% (question sample) | 483475 to 491 | 728619 to 876 | ≤ $1.041≤ $1,041each runAPI usage cost estimate; includes the web search fee | 0.9080.906 to 0.911 | 0of 169 |
Claude Sonnet 5.5 high · chat · CLI Closed · WorkhorseClaude Code CLI, Claude subscription; claude-sonnet-5-5Claude Code writes the prompt to its 1-hour cache, counted in the cost. | 80.0%runs 77.0% to 83.0%95% interval 72.7% to 86.7% (question sample) | 5,3685,225 to 5,650 | 10,0079,277 to 11,018 | $5.782$5,782$5.053$5,053 to $7.152$7,152API-equivalent estimate; no search fee | 0.7140.713 to 0.715stated by the model | No answers at ≥99% |
Claude Opus 5.5 high · chat · CLI Closed · FlagshipClaude Code CLI, Claude subscription; claude-opus-5-5Claude Code writes the prompt to its 1-hour cache, counted in the cost. | 78.7%runs 77.0% to 81.0%95% interval 70.7% to 86.0% (question sample) | 5,1904,996 to 5,488 | 9,3129,115 to 9,416 | $10.940$10,940$9.510$9,510 to $13.689$13,689API-equivalent estimate; no search fee | 0.7090.708 to 0.712stated by the model | No answers at ≥99% |
GPT-6.1 Sol high · chat · CLI Closed · WorkhorseCodex CLI, ChatGPT subscription; gpt-6.1-solThe Codex CLI adds its own context items to every request: 4,278 to 4,284 more input tokens than OpenAI's Responses API took for the same question (257 there; 62 questions compared), counted in the cost. | 75.7%runs 74.0% to 77.0%95% interval 67.7% to 83.0% (question sample) | 15,66514,857 to 16,472 | 43,52542,147 to 45,361 | $10.985$10,985$10.275$10,275 to $12.271$12,271API-equivalent estimate; no search fee | 0.8830.880 to 0.888stated by the model | 2of 83 |
GPT-6 Astra high · chat · CLI Closed · FlagshipCodex CLI, ChatGPT subscription; gpt-6-astraThe Codex CLI adds its own context items to every request: 4,278 to 4,284 more input tokens than OpenAI's Responses API took for the same question (257 there; 62 questions compared), counted in the cost. | 75.3%runs 74.0% to 77.0%95% interval 67.3% to 83.0% (question sample) | 10,22910,004 to 10,430 | 26,51225,395 to 27,551 | $41.488$41,488$39.060$39,060 to $45.523$45,523API-equivalent estimate; no search fee | 0.8800.873 to 0.886stated by the model | 1of 74 |
Claude Fable 5.1 high · chat · CLI · 5 requests handed to Claude Opus 4.8, counted wrong Closed · FlagshipClaude Code CLI, Claude subscription; claude-fable-5-1Claude Code writes the prompt to its 1-hour cache, counted in the cost. | 73.7%runs 71.0% to 76.0%95% interval 66.0% to 81.0% (question sample) | 7,8807,278 to 8,668 | 18,79618,218 to 19,320 | $36.773$36,773$32.015$32,015 to $44.432$44,432API-equivalent estimate; no search fee | 0.7020.699 to 0.704stated by the model | No answers at ≥99% |
GPT-6.1 Terra high · chat · CLI Closed · WorkhorseCodex CLI, ChatGPT subscription; gpt-6.1-terraThe Codex CLI adds its own context items to every request: 4,278 to 4,284 more input tokens than OpenAI's Responses API took for the same question (257 there; 62 questions compared), counted in the cost. | 73.0%runs 68.0% to 77.0%95% interval 65.0% to 80.7% (question sample) | 6,2176,131 to 6,318 | 10,5439,865 to 11,153 | $3.359$3,359$3.165$3,165 to $3.512$3,512API-equivalent estimate; no search fee | 0.8290.828 to 0.831stated by the model | 0of 37 |
Gemini 4 Argon low · chat Closed · FlagshipGemini API; gemini-4-argon | 68.7%runs 67.0% to 70.0%95% interval 60.0% to 77.0% (question sample) | 3,4843,448 to 3,505 | 12,7768,262 to 16,855 | $2.887$2,887$2.882$2,882 to $2.890$2,890API usage cost estimate; no search fee | 0.7180.714 to 0.723stated by the model | 0of 29 |
Kimi K3.1 Pro low · chat Open · 2.8T MoE · 104B activeOpenRouter chat API, Moonshot AI's endpoint (mxfp4), fallbacks off; moonshotai/kimi-k3.1-pro | 66.3%runs 66.0% to 67.0%95% interval 58.7% to 73.7% (question sample) | 5,3125,215 to 5,396 | 11,41410,802 to 11,787 | $1.994$1,994$1.785$1,785 to $2.410$2,410API usage cost estimate; no search fee | 0.9170.911 to 0.923stated by the model | 17of 69 |
Erk-14B reasoning off · chat Open · 14BvLLM, self-hosted, BF16; erk-14b | 65.0%each run95% interval 55.0% to 74.0% (question sample) | 255246 to 271 | 260248 to 284 | No priceon our GPU, priced only where an API list price exists | 0.7820.781 to 0.783 | No answers at ≥99% |
GPT-6 Luna low · chat · CLI · with web Closed · EfficientCodex CLI, ChatGPT subscription; gpt-6-lunaThe Codex CLI adds its own context items to every request: 4,278 to 4,284 more input tokens than OpenAI's Responses API took for the same question (257 there; 62 questions compared), counted in the cost. | 63.7%runs 61.0% to 67.0%95% interval 55.7% to 71.3% (question sample) | 5,9054,404 to 8,399 | 13,31511,754 to 15,715 | $4.886$4,886$4.452$4,452 to $5.478$5,478API-equivalent estimate; includes the web search fee | 0.9380.931 to 0.944stated by the model | 21of 127 |
GLM-5.3 low · chat Open · WorkhorseOpenRouter chat API, Fireworks endpoint, fallbacks off; z-ai/glm-5.3 | 61.0%runs 58.0% to 64.0%95% interval 53.0% to 68.3% (question sample) | 1,4121,387 to 1,451 | 2,4672,328 to 2,569 | $0.299$299$0.270$270 to $0.352$352API usage cost estimate; no search fee | 0.7220.705 to 0.733stated by the model | No answers at ≥99% |
Metask-Jev-4B Open · 4.5BMetask-Jev server, self-hosted; metask-jev-4b-policy-mix | 60.0%each run95% interval 50.0% to 69.0% (question sample) | 7473 to 74 | 7979 to 80 | No priceon our GPU, priced only where an API list price exists | 0.741each run | No answers at ≥99% |
DeepSeek-V4.1-Flash reasoning on · chat · 1017 Open · 552B MoE · 16B activeSGLang, self-hosted, published mixed precision; 1017 weights (revision 1bae0e1); deepseek-v4.1-flash-1017 | 59.3%runs 57.0% to 63.0%95% interval 51.0% to 67.3% (question sample) | 16,47415,086 to 18,742 | 70,10064,277 to 74,066 | $1.368$1,368$1.288$1,288 to $1.434$1,434API-equivalent estimate; no search fee | 0.9990.999 to 1.000 | 121of 296 |
Perplexity Decider Flash v1.2 Open · 4.0BPerplexity Decider server, self-hosted; pplx-decider-flash-v1.2picks the first choice in 7 of 10 questions; always picking the first choice scores 68% | 59.0%each run95% interval 50.0% to 69.0% (question sample) | 27each run | 3029 to 31 | $0.002$2.01each runlist-price estimate; no search fee | 0.668each run | No answers at ≥99% |
Qwen3-14B reasoning off · chat Open · 15BvLLM, self-hosted, BF16; qwen3-14b | 59.0%each run95% interval 49.0% to 69.0% (question sample) | 366359 to 381 | 391379 to 416 | No priceon our GPU, priced only where an API list price exists | 0.9560.955 to 0.958 | 73of 202 |
d3 Open · 26BvLLM System One endpoint (nightly), self-hosted, BF16; vllm-sr/d3 | 58.0%each run95% interval 49.0% to 68.0% (question sample) | 63each run | 7776 to 77 | No priceon our GPU, priced only where an API list price exists | 0.660each run | 0of 6 |
d1-ultra Closed · EfficientLiquid AI decisions API; d1-ultra | 57.7%runs 56.0% to 59.0%95% interval 48.3% to 66.7% (question sample) | 220216 to 226 | 560550 to 567 | $0.006$6.17each runAPI usage cost estimate; no search fee | 0.6750.671 to 0.680 | No answers at ≥99% |
Qwen3.8-27B low · chat Open · 27BvLLM, self-hosted; qwen38-27b-bf16 | 57.7%runs 57.0% to 58.0%95% interval 48.7% to 66.3% (question sample) | 5,6365,532 to 5,746 | 16,38016,106 to 16,832 | No priceon our GPU, priced only where an API list price exists | 0.9980.997 to 0.998 | 126of 291 |
Laya Open · 0.4BLaya server, self-hosted; laya | 57.0%each run95% interval 47.0% to 66.0% (question sample) | 11each run | 12each run | No priceon our GPU, priced only where an API list price exists | 0.702each run | No answers at ≥99% |
Perplexity Decider 27B v1.2 DGX Spark Open · 27BPerplexity Decider server, self-hosted; pplx-decider-v1.2 | 57.0%each run95% interval 47.0% to 66.0% (question sample) | 74each run | 9898 to 99 | $0.004$4.03each runlist-price estimate; no search fee | 0.655each run | 0of 3 |
d3-ultra Open · 26Bd3 package server (transformers, the release code), self-hosted, BF16; vllm-sr/d3-ultra | 55.0%each run95% interval 45.0% to 65.0% (question sample) | 125each run | 171each run | No priceon our GPU, priced only where an API list price exists | 0.789each run | 0of 6 |
REAX 1.3 Open · 57BREAX Engine server (reax13 mode, the maker's release code), self-hosted, FP8; reax13 | 55.0%each run95% interval 45.0% to 65.0% (question sample) | 215190 to 237 | 355333 to 368 | No priceon our GPU, priced only where an API list price exists | 0.662each run | No answers at ≥99% |
Strands Decider 2B Open · 2.0BStrands Decider server, self-hosted; strands-decider-2bpicks the first choice in 7 of 10 questions; always picking the first choice scores 68% | 55.0%each run95% interval 45.0% to 65.0% (question sample) | 2424 to 25 | 2827 to 30 | No priceon our GPU, priced only where an API list price exists | 0.707each run | No answers at ≥99% |
d1-2B Open · 1.9Bd1 server (Liquid's model code in Transformers, the model card's call), self-hosted, BF16; LiquidAI/d1-2B | 54.0%each run95% interval 44.0% to 64.0% (question sample) | 11each run | 1212 to 13 | No priceon our GPU, priced only where an API list price exists | 0.627each run | No answers at ≥99% |
Gemma 4 31B reasoning off · chat Open · 31BvLLM, self-hosted, FP8; gemma-4-31B-it | 54.0%each run95% interval 44.0% to 64.0% (question sample) | 321306 to 351 | 335308 to 389 | No priceon our GPU, priced only where an API list price exists | 0.971each run | 94of 223 |
Laya Multilingual Open · 0.3BLaya server, self-hosted; laya-multilingualpicks the first choice in 6 of 10 questions; always picking the first choice scores 68% | 54.0%each run95% interval 44.0% to 64.0% (question sample) | 9each run | 1110 to 11 | No priceon our GPU, priced only where an API list price exists | 0.747each run | 3of 6 |
Mercury Decide Closed · EfficientOpenRouter System One API; inception/mercury-decide | 53.3%runs 53.0% to 54.0%95% interval 44.0% to 62.7% (question sample) | 291284 to 301 | 365355 to 378 | $0.004$3.57$0.004$3.57 to $0.004$3.58API usage cost estimate; no search fee | 0.6500.647 to 0.655 | 3of 6 |
d1.1 Closed · EfficientLiquid AI decisions API; d1.1picks the first choice in 6 of 10 questions; always picking the first choice scores 68% | 53.0%runs 51.0% to 56.0%95% interval 44.0% to 62.0% (question sample) | 180177 to 185 | 415313 to 491 | $0.003$2.94each runAPI usage cost estimate; no search fee | 0.6640.657 to 0.676 | No answers at ≥99% |
GLiDE v1.2 Open · 27BGLiDE server (Fastino's release code), self-hosted; glide-v1.2 | 53.0%each run95% interval 43.0% to 63.0% (question sample) | 6463 to 64 | 7167 to 76 | $0.007$6.90each runlist-price estimate; no search fee | 0.676each run | No answers at ≥99% |
SubQ Snap 1.1 Closed · EfficientSubQ Decisions API (public preview); subq-snap-1.1 | 53.0%each run95% interval 43.0% to 63.0% (question sample) | 378377 to 379 | 390387 to 393 | $0.008$7.94each runAPI usage cost estimate; no search fee | 0.702each run | No answers at ≥99% |
SubQ Snap Preview Closed · EfficientSubQ Decisions API (public preview); subq-snap-preview | 53.0%each run95% interval 43.0% to 63.0% (question sample) | 380379 to 381 | 392391 to 393 | $0.010$9.92each runAPI usage cost estimate; no search fee | 0.702each run | No answers at ≥99% |
Clef Open · 27BClef server, self-hosted; clef | 52.0%each run95% interval 42.0% to 62.0% (question sample) | 126123 to 128 | 154153 to 154 | $0.058$58.40each runlist-price estimate; no search fee | 0.680each run | No answers at ≥99% |
Perplexity Decider 27B v1.2 API Open · 27BPerplexity Decisions API; pplx-decider-v1.2 | 50.7%runs 50.0% to 51.0%95% interval 41.0% to 60.0% (question sample) | 244240 to 249 | 656396 to 877 | $0.004$3.97each runAPI usage cost estimate; no search fee | 0.6460.645 to 0.646 | 0of 6 |
Grok 4.7 high · chat Closed · WorkhorsexAI Responses API, SuperGrok subscription; grok-4.7 | 50.3%runs 49.0% to 52.0%95% interval 41.7% to 59.3% (question sample) | 57,79650,653 to 67,045 | 343,779320,800 to 362,754 | $35.746$35,746$35.185$35,185 to $36.525$36,525API-equivalent estimate; no search fee | 0.6560.650 to 0.660stated by the model | No answers at ≥99% |
Decider 1 Closed · EfficientmeraGPT System One API; decider-1 | 50.0%each run95% interval 40.0% to 60.0% (question sample) | 629624 to 635 | 690680 to 706 | $0.007$6.81each runAPI usage cost estimate; no search fee | 0.603each run | No answers at ≥99% |
Microsoft-Decision-1 Closed · EfficientOpenRouter System One API; microsoft/microsoft-decision-1 | 49.0%each run95% interval 39.3% to 58.7% (question sample) | 317314 to 319 | 385379 to 392 | $0.006$6.11each runAPI usage cost estimate; no search fee | 0.6610.659 to 0.665 | No answers at ≥99% |
Open-Jev-27B-v1.1 Open · 27BOpen-Jev server, self-hosted; open-jev-27b-v1.1picks the first choice in 5 of 10 questions; always picking the first choice scores 68% | 49.0%each run95% interval 39.0% to 59.0% (question sample) | 146145 to 147 | 272271 to 272 | No priceon our GPU, priced only where an API list price exists | 0.532each run | No answers at ≥99% |
Clef-Flash Open · 9.0BClef server, self-hosted; clef-flash | 48.0%each run95% interval 38.0% to 58.0% (question sample) | 8566 to 121 | 10970 to 137 | $0.009$9.25each runlist-price estimate; no search fee | 0.698each run | No answers at ≥99% |
d1 Closed · EfficientLiquid AI decisions API; d1 | 47.7%runs 47.0% to 49.0%95% interval 38.7% to 56.7% (question sample) | 185183 to 186 | 425350 to 468 | $0.003$2.94each runAPI usage cost estimate; no search fee | 0.6880.682 to 0.693 | 3of 4 |
Drex Max v1.1 Closed · Efficient (provisional)Nace.AI System One API; drex-max-v1.1 | 47.7%runs 47.0% to 48.0%95% interval 38.0% to 57.3% (question sample) | 331330 to 332 | 470419 to 511 | $0.012$11.90each runAPI usage cost estimate; no search fee | 0.7170.716 to 0.717 | 0of 3 |
Gemini 3.8 Flash high · chat Closed · WorkhorseGemini API; gemini-3.8-flash | 46.0%runs 43.0% to 48.0%95% interval 37.0% to 55.3% (question sample) | 22,52321,567 to 23,447 | 167,428156,820 to 183,031 | $19.985$19,985$19.503$19,503 to $20.232$20,232API usage cost estimate; no search fee | 0.9360.935 to 0.938stated by the model | 16of 46 |
GPT-6 Luna low · chat · CLI Closed · EfficientCodex CLI, ChatGPT subscription; gpt-6-lunaThe Codex CLI adds its own context items to every request: 4,278 to 4,284 more input tokens than OpenAI's Responses API took for the same question (257 there; 62 questions compared), counted in the cost. | 45.3%runs 41.0% to 54.0%95% interval 37.0% to 54.0% (question sample) | 3,9813,884 to 4,085 | 5,9715,504 to 6,493 | $0.307$307$0.280$280 to $0.360$360API-equivalent estimate; no search fee | 0.9040.894 to 0.919stated by the model | 42of 82 |
OneJev-27B Open · 27BOneJev server (qev, OmniJev's release code), self-hosted, BF16; OmniJev/OneJev-27B | 45.0%each run95% interval 35.0% to 55.0% (question sample) | 94each run | 132131 to 132 | No priceon our GPU, priced only where an API list price exists | 0.602each run | No answers at ≥99% |
DeepSeek-V4.1-Flash reasoning off · chat · 1017 Open · 552B MoE · 16B activeSGLang, self-hosted, published mixed precision; 1017 weights (revision 1bae0e1); deepseek-v4.1-flash-1017 | 43.7%runs 43.0% to 45.0%95% interval 34.3% to 53.3% (question sample) | 944735 to 1,339 | 1,270946 to 1,902 | $0.078$77.53$0.078$77.52 to $0.078$77.54API-equivalent estimate; no search fee | 0.7820.782 to 0.783 | 8of 21 |
Drex Max Open · 27BDrex Max server (Nace's loader, drex_max), self-hosted, BF16; drex-max-v1 | 42.0%each run95% interval 32.0% to 52.0% (question sample) | 64each run | 78each run | $0.012$12.08each runlist-price estimate; no search fee | 0.818each run | 21of 36 |
GPT-6 Luna Decisions API Closed · EfficientOpenAI Decisions API (public beta); gpt-6-luna | 42.0%each run95% interval 32.0% to 52.0% (question sample) | 285278 to 294 | 522490 to 540 | $0.022$21.93each runAPI usage cost estimate; no search fee | 0.688each run | 9of 18 |
GLiNER2.5-multi-Decide Open · 0.3BGLiNER2 server, self-hosted; gliner2.5-multi-decidepicks the first choice in 4 of 10 questions; always picking the first choice scores 68% | 41.0%each run95% interval 32.0% to 51.0% (question sample) | 87 to 8 | 98 to 10 | No priceon our GPU, priced only where an API list price exists | 0.625each run | 3of 3 |
Mammoth-1 RTX PRO 6000 Blackwell Open · 27BBase Labs server (qev, the model card's loader), self-hosted, BF16; baselabs/mammoth-1 | 39.0%each run95% interval 30.0% to 49.0% (question sample) | 128127 to 128 | 135134 to 135 | $0.020$20.43each runlist-price estimate; no search fee | 0.699each run | No answers at ≥99% |
Winnow-12B Open · 12BWinnow server (llama.cpp), self-hosted; winnow-12b | 39.0%each run95% interval 29.0% to 49.0% (question sample) | 4847 to 48 | 5958 to 59 | No priceon our GPU, priced only where an API list price exists | 0.767each run | No answers at ≥99% |
Mammoth-1 Baseten Open · 27BBase Labs server (qev, the model card's loader), self-hosted, BF16; decision-models/baseten-mammoth-27b-latest | 37.0%each run95% interval 28.0% to 47.0% (question sample) | 184184 to 185 | 255198 to 354 | $0.020$20.42each runAPI usage cost estimate; no search fee | 0.699each run | No answers at ≥99% |
Jev 1.13 Closed · EfficientTypeSafe System One API; jev-1.13.0picks the first choice in 3 of 10 questions; always picking the first choice scores 68% | 35.0%runs 34.0% to 36.0%95% interval 25.7% to 44.7% (question sample) | 170167 to 174 | 242221 to 259 | $0.018$18.01each runAPI usage cost estimate; no search fee | 0.6920.690 to 0.693 | 3of 3 |
Jev 1.14 Closed · EfficientTypeSafe System One API; jev-1.14.0picks the first choice in 3 of 10 questions; always picking the first choice scores 68% | 34.3%runs 34.0% to 35.0%95% interval 25.3% to 43.7% (question sample) | 165163 to 168 | 203197 to 210 | $0.018$18.01each runAPI usage cost estimate; no search fee | 0.6620.659 to 0.664 | 3of 3 |
celeris-1-decision Closed · EfficientCeleris System One API; celeris-1-decision | 33.3%runs 33.0% to 34.0%95% interval 24.3% to 42.7% (question sample) | 384383 to 384 | 389each run | $0.008$8.46each runAPI usage cost estimate; no search fee | 0.6930.692 to 0.693 | No answers at ≥99% |
Elephant-1 Open · 9.0BBase Labs server (qev, the model card's loader), self-hosted, BF16; baselabs/elephant-1 | 33.0%each run95% interval 24.0% to 42.0% (question sample) | 5553 to 56 | 6361 to 67 | $0.007$7.15each runlist-price estimate; no search fee | 0.682each run | No answers at ≥99% |
Cygnet Open · 12BCygnet decision server on vLLM, self-hosted; cygnet | 32.0%each run95% interval 23.0% to 41.0% (question sample) | 2724 to 33 | 2926 to 36 | $0.023$22.82each runlist-price estimate; no search fee | 0.8050.804 to 0.805 | No answers at ≥99% |
Drex 2.0 Open · 35B MoE · 3.0B activeNace.AI System One API; drex-v2.0picks the first choice in 2 of 10 questions; always picking the first choice scores 68% | 31.0%each run95% interval 22.0% to 40.0% (question sample) | 293292 to 294 | 1,5071,500 to 1,512 | $0.008$8.11each runAPI usage cost estimate; no search fee | 0.685each run | No answers at ≥99% |
- ≤ Upper-bound estimate: one $0.001 web search counted per request; Perplexity bills only completed searches.
- ≥ Lower bound: the API reports no search count, so the cost counts the tokens only. Each search per question would add Gemini 4 Argon · low · chat · with web: $14.000 per 1,000 questions ($14,000 per 1M); Gemini 3.8 Flash · high · chat · with web: $14.000 per 1,000 questions ($14,000 per 1M).
- Retries over the three runs: Grok 4.7 · high · chat 1, Drex Max v1.1 8, Mammoth-1 · Baseten 3 and Drex 2.0 21; none for the other setups. Their request times include the retries. No request failed.
- Cost: an estimate, each run's recorded tokens at the runner's list price for the setup, the price the Text tab uses, over its 100 questions, plus the web search fee for setups with web ($0.001 per request for Perplexity). It is not a provider's bill: an API usage cost estimate for a run through an API key, an API-equivalent estimate for a run through a subscription, a list-price estimate for a model on our GPU. Where a fee is missing from the estimate, the setup's row says so.
- A line under a name, “picks the first choice in N of 10 questions; …”: that setup scores at most 1 point above what its answering habits alone would score (without knowing the facts).
With “I don't know” as an extra choice
Perplexity Decider 27B v1.2, off and with web, asked the same 100 questions with “I don't know” offered as an extra choice, 3 runs each. Run on 5 November 2026. Non-hallucination: the share of the answers that were not right where the setup chose “I don't know” instead of a wrong answer. Each value is the mean of the runs, with the lowest and highest run under it.
| Setup | Right · wrong · I don't knowmean of the runs | Accuracyright / questions; higher is better | Answered(right + wrong) / questions | Wrong of answerswrong / (right + wrong); lower is better | Non-hallucinationI don't know / (I don't know + wrong); higher is better | Errorsrequests without an answer, scored wrong |
|---|---|---|---|---|---|---|
| Perplexity Decider 27B v1.2 API Open · 27B3 runs | 47.7 · 36 · 16.3 | 47.7%47.0% to 48.0% | 83.7%83.0% to 84.0% | 43.0%42.2% to 44.0% | 31.2%30.2% to 32.7% | 0each run |
| Perplexity Decider 27B v1.2 API · with web Open · 27B3 runs | 84.3 · 8.3 · 7.3 | 84.3%84.0% to 85.0% | 92.7%92.0% to 93.0% | 9.0%8.6% to 9.7% | 46.8%43.8% to 50.0% | 0each run |
Not run, and why
- Perplexity Decider 27B v1. Perplexity's API now answers requests for v1 with v1.2. On five of these questions, each of its answers came within 0.007 of one of three v1.2 answers, while v1.2's own three answers differed by up to 0.085.
- Perplexity Decider 27B v1.1. Perplexity's API now answers requests for v1.1 with v1.2. On five of these questions, each of its answers matched one of three v1.2 answers exactly, while v1.2's own three answers differed by up to 0.085.
- GLiDE · API. Since 10 October Fastino's API answers with its glide_v2 engine, as it did again on 5 November, not the glide engine of the site's run.
- GPT-6 Luna with web. OpenAI's Decisions API documents no web option: a request takes only model, input, questions, prompt_cache_key, prompt_cache_options and safety_identifier (guide, reference, read 5 November 2026).
Waiting
- Parallel's own Decisions API: not run, no API key available to us yet.
- Decision models on our GPUs: running now, added when done (Copernicus-31B, KnowLine-31B-Gen1.5, KnowLine-31B-Gen2, Turion Jev 31B, Karpo Decision 31B, Logits Sigma-1 Duo, Sieve 31B v3, GLiDE 27B, Perplexity Decider 27B v1.3, KnowLine-12B-Gen1, Jeffan-9B-1211, KnowLine-9B-Gen1, Cerno 9B v1.2, KnowLine-9B-Gen2, Adjudex 9B v5, GLiDE 9B, Dexy-Decision-9B-V2, StartLux-Decision-v1.1-9B-Instant, Perplexity Decider Flash v1.3, vera-medium-omni, KnowLine 4B Gen5.5, StartLux-Decision-v1.1-4B-Instant, Valen-Decision-1.5-9B, Verdin-4B-v1.1, Chinese Decision Pointer 4B, KnowLine-2B-Gen1, GLiDE 2B, StartLux-Decision-v1.1-2B-Instant, GLiDE 0.8B, OneJev1.6-0.8B, StartLux-Decision-v1.1-0.8B-Instant, KnowLine-0.8B-Gen1, GLiClass v4 Edge and GLiDE 0.4B).
- GPT-6.1 Sol · chat · API: waiting for OpenAI API credit (the first attempt failed when the credit ran out and is kept apart).
- Claude Fable 5.1 · chat · API: not run: no Anthropic API key on our machine.
- Claude Opus 5.5 · chat · API: not run: no Anthropic API key on our machine.
- Claude Sonnet 5.5 · chat · API: not run: no Anthropic API key on our machine.
- GPT-6.1 Sol · chat · API · with web: waiting for OpenAI API credit (the first attempt failed when the credit ran out and is kept apart).
- Claude Fable 5.1 · chat · API · with web: not run: no Anthropic API key on our machine.
- Claude Opus 5.5 · chat · API · with web: not run: no Anthropic API key on our machine.
- Claude Sonnet 5.5 · chat · API · with web: not run: no Anthropic API key on our machine.
- Base Labs' Elephant-1 is also on Baseten's Model API, but not shown to be the same weights: on the first 20 Turkish questions (5 November 2026) Baseten's probabilities differed from our runs by up to 0.28. Not run on Baseten; our GPU run is on the page.
Our method
Each question went out as PWS-Decisions' own request (its state, and one choice question with its instructions and its choices), with the setup's model name, to the same API and endpoint as the setup's Text tab run. OpenAI's Decisions API takes the same parts under its own names. One question per request, each provider's requests one after another, the providers at the same time. Timeout 60 s; the runner's retries (up to 5, on 408, 425, 429 and 5xx, and transport failures). Scored with PWS-Decisions' own scorer: a refusal, an error or an invalid answer counts as wrong.
- Request time: from sending a request to its answer, retries included, over the requests that got an answer; the median and the 95th percentile by nearest rank.
- The chance line, 44.67%: the mean over the 100 questions of 1 / the number of choices, (77 × 1/2 + 5 × 1/3 + 18 × 1/4) / 100.
- The 95% interval: 10,000 resamples of the 100 questions with replacement (seed 20261105), each question scored as the mean of its 3 runs' correctness (a missing answer is wrong); the 2.5th and 97.5th percentiles of the resampled accuracy. It is the uncertainty from the question sample; the lowest to highest run is the run-to-run range.
- d1 was asked as
d1: Liquid's API no longer serves thed1:freename of its Text tab run. - Perplexity's own figures: 86% with web on these questions, with 370 ms at the median and 685 ms at the 95th percentile from Perplexity's client (Web grounding for the Decisions API, 4 November 2026); the 51% without is Parallel's leaderboard figure.
- Parallel's leaderboard (public set, mean of three runs, evaluated 29 October 2026): Parallel Decisions API (fast) 78.33%; Perplexity Decisions (pplx-decider-v1.2, through OpenRouter) 51.00%; OpenAI Decisions API (gpt-6-luna) 42.00%; Baseten Mammoth (baseten-mammoth-27b-latest) 37.00%; TypeSafe Jev 1.14 (jev-1.14.0) 33.33% (the benchmark's repository).
- The questions and answers are Parallel's: PWS-Decisions version 0.1.0-rc1, gated, for evaluation only. We publish totals only, never a question, an answer or a per-question result.
- The runs' API usage cost estimate at list prices (subscription runs as API-equivalent estimates): $258; not a provider's bill.
- GPT-6.1 Sol · high · chat · CLI · with web. The Codex CLI adds its own context items to every request: 4,278 to 4,284 more input tokens than OpenAI's Responses API took for the same question (257 there; 62 questions compared), counted in the cost.
- GPT-6 Astra · high · chat · CLI · with web. The Codex CLI adds its own context items to every request: 4,278 to 4,284 more input tokens than OpenAI's Responses API took for the same question (257 there; 62 questions compared), counted in the cost.
- Claude Fable 5.1 · high · chat · CLI · with web · 2 requests handed to Claude Opus 4.8, counted wrong. Claude Code writes the prompt to its 1-hour cache, counted in the cost; Claude Code runs each web search through a separate Claude Haiku 5.5 call, whose tokens are counted in the cost at Haiku 5.5's list price. Claude Code's safety stops passed 2 of its requests (0 + 1 + 1 over the runs, 1 question) to Claude Opus 4.8; the scorer counted each as a failure (a refusal), not with that model's answer.
- GPT-6.1 Terra · high · chat · CLI · with web. The Codex CLI adds its own context items to every request: 4,278 to 4,284 more input tokens than OpenAI's Responses API took for the same question (257 there; 62 questions compared), counted in the cost.
- Claude Opus 5.5 · high · chat · CLI · with web. Claude Code writes the prompt to its 1-hour cache, counted in the cost; Claude Code runs each web search through a separate Claude Haiku 5.5 call, whose tokens are counted in the cost at Haiku 5.5's list price.
- Claude Sonnet 5.5 · high · chat · CLI · with web. Claude Code writes the prompt to its 1-hour cache, counted in the cost; Claude Code runs each web search through a separate Claude Haiku 5.5 call, whose tokens are counted in the cost at Haiku 5.5's list price.
- Claude Sonnet 5.5 · high · chat · CLI. Claude Code writes the prompt to its 1-hour cache, counted in the cost.
- Claude Opus 5.5 · high · chat · CLI. Claude Code writes the prompt to its 1-hour cache, counted in the cost.
- GPT-6.1 Sol · high · chat · CLI. The Codex CLI adds its own context items to every request: 4,278 to 4,284 more input tokens than OpenAI's Responses API took for the same question (257 there; 62 questions compared), counted in the cost.
- GPT-6 Astra · high · chat · CLI. The Codex CLI adds its own context items to every request: 4,278 to 4,284 more input tokens than OpenAI's Responses API took for the same question (257 there; 62 questions compared), counted in the cost.
- Claude Fable 5.1 · high · chat · CLI · 5 requests handed to Claude Opus 4.8, counted wrong. Claude Code writes the prompt to its 1-hour cache, counted in the cost. Claude Code's safety stops passed 5 of its requests (2 + 1 + 2 over the runs, 2 questions) to Claude Opus 4.8; the scorer counted each as a failure (a refusal), not with that model's answer.
- GPT-6.1 Terra · high · chat · CLI. The Codex CLI adds its own context items to every request: 4,278 to 4,284 more input tokens than OpenAI's Responses API took for the same question (257 there; 62 questions compared), counted in the cost.
- GPT-6 Luna · low · chat · CLI · with web. The Codex CLI adds its own context items to every request: 4,278 to 4,284 more input tokens than OpenAI's Responses API took for the same question (257 there; 62 questions compared), counted in the cost.
- GPT-6 Luna · low · chat · CLI. The Codex CLI adds its own context items to every request: 4,278 to 4,284 more input tokens than OpenAI's Responses API took for the same question (257 there; 62 questions compared), counted in the cost.
- Attempts set aside whole, never merged into a run: GPT-6 Astra · chat, attempt-2: 3 runs, 300 answers, 297 errors, 3 right; GPT-6 Astra · chat, attempt-3: 3 runs, 205 answers, 203 errors, 1 right; GPT-6 Astra · chat, attempt-4: 2 runs, 132 answers, 128 errors, 2 right; GPT-6 Astra · chat · with web, attempt-2: 3 runs, 300 answers, 298 errors, 2 right; GPT-6 Astra · chat · with web, attempt-3: 3 runs, 210 answers, 208 errors, 2 right; GPT-6 Astra · chat · with web, attempt-4: 2 runs, 147 answers, 142 errors, 5 right; GPT-6.1 Sol · chat · API, attempt-1: 3 runs, 300 answers, 238 errors, 46 right; GPT-6.1 Sol · chat · API · with web, attempt-1: 3 runs, 300 answers, 218 errors, 82 right.
Download: pws-decisions.json (every number on this page, per run). The adapter is pws/bench.py in the repository.