Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Provisional: Drex Max v1.1, whose maker publishes only a promotional price. Left out here: the setups only this site ran (Erk-14B, GLiNER2.5 Multi, GLiNER2.5-multi-Decide, Laya, Laya Multilingual and Qwen3-14B); they show under Class: All. On this page a class narrows the rows of the tables and bar charts; the sentences, counts, tests and line plots still cover every model of the type in view.
Experiment
Web grounding: Perplexity Decider 27B v1.2 off and with web
Chat models: this page compares decision-model setups, so it shows under All models and Decision models. PWS-Decisions with chat models: its own page.
Turkish, 2,198 questions
66.4% → 68.2% (+1.8 points)
Mean of 3 runs; runs 66.3% to 66.5% off, runs 67.2% to 68.7% with web
$0.008$7.70 vs ≤ $1.034≤ $1,034 per 1,000 questionsper 1M questions
English reading, 547
88.5% → 89.7% (+1.2 points)
Mean of 2 runs; 88.5% each run off, runs 88.8% to 90.5% with web
$0.008$7.79 vs ≤ $1.036≤ $1,036 per 1,000 questionsper 1M questions
PWS-Decisions, 100
50.7% → 87.0% (+36.3 points)
Mean of 3 runs; runs 50.0% to 51.0% off, 87.0% each run with web
$0.004$3.97 vs ≤ $1.041≤ $1,041 per 1,000 questionsper 1M questions
Accuracy with web off → with web, the mean of the runs; cost web off vs with web. ≤ Upper-bound estimate: the search fee counted for every request with web.
Accuracy
Share of questions answered right, then the right answers; for a mean of several runs, the lowest to highest run under it; beside each set, the change with web in percentage points · higher is better
The Turkish full bank by part
The pilot, 200 questions, run first, one run each
Web off: Perplexity Decider 27B v1.2 API With web: Perplexity Decider 27B v1.2 API · with web Each heading names the questions both setups answered.
Non-hallucination
“I don't know”, in Turkish and in English, offered as an extra option: the bar is the option's share of the answers that were not right; beside it the times it was chosen, the answered share (right + wrong of the questions) and the wrong share of the answers · higher is better, read with the answered share. PWS-Decisions: “I don't know” as an extra choice, the mean of 3 runs per mode
Web off: Perplexity Decider 27B v1.2 API With web: Perplexity Decider 27B v1.2 API · with web Each heading names the questions both setups answered. A setup that always chose the option would reach 100% with no right answers.
Latency
Median time per request (the bar) and 95th percentile by nearest rank (the line), one question per request, from our machine; for a mean of several runs, the mean of the runs' values · lower is better
Web off: Perplexity Decider 27B v1.2 API With web: Perplexity Decider 27B v1.2 API · with web Each heading names the questions both setups answered. 95th percentile
Cost
Estimated dollars per 1,000 questionsper 1M questions, log scale: input tokens at $0.02 per 1M, plus $0.001 per request with web (incl. search); for a mean of several runs, the mean of the runs' costs · lower is better
Web off: Perplexity Decider 27B v1.2 API With web: Perplexity Decider 27B v1.2 API · with web Each heading names the questions both setups answered. ≤ Upper-bound estimate: one $0.001 web search counted per request; Perplexity bills only completed searches.
Perplexity's Decisions API without and with a web search per question.
PWS-Decisions: the decision models, with and without web
What this measures. Parallel Web Systems made PWS-Decisions. This is an end-to-end comparison of API setups with and without web search on its 100 public questions, which Parallel chose “for their difficulty and alignment with customer outcomes” (its methodology), around its customers' needs for current facts from the web. It is not a general ranking of models: without search, most setups score near random guessing, because the answers depend on dated web facts they were not given.
Each bar is the mean of our 3 runs per setup, decision models only; the PWS-Decisions rows of the charts above are the same means for Perplexity's two setups. Chat models, with and without their own web search, and every number: PWS-Decisions.
- Dataset
- PWS-Decisions 0.1.0-rc1, Hugging Face
parallel-web/pws-decisions-benchmarkat64da0d8a4ec0 - Scorer
- Parallel's own (
python -m pws_decisions score), unchanged,0f403cf99856 - Runs
- 3 per setup, 198 in all; 0 errors, 0 missing answers
- Run dates
- 5 November 2026 to 6 November 2026
- Published
- Totals only: no question, answer or per-question result
Accuracy
Share of the 100 public questions answered right, the mean of our 3 runs per setup · higher is better; beside it the cost (per 1,000 questionsper 1M questions) and the median and 95th percentile request time
Perplexity Metask vLLM-SR Liquid Laya REAX Strands Inception Fastino SubQ Clef meraGPT Microsoft Open-Jev Drex OneJev GPT Mammoth Winnow Jev Celeris Elephant Cygnet With web: the setup searched the web for each question (striped) The chance line: Parallel's random-guess baseline, 44.67% (the benchmark's README), uniform guessing over each question's choices, a property of the questions, not a run of ours.
Cost
Estimated dollars per 1,000 questionsper 1M questions, the mean of 3 runs, log scale; with web search, then without, each cheapest first · lower is better
Perplexity Liquid Inception Microsoft meraGPT Fastino Elephant SubQ Drex Celeris Clef Jev Mammoth GPT Cygnet With web (striped) Every cost is an estimate: the list price × the tokens and searches our runs recorded, not a provider's bill; for a subscription run, what the same usage would cost on the API (API-equivalent estimate). Without web: each run's tokens at the setup's list price, no search fee. With web: the same, plus the provider's web search fee (“incl. search”). ≤ Upper-bound estimate: one $0.001 web search counted per request; Perplexity bills only completed searches. Metask-Jev-4B: no price: on our GPU, priced only where an API list price exists. d3: no price: on our GPU, priced only where an API list price exists. Laya: no price: on our GPU, priced only where an API list price exists. d3-ultra: no price: on our GPU, priced only where an API list price exists. REAX 1.3: no price: on our GPU, priced only where an API list price exists. Strands Decider 2B: no price: on our GPU, priced only where an API list price exists. d1-2B: no price: on our GPU, priced only where an API list price exists. Laya Multilingual: no price: on our GPU, priced only where an API list price exists. Open-Jev-27B-v1.1: no price: on our GPU, priced only where an API list price exists. OneJev-27B: no price: on our GPU, priced only where an API list price exists. GLiNER2.5-multi-Decide: no price: on our GPU, priced only where an API list price exists. Winnow-12B: no price: on our GPU, priced only where an API list price exists.
Latency
Median time per request (the bar) and 95th percentile by nearest rank (the line), one question per request, from our machine, the mean of 3 runs, log scale; with web search, then without, each fastest first · lower is better
Fastino Laya Liquid Strands Cygnet Perplexity Winnow Elephant vLLM-SR Drex Metask Clef OneJev Mammoth Open-Jev Jev REAX GPT Inception Microsoft SubQ Celeris meraGPT With web (striped) 95th percentile Request times include the network and any retries.
What the search returned
In short. On the full bank, 388 to 414 questions per grounded run had 8+ consecutive question words in a returned title or snippet, and 123 to 159 got the source book itself. A word match alone does not prove that the answer was found.
Unit: returned pages, up to 5 per question; each site's count is out of the run's total pages.
The 8+ word check. A literal-match check: runs of 8 or more consecutive words of the question (its body and each option's text, without the option letters) in a returned page's title or snippet. It shows only such matches. Our translations are unpublished, but that does not prove the answer, or equivalent information, cannot be found on the web.
Turkish, full bank, first run (run 640)
10,989 pages for 2,198 questions (4 to 5 per question), from 966 sites. The most returned:
- scribd.com: 1,785 of 10,989 pages
- dergipark.org.tr: 1,070 of 10,989 pages
- dokuman.osym.gov.tr: 826 of 10,989 pages
- huggingface.co: 617 of 10,989 pages
- yksedebiyat.org: 372 of 10,989 pages
- youtube.com: 369 of 10,989 pages
- tr.wikipedia.org: 302 of 10,989 pages
- tdk.gov.tr: 297 of 10,989 pages
- turkedebiyati.org: 263 of 10,989 pages
- depo.pegem.net: 247 of 10,989 pages
414 questions had a literal match of 8+ words: 280 right off, 299 with web; 41 of the 147 gains and 22 of the 94 losses. 187 got a page from Paraf, the publisher of the bank's source book (138 right both ways). 125 got the source book itself, found by its title (99 right both ways).
Turkish pilot (run 632)
1,000 pages for 200 questions (5 per question), from 198 sites. The most returned:
- scribd.com: 162 of 1,000 pages
- dergipark.org.tr: 98 of 1,000 pages
- dokuman.osym.gov.tr: 71 of 1,000 pages
- huggingface.co: 61 of 1,000 pages
- youtube.com: 39 of 1,000 pages
- yksedebiyat.org: 36 of 1,000 pages
- depo.pegem.net: 28 of 1,000 pages
- ogmmateryal.eba.gov.tr: 26 of 1,000 pages
- turkedebiyati.org: 26 of 1,000 pages
- tr.wikipedia.org: 24 of 1,000 pages
36 questions had a literal match of 8+ words: 25 right off, 27 with web; 3 of the 9 gains and 1 of the 8 losses. 19 got a page from Paraf, the publisher of the bank's source book (15 right both ways). 15 got the source book itself, found by its title (12 right both ways).
English, first run (run 639)
2,734 pages for 547 questions (4 to 5 per question), from 1,025 sites. The most returned:
- scribd.com: 378 of 2,734 pages
- library.douglascollege.ca: 144 of 2,734 pages
- studocu.com: 108 of 2,734 pages
- agrigento2025.org: 56 of 2,734 pages
- unisq.edu.au: 52 of 2,734 pages
- en.wikipedia.org: 45 of 2,734 pages
- testbook.com: 39 of 2,734 pages
- pmc.ncbi.nlm.nih.gov: 38 of 2,734 pages
- hpbose.org: 36 of 2,734 pages
- cbseacademic.nic.in: 35 of 2,734 pages
8 of the 547 English questions had a literal match of 8+ words: 8 right off, 8 with web. None of the 13 English gains had the defined 8+ word overlap; this does not show that the answer, or its semantic equivalent, was absent from the returned pages.
Each grounded run
| Run with web | Pagesreturned, of up to 5 per question | Siteshost names | 8+ word matchquestions | A Paraf pagequestions | The source bookquestions |
|---|---|---|---|---|---|
| Turkish, full bank, run 1 of 3run 640, 2,198 questions | 10,989 | 966 | 414 | 187 | 125 |
| Turkish, full bank, run 2 of 3run 685, 2,198 questions | 10,990 | 953 | 406 | 179 | 123 |
| Turkish, full bank, run 3 of 3run 692, 2,198 questions | 10,990 | 918 | 388 | 194 | 159 |
| English, kept reading, run 1 of 2run 639, 547 questions | 2,734 | 1,025 | 8 | 0 | 0 |
| English, kept reading, run 2 of 2run 693, 547 questions | 2,734 | 1,029 | 9 | 0 | 0 |
N/A: the run returned no page, so it has no source checks. A Paraf page: questions with a page from the publisher of the bank's source book. The source book: questions with a page of the book itself, found by its title.
The 10 most returned sites, each run
Turkish, full bank, run 1 of 3 (run 640)
- scribd.com: 1,785 of 10,989 pages
- dergipark.org.tr: 1,070 of 10,989 pages
- dokuman.osym.gov.tr: 826 of 10,989 pages
- huggingface.co: 617 of 10,989 pages
- yksedebiyat.org: 372 of 10,989 pages
- youtube.com: 369 of 10,989 pages
- tr.wikipedia.org: 302 of 10,989 pages
- tdk.gov.tr: 297 of 10,989 pages
- turkedebiyati.org: 263 of 10,989 pages
- depo.pegem.net: 247 of 10,989 pages
Turkish, full bank, run 2 of 3 (run 685)
- scribd.com: 1,968 of 10,990 pages
- dergipark.org.tr: 1,066 of 10,990 pages
- dokuman.osym.gov.tr: 708 of 10,990 pages
- huggingface.co: 615 of 10,990 pages
- yksedebiyat.org: 426 of 10,990 pages
- youtube.com: 328 of 10,990 pages
- tdk.gov.tr: 305 of 10,990 pages
- tr.wikipedia.org: 290 of 10,990 pages
- kpsssorucoz.com: 279 of 10,990 pages
- turkedebiyati.org: 279 of 10,990 pages
Turkish, full bank, run 3 of 3 (run 692)
- scribd.com: 2,672 of 10,990 pages
- dergipark.org.tr: 1,283 of 10,990 pages
- huggingface.co: 691 of 10,990 pages
- kpsssorucoz.com: 490 of 10,990 pages
- tdk.gov.tr: 359 of 10,990 pages
- yksedebiyat.org: 332 of 10,990 pages
- youtube.com: 298 of 10,990 pages
- ogmmateryal.eba.gov.tr: 257 of 10,990 pages
- dokuman.osym.gov.tr: 252 of 10,990 pages
- tr.wikipedia.org: 231 of 10,990 pages
English, kept reading, run 1 of 2 (run 639)
- scribd.com: 378 of 2,734 pages
- library.douglascollege.ca: 144 of 2,734 pages
- studocu.com: 108 of 2,734 pages
- agrigento2025.org: 56 of 2,734 pages
- unisq.edu.au: 52 of 2,734 pages
- en.wikipedia.org: 45 of 2,734 pages
- testbook.com: 39 of 2,734 pages
- pmc.ncbi.nlm.nih.gov: 38 of 2,734 pages
- hpbose.org: 36 of 2,734 pages
- cbseacademic.nic.in: 35 of 2,734 pages
English, kept reading, run 2 of 2 (run 693)
- scribd.com: 486 of 2,734 pages
- studocu.com: 123 of 2,734 pages
- en.wikipedia.org: 49 of 2,734 pages
- pmc.ncbi.nlm.nih.gov: 45 of 2,734 pages
- bbc.com: 38 of 2,734 pages
- britannica.com: 36 of 2,734 pages
- englishfolio.com: 36 of 2,734 pages
- cbseacademic.nic.in: 35 of 2,734 pages
- useofenglishpro.com: 33 of 2,734 pages
- hpbose.org: 32 of 2,734 pages
PWS-Decisions, our first grounded run (5 November 2026)
500 pages for 100 questions (5 per question), from 240 sites. The most returned:
- github.com: 34 of 500 pages
- platform.claude.com: 15 of 500 pages
- dev.epicgames.com: 10 of 500 pages
- redis.io: 10 of 500 pages
- docs.godotengine.org: 9 of 500 pages
- code.claude.com: 8 of 500 pages
- docs.python.org: 8 of 500 pages
- docs.aws.amazon.com: 7 of 500 pages
- bun.com: 6 of 500 pages
- help.openai.com: 6 of 500 pages
2 of the 100 questions had a literal match of 8+ words (its state with its instructions, or a choice's description). This run came before the three runs whose mean the charts show; their responses were not kept for this check.
Sites: the page addresses' host names without “www.”. Words: runs of letters and digits, case-folded. Only these counts are published: the pages' addresses, titles and snippets stay with us.
Read it with these caveats
- Grounded searches send the questions to Perplexity's search. Each question with web went out as a web search.
- The pilot came first. Our protocol's first step was 200 questions (68 reading, 132 grammar; runs 631 and 632). Its rule: a full grounded run only for a gain of more than 5 points. The pilot gained +0.5 points, short of the rule; the full-bank grounded run was made afterwards so that the setup could be ranked.
- Costs with web are an upper bound. ≤ Upper-bound estimate: one $0.001 web search counted per request; Perplexity bills only completed searches. Price: $0.02 per 1M input tokens plus $0.001 per completed search (pricing, read 5 November 2026). Search evidence counts as input tokens.
- Runs per mode: 3 runs on the full bank and 2 runs on the English questions. The cards, charts and first table show the mean of each mode's runs, with the lowest and highest run where there are several; Each run gives every run and its paired test. The ranking keeps each setup's first run. The pilot and the Turkish and English “I don't know” pairs on our questions: one run each.
- Perplexity's own figures. Perplexity reported 86 of 100 PWS-Decisions questions right with grounding (its run of 4 November 2026); the 51% without is the benchmark authors' (Parallel's) leaderboard figure, which Perplexity's own run matched. Its grounded requests took 370 ms at the median and 685 ms at the 95th percentile, from Perplexity's client (Web grounding for the Decisions API, 4 November 2026). Ours are from our machine, the mean of 3 runs per mode on 5 November 2026 to 6 November 2026: 50.7% off, 87.0% with web; 483 and 728 ms at the median and 95th percentile with web.
- PWS-Decisions is gated. PWS-Decisions (version 0.1.0-rc1) is Parallel's dataset, for evaluation only; we publish aggregates only.
Detailed results The tables, with the pilot, each run with its paired counts, the unmatched answers and our method
Every number
| Setup | Rightof the questions | Accuracyhigher is better | Median per requestms; lower is better | 95th percentilems; lower is better | Costper 1,000 questionsper 1M questions; lower is better | Right only with this setupexact McNemar p |
|---|---|---|---|---|---|---|
| Perplexity Decider 27B v1.2 API Open · 27BTurkish, full bank, 2,198 · mean of 3 runs (633, 684, 687) | 1,459.7 of 2,198runs 1,458 to 1,462 | 66.4%runs 66.3% to 66.5% | 249runs 239 to 255 | 730runs 684 to 810 | $0.008$7.70$0.008$7.70 each runno search fee | per run, in Each run |
| Perplexity Decider 27B v1.2 API · with web Open · 27BTurkish, full bank, 2,198 · mean of 3 runs (640, 685, 692) | 1,500 of 2,198runs 1,478 to 1,511 | 68.2%runs 67.2% to 68.7% | 480runs 455 to 502 | 872runs 638 to 1,026 | ≤ $1.034≤ $1,034runs ≤ $1.034≤ $1,034 to ≤ $1.035≤ $1,035incl. search | per run, in Each run |
| Perplexity Decider 27B v1.2 API Open · 27BTurkish reading, 656 · mean of 3 runs (633, 684, 687) | 584 of 656584 each run | 89.0%89.0% each run | 271runs 250 to 308 | 778runs 684 to 917 | $0.009$9.11$0.009$9.11 each runno search fee | per run, in Each run |
| Perplexity Decider 27B v1.2 API · with web Open · 27BTurkish reading, 656 · mean of 3 runs (640, 685, 692) | 581.7 of 656runs 577 to 585 | 88.7%runs 88.0% to 89.2% | 504runs 474 to 530 | 947runs 694 to 1,165 | ≤ $1.040≤ $1,040≤ $1.040≤ $1,040 each runincl. search | per run, in Each run |
| Perplexity Decider 27B v1.2 API Open · 27BTurkish grammar, 1,542 · mean of 3 runs (633, 684, 687) | 875.7 of 1,542runs 874 to 878 | 56.8%runs 56.7% to 56.9% | 248runs 234 to 257 | 635runs 362 to 830 | $0.007$7.11$0.007$7.11 each runno search fee | per run, in Each run |
| Perplexity Decider 27B v1.2 API · with web Open · 27BTurkish grammar, 1,542 · mean of 3 runs (640, 685, 692) | 918.3 of 1,542runs 901 to 928 | 59.6%runs 58.4% to 60.2% | 472runs 447 to 500 | 816runs 616 to 1,060 | ≤ $1.032≤ $1,032runs ≤ $1.032≤ $1,032 to ≤ $1.033≤ $1,033incl. search | per run, in Each run |
| Perplexity Decider 27B v1.2 API Open · 27BTurkish pilot, 200 · run 631 | 142 of 200 | 71.0% | 247 | 476 | $0.008$7.71no search fee | 8 (p = 1.000) |
| Perplexity Decider 27B v1.2 API · with web Open · 27BTurkish pilot, 200 · run 632 | 143 of 200 | 71.5% | 476 | 907 | ≤ $1.035≤ $1,035incl. search | 9 (p = 1.000) |
| Perplexity Decider 27B v1.2 API Open · 27BPilot reading, 68 · run 631 | 62 of 68 | 91.2% | 250 | 456 | $0.009$9.37no search fee | 0 (p = 1.000) |
| Perplexity Decider 27B v1.2 API · with web Open · 27BPilot reading, 68 · run 632 | 62 of 68 | 91.2% | 484 | 952 | ≤ $1.041≤ $1,041incl. search | 0 (p = 1.000) |
| Perplexity Decider 27B v1.2 API Open · 27BPilot grammar, 132 · run 631 | 80 of 132 | 60.6% | 246 | 500 | $0.007$6.86no search fee | 8 (p = 1.000) |
| Perplexity Decider 27B v1.2 API · with web Open · 27BPilot grammar, 132 · run 632 | 81 of 132 | 61.4% | 466 | 907 | ≤ $1.032≤ $1,032incl. search | 9 (p = 1.000) |
| Perplexity Decider 27B v1.2 API Open · 27BEnglish, 547 kept reading · mean of 2 runs (637, 688) | 484 of 547484 each run | 88.5%88.5% each run | 246runs 235 to 256 | 546runs 394 to 698 | $0.008$7.79$0.008$7.79 each runno search fee | per run, in Each run |
| Perplexity Decider 27B v1.2 API · with web Open · 27BEnglish, 547 kept reading · mean of 2 runs (639, 693) | 490.5 of 547runs 486 to 495 | 89.7%runs 88.8% to 90.5% | 471runs 457 to 484 | 712runs 587 to 836 | ≤ $1.036≤ $1,036≤ $1.036≤ $1,036 each runincl. search | per run, in Each run |
| Perplexity Decider 27B v1.2 API Open · 27BPWS-Decisions, 100 · mean of 3 runs | 50.7 of 100runs 50 to 51 | 50.7%runs 50.0% to 51.0% | 244runs 240 to 249 | 656runs 396 to 877 | $0.004$3.97$0.004$3.97 each runno search fee | no per-question results |
| Perplexity Decider 27B v1.2 API · with web Open · 27BPWS-Decisions, 100 · mean of 3 runs | 87 of 10087 each run | 87.0%87.0% each run | 483runs 475 to 491 | 728runs 619 to 876 | ≤ $1.041≤ $1,041≤ $1.041≤ $1,041 each runincl. search | no per-question results |
Two rows per set of questions: off, then with web; under each name, its questions and runs. The full bank, the English questions and PWS-Decisions: the mean of the runs, with the lowest and highest run under each value; each run, with its paired counts, is in Each run below. Times: per request from our machine; the 95th percentile by nearest rank. Cost: each run's input tokens at the price below, plus the fee per search for every request with web (≤ Upper-bound estimate: one $0.001 web search counted per request; Perplexity bills only completed searches.) Right only with this setup: the questions it got right and the other setup did not, with the exact McNemar p of the pair.
With “I don't know” (Turkish or English) as an extra option
| Setup | Right · wrong · the optioncounts | Answered(right + wrong) / questions | Wrong of answerswrong / (right + wrong) | Non-hallucinationthe option / (the option + wrong); higher is better |
|---|---|---|---|---|
| Perplexity Decider 27B v1.2 API Open · 27BTurkish, full bank, 2,198 · run 641 | 1,469 · 726 · 3 | 99.9% | 33.1% | 0.4% |
| Perplexity Decider 27B v1.2 API · with web Open · 27BTurkish, full bank, 2,198 · run 642 | 1,473 · 702 · 23 | 99.0% | 32.3% | 3.2% |
| Perplexity Decider 27B v1.2 API Open · 27BTurkish pilot, 200 · run 634 | 145 · 55 · 0 | 100.0% | 27.5% | 0.0% |
| Perplexity Decider 27B v1.2 API · with web Open · 27BTurkish pilot, 200 · run 635 | 147 · 51 · 2 | 99.0% | 25.8% | 3.8% |
| Perplexity Decider 27B v1.2 API Open · 27BEnglish, 547 kept questions, I don't know · run 643 | 481 · 63 · 3 | 99.5% | 11.6% | 4.5% |
| Perplexity Decider 27B v1.2 API · with web Open · 27BEnglish, 547 kept questions, I don't know · run 644 | 481 · 56 · 10 | 98.2% | 10.4% | 15.2% |
| Perplexity Decider 27B v1.2 API Open · 27BPWS-Decisions, 100, I don't know · mean of 3 runs | 47.7 · 36 · 16.3 | 83.7%runs 83.0% to 84.0% | 43.0%runs 42.2% to 44.0% | 31.2%runs 30.2% to 32.7% |
| Perplexity Decider 27B v1.2 API · with web Open · 27BPWS-Decisions, 100, I don't know · mean of 3 runs | 84.3 · 8.3 · 7.3 | 92.7%runs 92.0% to 93.0% | 9.0%runs 8.6% to 9.7% | 46.8%runs 43.8% to 50.0% |
Each run
Runs per mode: 3 runs on the full bank and 2 runs on the English questions: each mode's first run (the ranking's run), then its repeats with the same questions in the same order, answer key, runner and settings. Each run of one repeat number is paired with the other mode's run of the same number, on the first runs' questions.
| Setup | Rightof the questions | Accuracyhigher is better | Median per requestms; lower is better | 95th percentilems; lower is better | Costper 1,000 questionsper 1M questions; lower is better | Right only with this setupexact McNemar p |
|---|---|---|---|---|---|---|
| Perplexity Decider 27B v1.2 API Open · 27BTurkish, full bank, 2,198, run 1 of 3 · run 633 | 1,458 of 2,198 | 66.3% | 254 | 697 | $0.008$7.70no search fee | 94 (p < 0.001) |
| Perplexity Decider 27B v1.2 API · with web Open · 27BTurkish, full bank, 2,198, run 1 of 3 · run 640 | 1,511 of 2,198 | 68.7% | 482 | 953 | ≤ $1.034≤ $1,034incl. search | 147 (p < 0.001) |
| Perplexity Decider 27B v1.2 API Open · 27BTurkish, full bank, 2,198, run 2 of 3 · run 684 | 1,459 of 2,198 | 66.4% | 255 | 810 | $0.008$7.70no search fee | 100 (p = 0.001) |
| Perplexity Decider 27B v1.2 API · with web Open · 27BTurkish, full bank, 2,198, run 2 of 3 · run 685 | 1,511 of 2,198 | 68.7% | 502 | 1,026 | ≤ $1.034≤ $1,034incl. search | 152 (p = 0.001) |
| Perplexity Decider 27B v1.2 API Open · 27BTurkish, full bank, 2,198, run 3 of 3 · run 687 | 1,462 of 2,198 | 66.5% | 239 | 684 | $0.008$7.70no search fee | 109 (p = 0.327) |
| Perplexity Decider 27B v1.2 API · with web Open · 27BTurkish, full bank, 2,198, run 3 of 3 · run 692 | 1,478 of 2,198 | 67.2% | 455 | 638 | ≤ $1.035≤ $1,035incl. search | 125 (p = 0.327) |
| Perplexity Decider 27B v1.2 API Open · 27BEnglish, kept reading, 547, run 1 of 2 · run 637 | 484 of 547 | 88.5% | 256 | 698 | $0.008$7.79no search fee | 2 (p = 0.007) |
| Perplexity Decider 27B v1.2 API · with web Open · 27BEnglish, kept reading, 547, run 1 of 2 · run 639 | 495 of 547 | 90.5% | 484 | 836 | ≤ $1.036≤ $1,036incl. search | 13 (p = 0.007) |
| Perplexity Decider 27B v1.2 API Open · 27BEnglish, kept reading, 547, run 2 of 2 · run 688 | 484 of 547 | 88.5% | 235 | 394 | $0.008$7.79no search fee | 6 (p = 0.791) |
| Perplexity Decider 27B v1.2 API · with web Open · 27BEnglish, kept reading, 547, run 2 of 2 · run 693 | 486 of 547 | 88.8% | 457 | 587 | ≤ $1.036≤ $1,036incl. search | 8 (p = 0.791) |
Paired counts, off against with web, per run
| Runs | Both rightquestions | Right only offquestions | Right only with webquestions | Both wrongquestions | Errorsoff · with web; scored wrong | Missingoff · with web; not answered | Exact McNemar pright only off against only with web |
|---|---|---|---|---|---|---|---|
| Turkish, full bank, run 1 of 3runs 633 and 640; 2,198 questions | 1,364 | 94 | 147 | 593 | 0 · 0 | 0 · 0 | p < 0.001 |
| Turkish, full bank, run 2 of 3runs 684 and 685; 2,198 questions | 1,359 | 100 | 152 | 587 | 0 · 0 | 0 · 0 | p = 0.001 |
| Turkish, full bank, run 3 of 3runs 687 and 692; 2,198 questions | 1,353 | 109 | 125 | 611 | 0 · 0 | 0 · 0 | p = 0.327 |
| English, kept reading, run 1 of 2runs 637 and 639; 547 questions | 482 | 2 | 13 | 50 | 0 · 0 | 0 · 0 | p = 0.007 |
| English, kept reading, run 2 of 2runs 688 and 693; 547 questions | 478 | 6 | 8 | 55 | 0 · 0 | 0 · 0 | p = 0.791 |
Our method
Model pplx-decider-v1.2 through Perplexity's Decisions API, one question per request, the Text tab's System One prompts. Each pair of runs differs only in the request's mode: off, or web_grounding, which runs a web search per question and adds its results as evidence. Runs from 5 November 2026 to 5 November 2026. Runs per mode: 3 runs on the full bank and 2 runs on the English questions: each mode's first run and its repeats with the same questions in the same order, answer key, runner and settings; the ranking keeps each setup's first run, and the repeats enter only this page.
- Turkish, full bank: 2,198 questions = 656 reading + 1,542 grammar (first runs 633 and 640, the Text tab rows Perplexity Decider 27B v1.2 · API and Perplexity Decider 27B v1.2 · API · with web). Repeats with the same questions in the same order: repeat 2: runs 684 and 685; repeat 3: runs 687 and 692. Reading: units 1, 2, 3, 4, 5 and 6.
- Turkish pilot: 200 = 68 reading + 132 grammar, the runner's sample with seed 0 (runs 631 and 632).
- “I don't know” as an extra option in Turkish, as on the Hallucination tab: the full bank (2,198 questions, runs 641 and 642) and the pilot (200, runs 634 and 635). “I don't know” as an extra option in English, as on English and Turkish: the 547 kept questions (runs 643 and 644).
- English: the 547 kept reading questions of the English pages in our translations (first runs 637 and 639). Repeats with the same questions in the same order: repeat 2: runs 688 and 693.
- PWS-Decisions: its 100 public English questions with the benchmark's own request, 3 runs per mode by us on 5 November 2026, as on PWS-Decisions; the source checks come from our first grounded run of 5 November 2026.
Accuracy: right answers over the questions asked. Time: each request's wall-clock time on our machine; the median and the 95th percentile by nearest rank. Cost per 1M questions: the run's input tokens at $0.02 per 1M plus $0.001 per request with web, over its questions. A mean of several runs: the mean of the runs' values, with the lowest and highest run. The paired test: the exact McNemar test on the questions one arm got right and the other did not, per pair of runs.
Download: results.json (every number on this page but PWS-Decisions' means, under web_grounding), answers.json (the pilots and the repeats under web_grounding_runs, the other runs in their own lists), web-grounding.json (tokens by group, the source checks' question ids and the first PWS-Decisions run's aggregates) and pws-decisions.json (PWS-Decisions' runs). reproduce.py recomputes results.json.