Experiment

Web grounding: Perplexity Decider 27B v1.2 off and with web

Turkish, 2,198 questions

66.4% → 68.2% (+1.8 points)

Mean of 3 runs; runs 66.3% to 66.5% off, runs 67.2% to 68.7% with web

$0.008$7.70 vs ≤ $1.034≤ $1,034 per 1,000 questionsper 1M questions

English reading, 547

88.5% → 89.7% (+1.2 points)

Mean of 2 runs; 88.5% each run off, runs 88.8% to 90.5% with web

$0.008$7.79 vs ≤ $1.036≤ $1,036 per 1,000 questionsper 1M questions

PWS-Decisions, 100

50.7% → 87.0% (+36.3 points)

Mean of 3 runs; runs 50.0% to 51.0% off, 87.0% each run with web

$0.004$3.97 vs ≤ $1.041≤ $1,041 per 1,000 questionsper 1M questions

Accuracy with web off → with web, the mean of the runs; cost web off vs with web. ≤ Upper-bound estimate: the search fee counted for every request with web.

Accuracy

Share of questions answered right, then the right answers; for a mean of several runs, the lowest to highest run under it; beside each set, the change with web in percentage points · higher is better

  1. Turkish, full bank, 2,198 · mean of 3 runs · +1.8 pt
  2. Perplexity Decider 27B v1.2 API66.4% · 1,459.766.3 to 66.5%
  3. Perplexity Decider 27B v1.2 API · with web68.2% · 1,50067.2 to 68.7%
  4. English, 547 kept reading · mean of 2 runs · +1.2 pt
  5. Perplexity Decider 27B v1.2 API88.5% · 48488.5% each run
  6. Perplexity Decider 27B v1.2 API · with web89.7% · 490.588.8 to 90.5%
  7. PWS-Decisions, 100 · mean of 3 runs · +36.3 pt
  8. Perplexity Decider 27B v1.2 API50.7% · 50.750.0 to 51.0%
  9. Perplexity Decider 27B v1.2 API · with web87.0% · 8787.0% each run

The Turkish full bank by part

  1. Turkish reading, 656 · mean of 3 runs · -0.4 pt
  2. Perplexity Decider 27B v1.2 API89.0% · 58489.0% each run
  3. Perplexity Decider 27B v1.2 API · with web88.7% · 581.788.0 to 89.2%
  4. Turkish grammar, 1,542 · mean of 3 runs · +2.8 pt
  5. Perplexity Decider 27B v1.2 API56.8% · 875.756.7 to 56.9%
  6. Perplexity Decider 27B v1.2 API · with web59.6% · 918.358.4 to 60.2%
The pilot, 200 questions, run first, one run each
  1. Turkish pilot, 200 · +0.5 pt
  2. Perplexity Decider 27B v1.2 API71.0% · 142
  3. Perplexity Decider 27B v1.2 API · with web71.5% · 143
  4. Pilot reading, 68 · +0.0 pt
  5. Perplexity Decider 27B v1.2 API91.2% · 62
  6. Perplexity Decider 27B v1.2 API · with web91.2% · 62
  7. Pilot grammar, 132 · +0.8 pt
  8. Perplexity Decider 27B v1.2 API60.6% · 80
  9. Perplexity Decider 27B v1.2 API · with web61.4% · 81

Web off: Perplexity Decider 27B v1.2 API With web: Perplexity Decider 27B v1.2 API · with web Each heading names the questions both setups answered.

Non-hallucination

“I don't know”, in Turkish and in English, offered as an extra option: the bar is the option's share of the answers that were not right; beside it the times it was chosen, the answered share (right + wrong of the questions) and the wrong share of the answers · higher is better, read with the answered share. PWS-Decisions: “I don't know” as an extra choice, the mean of 3 runs per mode

  1. Turkish, full bank, 2,198: right 1,469 → 1,473, wrong 726 → 702
  2. Perplexity Decider 27B v1.2 API0.4% | 3 · 99.9% · 33.1%
  3. Perplexity Decider 27B v1.2 API · with web3.2% | 23 · 99.0% · 32.3%
  4. Turkish pilot, 200: right 145 → 147, wrong 55 → 51
  5. Perplexity Decider 27B v1.2 API0.0% | 0 · 100.0% · 27.5%
  6. Perplexity Decider 27B v1.2 API · with web3.8% | 2 · 99.0% · 25.8%
  7. English, 547 kept questions, I don't know: 481 right both ways, wrong 63 → 56
  8. Perplexity Decider 27B v1.2 API4.5% | 3 · 99.5% · 11.6%
  9. Perplexity Decider 27B v1.2 API · with web15.2% | 10 · 98.2% · 10.4%
  10. PWS-Decisions, 100, I don't know · mean of 3 runs: right 47.7 → 84.3, wrong 36 → 8.3
  11. Perplexity Decider 27B v1.2 API31.2% | 16.3 · 83.7% · 43.0%
  12. Perplexity Decider 27B v1.2 API · with web46.8% | 7.3 · 92.7% · 9.0%

Web off: Perplexity Decider 27B v1.2 API With web: Perplexity Decider 27B v1.2 API · with web Each heading names the questions both setups answered. A setup that always chose the option would reach 100% with no right answers.

Latency

Median time per request (the bar) and 95th percentile by nearest rank (the line), one question per request, from our machine; for a mean of several runs, the mean of the runs' values · lower is better

  1. Turkish, full bank, 2,198 · mean of 3 runs
  2. Perplexity Decider 27B v1.2 API249 · 730 ms
  3. Perplexity Decider 27B v1.2 API · with web480 · 872 ms
  4. English, 547 kept reading · mean of 2 runs
  5. Perplexity Decider 27B v1.2 API246 · 546 ms
  6. Perplexity Decider 27B v1.2 API · with web471 · 712 ms
  7. PWS-Decisions, 100 · mean of 3 runs
  8. Perplexity Decider 27B v1.2 API244 · 656 ms
  9. Perplexity Decider 27B v1.2 API · with web483 · 728 ms

Web off: Perplexity Decider 27B v1.2 API With web: Perplexity Decider 27B v1.2 API · with web Each heading names the questions both setups answered. 95th percentile

Cost

Estimated dollars per 1,000 questionsper 1M questions, log scale: input tokens at $0.02 per 1M, plus $0.001 per request with web (incl. search); for a mean of several runs, the mean of the runs' costs · lower is better

  1. Turkish, full bank, 2,198 · mean of 3 runs
  2. Perplexity Decider 27B v1.2 API$0.008$7.70
  3. Perplexity Decider 27B v1.2 API · with web≤ $1.034≤ $1,034 incl. search
  4. English, 547 kept reading · mean of 2 runs
  5. Perplexity Decider 27B v1.2 API$0.008$7.79
  6. Perplexity Decider 27B v1.2 API · with web≤ $1.036≤ $1,036 incl. search
  7. PWS-Decisions, 100 · mean of 3 runs
  8. Perplexity Decider 27B v1.2 API$0.004$3.97
  9. Perplexity Decider 27B v1.2 API · with web≤ $1.041≤ $1,041 incl. search

Web off: Perplexity Decider 27B v1.2 API With web: Perplexity Decider 27B v1.2 API · with web Each heading names the questions both setups answered. ≤ Upper-bound estimate: one $0.001 web search counted per request; Perplexity bills only completed searches.

Perplexity's Decisions API without and with a web search per question.

More about this page

PWS-Decisions: the decision models, with and without web

What this measures. Parallel Web Systems made PWS-Decisions. This is an end-to-end comparison of API setups with and without web search on its 100 public questions, which Parallel chose “for their difficulty and alignment with customer outcomes” (its methodology), around its customers' needs for current facts from the web. It is not a general ranking of models: without search, most setups score near random guessing, because the answers depend on dated web facts they were not given.

Each bar is the mean of our 3 runs per setup, decision models only; the PWS-Decisions rows of the charts above are the same means for Perplexity's two setups. Chat models, with and without their own web search, and every number: PWS-Decisions.

Dataset
PWS-Decisions 0.1.0-rc1, Hugging Face parallel-web/pws-decisions-benchmark at 64da0d8a4ec0
Scorer
Parallel's own (python -m pws_decisions score), unchanged, 0f403cf99856
Runs
3 per setup, 198 in all; 0 errors, 0 missing answers
Run dates
5 November 2026 to 6 November 2026
Published
Totals only: no question, answer or per-question result

Accuracy

Share of the 100 public questions answered right, the mean of our 3 runs per setup · higher is better; beside it the cost (per 1,000 questionsper 1M questions) and the median and 95th percentile request time

  1. With web search
  2. Perplexity Decider 27B v1.2 API · with web87.0% | ≤ $1.041≤ $1,041 | 483 · 728
  3. Without search
  4. Metask-Jev-4B60.0% | no price | 74 · 79
  5. Perplexity Decider Flash v1.259.0% | $0.002$2.01 | 27 · 30
  6. d358.0% | no price | 63 · 77
  7. d1-ultra57.7% | $0.006$6.17 | 220 · 560
  8. Laya57.0% | no price | 11 · 12
  9. Perplexity Decider 27B v1.2 DGX Spark57.0% | $0.004$4.03 | 74 · 98
  10. d3-ultra55.0% | no price | 125 · 171
  11. REAX 1.355.0% | no price | 215 · 355
  12. Strands Decider 2B55.0% | no price | 24 · 28
  13. d1-2B54.0% | no price | 11 · 12
  14. Laya Multilingual54.0% | no price | 9 · 11
  15. Mercury Decide53.3% | $0.004$3.57 | 291 · 365
  16. d1.153.0% | $0.003$2.94 | 180 · 415
  17. GLiDE v1.253.0% | $0.007$6.90 | 64 · 71
  18. SubQ Snap 1.153.0% | $0.008$7.94 | 378 · 390
  19. SubQ Snap Preview53.0% | $0.010$9.92 | 380 · 392
  20. Clef52.0% | $0.058$58.40 | 126 · 154
  21. Perplexity Decider 27B v1.2 API50.7% | $0.004$3.97 | 244 · 656
  22. Decider 150.0% | $0.007$6.81 | 629 · 690
  23. Microsoft-Decision-149.0% | $0.006$6.11 | 317 · 385
  24. Open-Jev-27B-v1.149.0% | no price | 146 · 272
  25. Clef-Flash48.0% | $0.009$9.25 | 85 · 109
  26. d147.7% | $0.003$2.94 | 185 · 425
  27. Drex Max v1.147.7% | $0.012$11.90 | 331 · 470
  28. OneJev-27B45.0% | no price | 94 · 132
  29. Drex Max42.0% | $0.012$12.08 | 64 · 78
  30. GPT-6 Luna Decisions API42.0% | $0.022$21.93 | 285 · 522
  31. GLiNER2.5-multi-Decide41.0% | no price | 8 · 9
  32. Mammoth-1 RTX PRO 6000 Blackwell39.0% | $0.020$20.43 | 128 · 135
  33. Winnow-12B39.0% | no price | 48 · 59
  34. Mammoth-1 Baseten37.0% | $0.020$20.42 | 184 · 255
  35. Jev 1.1335.0% | $0.018$18.01 | 170 · 242
  36. Jev 1.1434.3% | $0.018$18.01 | 165 · 203
  37. celeris-1-decision33.3% | $0.008$8.46 | 384 · 389
  38. Elephant-133.0% | $0.007$7.15 | 55 · 63
  39. Cygnet32.0% | $0.023$22.82 | 27 · 29
  40. Drex 2.031.0% | $0.008$8.11 | 293 · 1,507

Perplexity Metask vLLM-SR Liquid Laya REAX Strands Inception Fastino SubQ Clef meraGPT Microsoft Open-Jev Drex OneJev GPT Mammoth Winnow Jev Celeris Elephant Cygnet With web: the setup searched the web for each question (striped) The chance line: Parallel's random-guess baseline, 44.67% (the benchmark's README), uniform guessing over each question's choices, a property of the questions, not a run of ours.

Cost

Estimated dollars per 1,000 questionsper 1M questions, the mean of 3 runs, log scale; with web search, then without, each cheapest first · lower is better

  1. With web search
  2. Perplexity Decider 27B v1.2 API · with web≤ $1.041≤ $1,041 incl. search
  3. Without search
  4. Perplexity Decider Flash v1.2$0.002$2.01
  5. d1$0.003$2.94
  6. d1.1$0.003$2.94
  7. Mercury Decide$0.004$3.57
  8. Perplexity Decider 27B v1.2 API$0.004$3.97
  9. Perplexity Decider 27B v1.2 DGX Spark$0.004$4.03
  10. Microsoft-Decision-1$0.006$6.11
  11. d1-ultra$0.006$6.17
  12. Decider 1$0.007$6.81
  13. GLiDE v1.2$0.007$6.90
  14. Elephant-1$0.007$7.15
  15. SubQ Snap 1.1$0.008$7.94
  16. Drex 2.0$0.008$8.11
  17. celeris-1-decision$0.008$8.46
  18. Clef-Flash$0.009$9.25
  19. SubQ Snap Preview$0.010$9.92
  20. Drex Max v1.1$0.012$11.90
  21. Drex Max$0.012$12.08
  22. Jev 1.13$0.018$18.01
  23. Jev 1.14$0.018$18.01
  24. Mammoth-1 Baseten$0.020$20.42
  25. Mammoth-1 RTX PRO 6000 Blackwell$0.020$20.43
  26. GPT-6 Luna Decisions API$0.022$21.93
  27. Cygnet$0.023$22.82
  28. Clef$0.058$58.40

Perplexity Liquid Inception Microsoft meraGPT Fastino Elephant SubQ Drex Celeris Clef Jev Mammoth GPT Cygnet With web (striped) Every cost is an estimate: the list price × the tokens and searches our runs recorded, not a provider's bill; for a subscription run, what the same usage would cost on the API (API-equivalent estimate). Without web: each run's tokens at the setup's list price, no search fee. With web: the same, plus the provider's web search fee (“incl. search”). ≤ Upper-bound estimate: one $0.001 web search counted per request; Perplexity bills only completed searches. Metask-Jev-4B: no price: on our GPU, priced only where an API list price exists. d3: no price: on our GPU, priced only where an API list price exists. Laya: no price: on our GPU, priced only where an API list price exists. d3-ultra: no price: on our GPU, priced only where an API list price exists. REAX 1.3: no price: on our GPU, priced only where an API list price exists. Strands Decider 2B: no price: on our GPU, priced only where an API list price exists. d1-2B: no price: on our GPU, priced only where an API list price exists. Laya Multilingual: no price: on our GPU, priced only where an API list price exists. Open-Jev-27B-v1.1: no price: on our GPU, priced only where an API list price exists. OneJev-27B: no price: on our GPU, priced only where an API list price exists. GLiNER2.5-multi-Decide: no price: on our GPU, priced only where an API list price exists. Winnow-12B: no price: on our GPU, priced only where an API list price exists.

Latency

Median time per request (the bar) and 95th percentile by nearest rank (the line), one question per request, from our machine, the mean of 3 runs, log scale; with web search, then without, each fastest first · lower is better

  1. With web search
  2. Perplexity Decider 27B v1.2 API · with web483 · 728 ms
  3. Without search
  4. GLiNER2.5-multi-Decide8 · 9 ms
  5. Laya Multilingual9 · 11 ms
  6. Laya11 · 12 ms
  7. d1-2B11 · 12 ms
  8. Strands Decider 2B24 · 28 ms
  9. Cygnet27 · 29 ms
  10. Perplexity Decider Flash v1.227 · 30 ms
  11. Winnow-12B48 · 59 ms
  12. Elephant-155 · 63 ms
  13. d363 · 77 ms
  14. GLiDE v1.264 · 71 ms
  15. Drex Max64 · 78 ms
  16. Metask-Jev-4B74 · 79 ms
  17. Perplexity Decider 27B v1.2 DGX Spark74 · 98 ms
  18. Clef-Flash85 · 109 ms
  19. OneJev-27B94 · 132 ms
  20. d3-ultra125 · 171 ms
  21. Clef126 · 154 ms
  22. Mammoth-1 RTX PRO 6000 Blackwell128 · 135 ms
  23. Open-Jev-27B-v1.1146 · 272 ms
  24. Jev 1.14165 · 203 ms
  25. Jev 1.13170 · 242 ms
  26. d1.1180 · 415 ms
  27. Mammoth-1 Baseten184 · 255 ms
  28. d1185 · 425 ms
  29. REAX 1.3215 · 355 ms
  30. d1-ultra220 · 560 ms
  31. Perplexity Decider 27B v1.2 API244 · 656 ms
  32. GPT-6 Luna Decisions API285 · 522 ms
  33. Mercury Decide291 · 365 ms
  34. Drex 2.0293 · 1,507 ms
  35. Microsoft-Decision-1317 · 385 ms
  36. Drex Max v1.1331 · 470 ms
  37. SubQ Snap 1.1378 · 390 ms
  38. SubQ Snap Preview380 · 392 ms
  39. celeris-1-decision384 · 389 ms
  40. Decider 1629 · 690 ms

Fastino Laya Liquid Strands Cygnet Perplexity Winnow Elephant vLLM-SR Drex Metask Clef OneJev Mammoth Open-Jev Jev REAX GPT Inception Microsoft SubQ Celeris meraGPT With web (striped) 95th percentile Request times include the network and any retries.

What the search returned

In short. On the full bank, 388 to 414 questions per grounded run had 8+ consecutive question words in a returned title or snippet, and 123 to 159 got the source book itself. A word match alone does not prove that the answer was found.

Unit: returned pages, up to 5 per question; each site's count is out of the run's total pages.

The 8+ word check. A literal-match check: runs of 8 or more consecutive words of the question (its body and each option's text, without the option letters) in a returned page's title or snippet. It shows only such matches. Our translations are unpublished, but that does not prove the answer, or equivalent information, cannot be found on the web.

Turkish, full bank, first run (run 640)

10,989 pages for 2,198 questions (4 to 5 per question), from 966 sites. The most returned:

  1. scribd.com: 1,785 of 10,989 pages
  2. dergipark.org.tr: 1,070 of 10,989 pages
  3. dokuman.osym.gov.tr: 826 of 10,989 pages
  4. huggingface.co: 617 of 10,989 pages
  5. yksedebiyat.org: 372 of 10,989 pages
  6. youtube.com: 369 of 10,989 pages
  7. tr.wikipedia.org: 302 of 10,989 pages
  8. tdk.gov.tr: 297 of 10,989 pages
  9. turkedebiyati.org: 263 of 10,989 pages
  10. depo.pegem.net: 247 of 10,989 pages

414 questions had a literal match of 8+ words: 280 right off, 299 with web; 41 of the 147 gains and 22 of the 94 losses. 187 got a page from Paraf, the publisher of the bank's source book (138 right both ways). 125 got the source book itself, found by its title (99 right both ways).

Turkish pilot (run 632)

1,000 pages for 200 questions (5 per question), from 198 sites. The most returned:

  1. scribd.com: 162 of 1,000 pages
  2. dergipark.org.tr: 98 of 1,000 pages
  3. dokuman.osym.gov.tr: 71 of 1,000 pages
  4. huggingface.co: 61 of 1,000 pages
  5. youtube.com: 39 of 1,000 pages
  6. yksedebiyat.org: 36 of 1,000 pages
  7. depo.pegem.net: 28 of 1,000 pages
  8. ogmmateryal.eba.gov.tr: 26 of 1,000 pages
  9. turkedebiyati.org: 26 of 1,000 pages
  10. tr.wikipedia.org: 24 of 1,000 pages

36 questions had a literal match of 8+ words: 25 right off, 27 with web; 3 of the 9 gains and 1 of the 8 losses. 19 got a page from Paraf, the publisher of the bank's source book (15 right both ways). 15 got the source book itself, found by its title (12 right both ways).

English, first run (run 639)

2,734 pages for 547 questions (4 to 5 per question), from 1,025 sites. The most returned:

  1. scribd.com: 378 of 2,734 pages
  2. library.douglascollege.ca: 144 of 2,734 pages
  3. studocu.com: 108 of 2,734 pages
  4. agrigento2025.org: 56 of 2,734 pages
  5. unisq.edu.au: 52 of 2,734 pages
  6. en.wikipedia.org: 45 of 2,734 pages
  7. testbook.com: 39 of 2,734 pages
  8. pmc.ncbi.nlm.nih.gov: 38 of 2,734 pages
  9. hpbose.org: 36 of 2,734 pages
  10. cbseacademic.nic.in: 35 of 2,734 pages

8 of the 547 English questions had a literal match of 8+ words: 8 right off, 8 with web. None of the 13 English gains had the defined 8+ word overlap; this does not show that the answer, or its semantic equivalent, was absent from the returned pages.

Each grounded run

Run with webPagesreturned, of up to 5 per questionSiteshost names8+ word matchquestionsA Paraf pagequestionsThe source bookquestions
Turkish, full bank, run 1 of 3run 640, 2,198 questions10,989966414187125
Turkish, full bank, run 2 of 3run 685, 2,198 questions10,990953406179123
Turkish, full bank, run 3 of 3run 692, 2,198 questions10,990918388194159
English, kept reading, run 1 of 2run 639, 547 questions2,7341,025800
English, kept reading, run 2 of 2run 693, 547 questions2,7341,029900

N/A: the run returned no page, so it has no source checks. A Paraf page: questions with a page from the publisher of the bank's source book. The source book: questions with a page of the book itself, found by its title.

The 10 most returned sites, each run

Turkish, full bank, run 1 of 3 (run 640)

  1. scribd.com: 1,785 of 10,989 pages
  2. dergipark.org.tr: 1,070 of 10,989 pages
  3. dokuman.osym.gov.tr: 826 of 10,989 pages
  4. huggingface.co: 617 of 10,989 pages
  5. yksedebiyat.org: 372 of 10,989 pages
  6. youtube.com: 369 of 10,989 pages
  7. tr.wikipedia.org: 302 of 10,989 pages
  8. tdk.gov.tr: 297 of 10,989 pages
  9. turkedebiyati.org: 263 of 10,989 pages
  10. depo.pegem.net: 247 of 10,989 pages

Turkish, full bank, run 2 of 3 (run 685)

  1. scribd.com: 1,968 of 10,990 pages
  2. dergipark.org.tr: 1,066 of 10,990 pages
  3. dokuman.osym.gov.tr: 708 of 10,990 pages
  4. huggingface.co: 615 of 10,990 pages
  5. yksedebiyat.org: 426 of 10,990 pages
  6. youtube.com: 328 of 10,990 pages
  7. tdk.gov.tr: 305 of 10,990 pages
  8. tr.wikipedia.org: 290 of 10,990 pages
  9. kpsssorucoz.com: 279 of 10,990 pages
  10. turkedebiyati.org: 279 of 10,990 pages

Turkish, full bank, run 3 of 3 (run 692)

  1. scribd.com: 2,672 of 10,990 pages
  2. dergipark.org.tr: 1,283 of 10,990 pages
  3. huggingface.co: 691 of 10,990 pages
  4. kpsssorucoz.com: 490 of 10,990 pages
  5. tdk.gov.tr: 359 of 10,990 pages
  6. yksedebiyat.org: 332 of 10,990 pages
  7. youtube.com: 298 of 10,990 pages
  8. ogmmateryal.eba.gov.tr: 257 of 10,990 pages
  9. dokuman.osym.gov.tr: 252 of 10,990 pages
  10. tr.wikipedia.org: 231 of 10,990 pages

English, kept reading, run 1 of 2 (run 639)

  1. scribd.com: 378 of 2,734 pages
  2. library.douglascollege.ca: 144 of 2,734 pages
  3. studocu.com: 108 of 2,734 pages
  4. agrigento2025.org: 56 of 2,734 pages
  5. unisq.edu.au: 52 of 2,734 pages
  6. en.wikipedia.org: 45 of 2,734 pages
  7. testbook.com: 39 of 2,734 pages
  8. pmc.ncbi.nlm.nih.gov: 38 of 2,734 pages
  9. hpbose.org: 36 of 2,734 pages
  10. cbseacademic.nic.in: 35 of 2,734 pages

English, kept reading, run 2 of 2 (run 693)

  1. scribd.com: 486 of 2,734 pages
  2. studocu.com: 123 of 2,734 pages
  3. en.wikipedia.org: 49 of 2,734 pages
  4. pmc.ncbi.nlm.nih.gov: 45 of 2,734 pages
  5. bbc.com: 38 of 2,734 pages
  6. britannica.com: 36 of 2,734 pages
  7. englishfolio.com: 36 of 2,734 pages
  8. cbseacademic.nic.in: 35 of 2,734 pages
  9. useofenglishpro.com: 33 of 2,734 pages
  10. hpbose.org: 32 of 2,734 pages

PWS-Decisions, our first grounded run (5 November 2026)

500 pages for 100 questions (5 per question), from 240 sites. The most returned:

  1. github.com: 34 of 500 pages
  2. platform.claude.com: 15 of 500 pages
  3. dev.epicgames.com: 10 of 500 pages
  4. redis.io: 10 of 500 pages
  5. docs.godotengine.org: 9 of 500 pages
  6. code.claude.com: 8 of 500 pages
  7. docs.python.org: 8 of 500 pages
  8. docs.aws.amazon.com: 7 of 500 pages
  9. bun.com: 6 of 500 pages
  10. help.openai.com: 6 of 500 pages

2 of the 100 questions had a literal match of 8+ words (its state with its instructions, or a choice's description). This run came before the three runs whose mean the charts show; their responses were not kept for this check.

Sites: the page addresses' host names without “www.”. Words: runs of letters and digits, case-folded. Only these counts are published: the pages' addresses, titles and snippets stay with us.

Read it with these caveats

  • Grounded searches send the questions to Perplexity's search. Each question with web went out as a web search.
  • The pilot came first. Our protocol's first step was 200 questions (68 reading, 132 grammar; runs 631 and 632). Its rule: a full grounded run only for a gain of more than 5 points. The pilot gained +0.5 points, short of the rule; the full-bank grounded run was made afterwards so that the setup could be ranked.
  • Costs with web are an upper bound. ≤ Upper-bound estimate: one $0.001 web search counted per request; Perplexity bills only completed searches. Price: $0.02 per 1M input tokens plus $0.001 per completed search (pricing, read 5 November 2026). Search evidence counts as input tokens.
  • Runs per mode: 3 runs on the full bank and 2 runs on the English questions. The cards, charts and first table show the mean of each mode's runs, with the lowest and highest run where there are several; Each run gives every run and its paired test. The ranking keeps each setup's first run. The pilot and the Turkish and English “I don't know” pairs on our questions: one run each.
  • Perplexity's own figures. Perplexity reported 86 of 100 PWS-Decisions questions right with grounding (its run of 4 November 2026); the 51% without is the benchmark authors' (Parallel's) leaderboard figure, which Perplexity's own run matched. Its grounded requests took 370 ms at the median and 685 ms at the 95th percentile, from Perplexity's client (Web grounding for the Decisions API, 4 November 2026). Ours are from our machine, the mean of 3 runs per mode on 5 November 2026 to 6 November 2026: 50.7% off, 87.0% with web; 483 and 728 ms at the median and 95th percentile with web.
  • PWS-Decisions is gated. PWS-Decisions (version 0.1.0-rc1) is Parallel's dataset, for evaluation only; we publish aggregates only.
Detailed results The tables, with the pilot, each run with its paired counts, the unmatched answers and our method

Every number

SetupRightof the questionsAccuracyhigher is betterMedian per requestms; lower is better95th percentilems; lower is betterCostper 1,000 questionsper 1M questions; lower is betterRight only with this setupexact McNemar p
Perplexity Decider 27B v1.2 API Open · 27BTurkish, full bank, 2,198 · mean of 3 runs (633, 684, 687)1,459.7 of 2,198runs 1,458 to 1,46266.4%runs 66.3% to 66.5%249runs 239 to 255730runs 684 to 810$0.008$7.70$0.008$7.70 each runno search feeper run, in Each run
Perplexity Decider 27B v1.2 API · with web Open · 27BTurkish, full bank, 2,198 · mean of 3 runs (640, 685, 692)1,500 of 2,198runs 1,478 to 1,51168.2%runs 67.2% to 68.7%480runs 455 to 502872runs 638 to 1,026≤ $1.034≤ $1,034runs ≤ $1.034≤ $1,034 to ≤ $1.035≤ $1,035incl. searchper run, in Each run
Perplexity Decider 27B v1.2 API Open · 27BTurkish reading, 656 · mean of 3 runs (633, 684, 687)584 of 656584 each run89.0%89.0% each run271runs 250 to 308778runs 684 to 917$0.009$9.11$0.009$9.11 each runno search feeper run, in Each run
Perplexity Decider 27B v1.2 API · with web Open · 27BTurkish reading, 656 · mean of 3 runs (640, 685, 692)581.7 of 656runs 577 to 58588.7%runs 88.0% to 89.2%504runs 474 to 530947runs 694 to 1,165≤ $1.040≤ $1,040≤ $1.040≤ $1,040 each runincl. searchper run, in Each run
Perplexity Decider 27B v1.2 API Open · 27BTurkish grammar, 1,542 · mean of 3 runs (633, 684, 687)875.7 of 1,542runs 874 to 87856.8%runs 56.7% to 56.9%248runs 234 to 257635runs 362 to 830$0.007$7.11$0.007$7.11 each runno search feeper run, in Each run
Perplexity Decider 27B v1.2 API · with web Open · 27BTurkish grammar, 1,542 · mean of 3 runs (640, 685, 692)918.3 of 1,542runs 901 to 92859.6%runs 58.4% to 60.2%472runs 447 to 500816runs 616 to 1,060≤ $1.032≤ $1,032runs ≤ $1.032≤ $1,032 to ≤ $1.033≤ $1,033incl. searchper run, in Each run
Perplexity Decider 27B v1.2 API Open · 27BTurkish pilot, 200 · run 631142 of 20071.0%247476$0.008$7.71no search fee8 (p = 1.000)
Perplexity Decider 27B v1.2 API · with web Open · 27BTurkish pilot, 200 · run 632143 of 20071.5%476907≤ $1.035≤ $1,035incl. search9 (p = 1.000)
Perplexity Decider 27B v1.2 API Open · 27BPilot reading, 68 · run 63162 of 6891.2%250456$0.009$9.37no search fee0 (p = 1.000)
Perplexity Decider 27B v1.2 API · with web Open · 27BPilot reading, 68 · run 63262 of 6891.2%484952≤ $1.041≤ $1,041incl. search0 (p = 1.000)
Perplexity Decider 27B v1.2 API Open · 27BPilot grammar, 132 · run 63180 of 13260.6%246500$0.007$6.86no search fee8 (p = 1.000)
Perplexity Decider 27B v1.2 API · with web Open · 27BPilot grammar, 132 · run 63281 of 13261.4%466907≤ $1.032≤ $1,032incl. search9 (p = 1.000)
Perplexity Decider 27B v1.2 API Open · 27BEnglish, 547 kept reading · mean of 2 runs (637, 688)484 of 547484 each run88.5%88.5% each run246runs 235 to 256546runs 394 to 698$0.008$7.79$0.008$7.79 each runno search feeper run, in Each run
Perplexity Decider 27B v1.2 API · with web Open · 27BEnglish, 547 kept reading · mean of 2 runs (639, 693)490.5 of 547runs 486 to 49589.7%runs 88.8% to 90.5%471runs 457 to 484712runs 587 to 836≤ $1.036≤ $1,036≤ $1.036≤ $1,036 each runincl. searchper run, in Each run
Perplexity Decider 27B v1.2 API Open · 27BPWS-Decisions, 100 · mean of 3 runs50.7 of 100runs 50 to 5150.7%runs 50.0% to 51.0%244runs 240 to 249656runs 396 to 877$0.004$3.97$0.004$3.97 each runno search feeno per-question results
Perplexity Decider 27B v1.2 API · with web Open · 27BPWS-Decisions, 100 · mean of 3 runs87 of 10087 each run87.0%87.0% each run483runs 475 to 491728runs 619 to 876≤ $1.041≤ $1,041≤ $1.041≤ $1,041 each runincl. searchno per-question results

Two rows per set of questions: off, then with web; under each name, its questions and runs. The full bank, the English questions and PWS-Decisions: the mean of the runs, with the lowest and highest run under each value; each run, with its paired counts, is in Each run below. Times: per request from our machine; the 95th percentile by nearest rank. Cost: each run's input tokens at the price below, plus the fee per search for every request with web (≤ Upper-bound estimate: one $0.001 web search counted per request; Perplexity bills only completed searches.) Right only with this setup: the questions it got right and the other setup did not, with the exact McNemar p of the pair.

With “I don't know” (Turkish or English) as an extra option

SetupRight · wrong · the optioncountsAnswered(right + wrong) / questionsWrong of answerswrong / (right + wrong)Non-hallucinationthe option / (the option + wrong); higher is better
Perplexity Decider 27B v1.2 API Open · 27BTurkish, full bank, 2,198 · run 6411,469 · 726 · 399.9%33.1%0.4%
Perplexity Decider 27B v1.2 API · with web Open · 27BTurkish, full bank, 2,198 · run 6421,473 · 702 · 2399.0%32.3%3.2%
Perplexity Decider 27B v1.2 API Open · 27BTurkish pilot, 200 · run 634145 · 55 · 0100.0%27.5%0.0%
Perplexity Decider 27B v1.2 API · with web Open · 27BTurkish pilot, 200 · run 635147 · 51 · 299.0%25.8%3.8%
Perplexity Decider 27B v1.2 API Open · 27BEnglish, 547 kept questions, I don't know · run 643481 · 63 · 399.5%11.6%4.5%
Perplexity Decider 27B v1.2 API · with web Open · 27BEnglish, 547 kept questions, I don't know · run 644481 · 56 · 1098.2%10.4%15.2%
Perplexity Decider 27B v1.2 API Open · 27BPWS-Decisions, 100, I don't know · mean of 3 runs47.7 · 36 · 16.383.7%runs 83.0% to 84.0%43.0%runs 42.2% to 44.0%31.2%runs 30.2% to 32.7%
Perplexity Decider 27B v1.2 API · with web Open · 27BPWS-Decisions, 100, I don't know · mean of 3 runs84.3 · 8.3 · 7.392.7%runs 92.0% to 93.0%9.0%runs 8.6% to 9.7%46.8%runs 43.8% to 50.0%

Each run

Runs per mode: 3 runs on the full bank and 2 runs on the English questions: each mode's first run (the ranking's run), then its repeats with the same questions in the same order, answer key, runner and settings. Each run of one repeat number is paired with the other mode's run of the same number, on the first runs' questions.

SetupRightof the questionsAccuracyhigher is betterMedian per requestms; lower is better95th percentilems; lower is betterCostper 1,000 questionsper 1M questions; lower is betterRight only with this setupexact McNemar p
Perplexity Decider 27B v1.2 API Open · 27BTurkish, full bank, 2,198, run 1 of 3 · run 6331,458 of 2,19866.3%254697$0.008$7.70no search fee94 (p < 0.001)
Perplexity Decider 27B v1.2 API · with web Open · 27BTurkish, full bank, 2,198, run 1 of 3 · run 6401,511 of 2,19868.7%482953≤ $1.034≤ $1,034incl. search147 (p < 0.001)
Perplexity Decider 27B v1.2 API Open · 27BTurkish, full bank, 2,198, run 2 of 3 · run 6841,459 of 2,19866.4%255810$0.008$7.70no search fee100 (p = 0.001)
Perplexity Decider 27B v1.2 API · with web Open · 27BTurkish, full bank, 2,198, run 2 of 3 · run 6851,511 of 2,19868.7%5021,026≤ $1.034≤ $1,034incl. search152 (p = 0.001)
Perplexity Decider 27B v1.2 API Open · 27BTurkish, full bank, 2,198, run 3 of 3 · run 6871,462 of 2,19866.5%239684$0.008$7.70no search fee109 (p = 0.327)
Perplexity Decider 27B v1.2 API · with web Open · 27BTurkish, full bank, 2,198, run 3 of 3 · run 6921,478 of 2,19867.2%455638≤ $1.035≤ $1,035incl. search125 (p = 0.327)
Perplexity Decider 27B v1.2 API Open · 27BEnglish, kept reading, 547, run 1 of 2 · run 637484 of 54788.5%256698$0.008$7.79no search fee2 (p = 0.007)
Perplexity Decider 27B v1.2 API · with web Open · 27BEnglish, kept reading, 547, run 1 of 2 · run 639495 of 54790.5%484836≤ $1.036≤ $1,036incl. search13 (p = 0.007)
Perplexity Decider 27B v1.2 API Open · 27BEnglish, kept reading, 547, run 2 of 2 · run 688484 of 54788.5%235394$0.008$7.79no search fee6 (p = 0.791)
Perplexity Decider 27B v1.2 API · with web Open · 27BEnglish, kept reading, 547, run 2 of 2 · run 693486 of 54788.8%457587≤ $1.036≤ $1,036incl. search8 (p = 0.791)

Paired counts, off against with web, per run

RunsBoth rightquestionsRight only offquestionsRight only with webquestionsBoth wrongquestionsErrorsoff · with web; scored wrongMissingoff · with web; not answeredExact McNemar pright only off against only with web
Turkish, full bank, run 1 of 3runs 633 and 640; 2,198 questions1,364941475930 · 00 · 0p < 0.001
Turkish, full bank, run 2 of 3runs 684 and 685; 2,198 questions1,3591001525870 · 00 · 0p = 0.001
Turkish, full bank, run 3 of 3runs 687 and 692; 2,198 questions1,3531091256110 · 00 · 0p = 0.327
English, kept reading, run 1 of 2runs 637 and 639; 547 questions482213500 · 00 · 0p = 0.007
English, kept reading, run 2 of 2runs 688 and 693; 547 questions47868550 · 00 · 0p = 0.791

Our method

Model pplx-decider-v1.2 through Perplexity's Decisions API, one question per request, the Text tab's System One prompts. Each pair of runs differs only in the request's mode: off, or web_grounding, which runs a web search per question and adds its results as evidence. Runs from 5 November 2026 to 5 November 2026. Runs per mode: 3 runs on the full bank and 2 runs on the English questions: each mode's first run and its repeats with the same questions in the same order, answer key, runner and settings; the ranking keeps each setup's first run, and the repeats enter only this page.

  • Turkish, full bank: 2,198 questions = 656 reading + 1,542 grammar (first runs 633 and 640, the Text tab rows Perplexity Decider 27B v1.2 · API and Perplexity Decider 27B v1.2 · API · with web). Repeats with the same questions in the same order: repeat 2: runs 684 and 685; repeat 3: runs 687 and 692. Reading: units 1, 2, 3, 4, 5 and 6.
  • Turkish pilot: 200 = 68 reading + 132 grammar, the runner's sample with seed 0 (runs 631 and 632).
  • “I don't know” as an extra option in Turkish, as on the Hallucination tab: the full bank (2,198 questions, runs 641 and 642) and the pilot (200, runs 634 and 635). “I don't know” as an extra option in English, as on English and Turkish: the 547 kept questions (runs 643 and 644).
  • English: the 547 kept reading questions of the English pages in our translations (first runs 637 and 639). Repeats with the same questions in the same order: repeat 2: runs 688 and 693.
  • PWS-Decisions: its 100 public English questions with the benchmark's own request, 3 runs per mode by us on 5 November 2026, as on PWS-Decisions; the source checks come from our first grounded run of 5 November 2026.

Accuracy: right answers over the questions asked. Time: each request's wall-clock time on our machine; the median and the 95th percentile by nearest rank. Cost per 1M questions: the run's input tokens at $0.02 per 1M plus $0.001 per request with web, over its questions. A mean of several runs: the mean of the runs' values, with the lowest and highest run. The paired test: the exact McNemar test on the questions one arm got right and the other did not, per pair of runs.

Download: results.json (every number on this page but PWS-Decisions' means, under web_grounding), answers.json (the pilots and the repeats under web_grounding_runs, the other runs in their own lists), web-grounding.json (tokens by group, the source checks' question ids and the first PWS-Decisions run's aggregates) and pws-decisions.json (PWS-Decisions' runs). reproduce.py recomputes results.json.