Decision model
Perplexity Decider 27B v1
1 question per request
60.4% of the 2,198 questions right (58.3% to 62.4%, 95% interval). Back to the leaderboard.
How we called it
- Kind
- Decision model
- Access
- Perplexity Decisions API
- Paid as
- Provider service
- Model ID
pplx-decider-v1-27b- Parameters
- 26.1B
- Confidence from
- Option score
- Run
- 44, 4 October 2026
Settings
- Reasoning
- off
- Request format
- 1 question per request
- Requests at once
- 1
- Temperature
- not recorded
- Harness seed
- 0
- Harness commit
- not recorded
- Prompt version
- not recorded
Results
| Questions | Accuracy | 95% interval | Right |
|---|---|---|---|
| All questions | 60.4% | 58.3% to 62.4% | 1,327 of 2,198 |
| Reading | 86.0% | 83.1% to 88.4% | 564 of 656 |
| Grammar | 49.5% | 47.0% to 52.0% | 763 of 1,542 |
Comparison group: uvwxy. Compare with another setup.
Accuracy by topic
Confidence
- Answers kept at 99% or more
- 21, of which 0 wrong (share right: 84.5% to 100.0%, 95% interval); 2,177 skipped, the model less than 99% sure
- Calibration gap
- 5.0 points
- AUROC, all questions
- 0.83 0.81 to 0.84
- AUROC, reading
- 0.91 0.88 to 0.93
- AUROC, grammar
- 0.75 0.73 to 0.78
- Brier score
- 0.167
- Accuracy of its surest answers
- most confident fifth: 99.3% right; most confident half: 84.2% right; most confident four fifths: 69.3% right
- Typical score of a right answer
- 70.8%
- Typical score of a wrong answer
- 34.4%
Repeats
Run 5 times on all questions.
- Answers changed in 5 runs
- 2 of 2,198
- Same wrong answer in all 5
- 869 of 2,198
- Wrong in at least one run
- 871 of 2,198
- Accuracy in each run
- 60.4%, 60.4%, 60.4%, 60.4%, 60.4%
Latency
- Latency, median request
- 0.216 s (its repeated runs)
- Latency, 90th percentile
- 0.272 s
- Requests at once
- 1
- Throughput
- 284.5 questions per minute of request time
- Request time
- 7.8 minutes, over the requests whose answers were kept
Latency: the time from sending one request to receiving its answer, measured from our computer, network and any retries within the request included. Request time: the time at least one request was in flight, over the requests whose answers were kept. It is not the run's elapsed time: it leaves out requests that failed and any pause before a run was resumed.
Cost
- $ per 1,000 questions
- $0.016
- Whole run
- $0.034
- Basis
- Estimated from published prices and recorded token usage
- Price
- $0.04 input, output free per 1M tokens (source, 2026-10-04)
- Input tokens per question
- 388
- Output tokens per question
- 1
Going first in front of an LLM
Replay of recorded answers, cutoff chosen on half the questions, averaged over the splits. Savings are estimates from API prices; the Method page explains how.
| LLM behind it | Share taken | Accuracy change | Saving |
|---|---|---|---|
| Gemini 3.8 Flashhigh reasoning · 16 questions per request | 18.8% | −0.11 points | 18% |
| GPT-6 Astrahigh reasoning · 16 questions per request | 18.6% | −0.10 points | 18% |
| GPT-6.1 Solhigh reasoning · 16 questions per request | 19.8% | −0.11 points | 19% |
| GPT-6 Astralow reasoning · 16 questions per request | 18.6% | −0.10 points | 18% |
| GPT-6 Astralow reasoning · 1 question per request | 18.6% | −0.10 points | 19% |
| Gemini 3.8 Flashlow reasoning · 1 question per request | 20.5% | −0.16 points | 19% |
| GPT-6.1 Sollow reasoning · 1 question per request | 19.8% | −0.12 points | 19% |
| GPT-6.1 Sollow reasoning · 16 questions per request | 18.6% | −0.10 points | 17% |
| Claude Opus 5.5high reasoning · 16 questions per request | 18.8% | −0.10 points | 19% |
| GPT-6.1 Terrahigh reasoning · 16 questions per request | 18.6% | −0.10 points | 16% |
| Claude Opus 5.5low reasoning · 1 question per request | 20.1% | −0.14 points | 20% |
| Gemini 3.8 Flashlow reasoning · 16 questions per request | 20.8% | −0.14 points | 15% |
| GPT-6.1 Terralow reasoning · 16 questions per request | 21.2% | −0.15 points | 18% |
| Claude Opus 5.5low reasoning · 16 questions per request | 20.0% | −0.15 points | 20% |
| Claude Sonnet 5.5high reasoning · 1 question per request | 23.9% | −0.15 points | 24% |
| Claude Sonnet 5.5high reasoning · 16 questions per request | 36.2% | −0.21 points | 36% |