Compared with Artificial Analysis

Artificial Analysis's Decision Index models, on our benchmark

Our five reply charts

Column chart, English reading: % right of 17 decision models, highest first. d1-ultra 90.7%, GLiDE v1.2 90.3%, Mammoth-1 90.1%, vLLM-SR d3 89.8%, Drex Max v1 89.6%, d3-ultra 89.4%, Decider v1.2 88.7%, Drex Max 1.1 88.5%, d1 86.8%, Jev 1.14 86.1%, SubQ Snap 1.1 85.9%, Mercury Decide 85.7%, Microsoft-Decision-1 85.7%, Elephant-1 85.4%, Celeris-1 84.1%, GPT-6 Luna 84.1%, Decider Flash v1.2 81.9%.
Card 1 of 5 · PNG, 2,000 × 1,600
Column chart, Turkish reading and grammar: % right of the same 17 decision models, highest first. d1-ultra 77.1%, GLiDE v1.2 68.5%, vLLM-SR d3 and d3-ultra 67.8%, Drex Max v1 67.7%, Drex Max 1.1 67.4%, Mammoth-1 and Decider v1.2 67.1%, SubQ Snap 1.1 63.9%, GPT-6 Luna 63.6%, Celeris-1 60.5%, Jev 1.14 59.2%, Mercury Decide 58.3%, d1 57.0%, Elephant-1 52.4%, Decider Flash v1.2 47.8%, Microsoft-Decision-1 46.5%.
Card 2 of 5 · PNG, 1,225 × 980
Two column charts, Turkish and English, on the same 547 reading questions: non-hallucination rate of 7 models, with the share of questions each answered. Turkish: d1-ultra 49.1%, Mammoth-1 43.9%, SubQ Snap 1.1 29.7%, Drex Max 1.1 27.3%, Elephant-1 22.6%, GLiDE v1.2 16.1%, Decider v1.2 1.9%. English: d1-ultra 56.7%, Drex Max 1.1 51.2%, Mammoth-1 50.7%, GLiDE v1.2 45.5%, Elephant-1 25.6%, SubQ Snap 1.1 25.0%, Decider v1.2 3.0%.
Card 3 of 5 · PNG, 1,225 × 980
Scatter chart, our Turkish questions: % right against US$ per 1 million questions (log scale) for AA's three tied models and ten other decision models. d1-ultra 77.1% at $14.3, GLiDE v1.2 68.5% at $14.4, Decider v1.2 67.1% at $7.8, SubQ Snap 1.1 63.9% at $15.4, GPT-6 Luna 63.6% at $40.4, Celeris-1 60.5% at $15.3, Jev 1.14 59.2% at $28.8, Mercury Decide 58.3% at $7.7, Clef 57.3% at $110, d1 57.0% at $7.4, Elephant-1 52.4% at $13.9, Decider Flash v1.2 47.8% at $3.9, Microsoft-Decision-1 46.5% at $13.8.
Card 4 of 5 · PNG, 1,225 × 980
Column chart: for 17 decision models, the answers each gave with at least 95% probability on 2,198 Turkish questions, and the % of those that were wrong. d1-ultra kept 549, none wrong; Celeris-1 kept 123, none wrong; SubQ Snap 1.1 kept 665, 2.7% wrong; Elephant-1 kept 393, 3.3% wrong; Mercury Decide kept 775, 8.6% wrong.
Card 5 of 5 · PNG, 1,225 × 980

Our numbers for 13 of the 14 decision models of Artificial Analysis's Decision Index v1.0; our tests differ from theirs, so the two rank different things.

The cards of our thread on X in reply to Artificial Analysis's post, in the thread's order (results 8f69ed8d); each opens as an image.

Artificial Analysis does not name its GLiDE version. The rest of this page uses Fastino's hosted GLiDE; the cards show GLiDE v1.2, which we ran on our GPU.

Accuracy

Share of the Text tab's 2,198 questions answered right · higher is better

  1. d1-ultra77.1%
  2. GLiDE API67.9%
  3. Perplexity Decider 27B v1.2 DGX Spark67.1%
  4. SubQ Snap 1.163.9%
  5. GPT-6 Luna Decisions API63.6%
  6. celeris-1-decision60.5%
  7. Jev 1.1459.2%
  8. Mercury Decide58.3%
  9. Clef57.3%
  10. d157.0%
  11. Elephant-152.4%
  12. Perplexity Decider Flash v1.247.8%
  13. Microsoft-Decision-146.5%

Decision models: worst 46.5%best 77.1% (higher is better)

Non-hallucination rate

“Bilmiyorum” among the answers that were not right, with it offered as a sixth option (the Hallucination tab); beside it the share answered and the error rate among the answers · higher non-hallucination is better, read with the share answered

  1. d1-ultra75.1%71.9%13.0%
  2. GLiDE API59.4%76.3%21.2%
  3. SubQ Snap 1.147.1%80.1%27.8%
  4. Elephant-134.8%81.6%42.3%
  5. Perplexity Decider Flash v1.214.7%91.8%51.7%
  6. Clef7.4%96.8%42.0%
  7. Perplexity Decider 27B v1.2 DGX Spark0.3%99.9%32.7%

Decision models: worst 0.3%best 75.1% (higher is better) Not run with “Bilmiyorum”: GPT-6 Luna · Decisions API, celeris-1-decision, Jev 1.14, Mercury Decide, d1 and Microsoft-Decision-1.

Cost per 1M questions

Estimated from list prices and the token use recorded in each run, in US dollars per 1 million questions (the Text tab gives it per 1,000 questions) · lower is better

  1. Perplexity Decider Flash v1.2$4
  2. d1$7
  3. Mercury Decide$8
  4. Perplexity Decider 27B v1.2 DGX Spark$8
  5. Microsoft-Decision-1$14
  6. Elephant-1$14
  7. d1-ultra$14
  8. celeris-1-decision$15
  9. SubQ Snap 1.1$15
  10. Jev 1.14$29
  11. GPT-6 Luna Decisions API$40
  12. Clef$110
  13. GLiDE API$160

Decision models: worst $160best $4 (lower is better) Costs: API list prices and recorded token use. Setups on our own GPUs: the maker's list price for the same model (each value's tooltip names it); our hardware cost is not measured.

Median latency per request

Time from sending one request to its answer, measured from our computer: on our GPU for the setups run there, through the network for the APIs (with OpenRouter’s time for those via OpenRouter) · lower is better

  1. Elephant-10.074 s
  2. Perplexity Decider Flash v1.20.132 s
  3. GPT-6 Luna Decisions API0.191 s
  4. Clef0.205 s
  5. Jev 1.140.211 s
  6. d1-ultra0.223 s
  7. Microsoft-Decision-10.286 s
  8. d10.305 s
  9. Mercury Decide0.313 s
  10. SubQ Snap 1.10.378 s
  11. celeris-1-decision0.401 s
  12. Perplexity Decider 27B v1.2 DGX Spark0.580 s
  13. GLiDE API2.09 s

Decision models: worst 2.09 sbest 0.074 s (lower is better) GPT-6 Luna · Decisions API and GLiDE · API sent 4 requests at a time; the others one.

Every number, model by model

Ranked by accuracy. Accuracy, reading and grammar: each model's run on the Text tab, one question per request, on all its questions and on each part. Page images: the Page images tab, the page's picture alone (condition B). Non-hallucination and attempt rates: the Hallucination tab, each model's first run (d1-ultra and GLiDE · API have five; that tab gives their range). Cost: estimated from list prices and the token use recorded in each run, per 1,000 questions as on the Text tab, and the same times 1,000 for comparison with Artificial Analysis's cost per 1 million tasks. Artificial Analysis's Elephant 9B is Base Labs' elephant-1, our Elephant-1.

ModelAccuracy2,198 questions; higher is betterReading656 questions; higher is betterGrammar1,542 questions; higher is betterPage images, picture alone763 questions; higher is betterNon-hallucinationof answers not right, % “Bilmiyorum” · higher is betterAnswerednot “Bilmiyorum”: (right + wrong) / 2,198Error rateof its answers: wrong / (right + wrong) · lower is betterCost per 1,000 questionsestimated; lower is betterCost per 1M questionsthe same, × 1,000; lower is betterMedian latency per requestseconds; lower is better
d1-ultraArtificial Analysis: d1-ultra77.1%1,694 of 2,19892.1%604 of 65670.7%1,090 of 1,54250.1%382 of 76375.1%617 “Bilmiyorum” of 822 not right71.9%1,581 of 2,198 answered13.0%$0.014$140.223 s
GLiDE APIArtificial Analysis: GLiDE67.9%1,492 of 2,19887.8%576 of 65659.4%916 of 1,542Not measured59.4%521 “Bilmiyorum” of 877 not right76.3%1,677 of 2,198 answered21.2%$0.160$1602.09 s4 in flight
Perplexity Decider 27B v1.2 DGX SparkArtificial Analysis: Decider v1.267.1%1,474 of 2,19889.2%585 of 65657.7%889 of 1,542No image input0.3%2 “Bilmiyorum” of 719 not right99.9%2,196 of 2,198 answered32.7%$0.008$80.580 s
SubQ Snap 1.1Artificial Analysis: SubQ Snap 1.163.9%1,404 of 2,19887.0%571 of 65654.0%833 of 1,54245.7%349 of 76347.1%437 “Bilmiyorum” of 927 not right80.1%1,761 of 2,198 answered27.8%$0.015$150.378 s
GPT-6 Luna Decisions APIArtificial Analysis: GPT-6 Luna (Decisions)63.6%1,398 of 2,19885.5%561 of 65654.3%837 of 1,54232.6%249 of 763Not runNot runNot run$0.040$400.191 s4 in flight
celeris-1-decisionArtificial Analysis: Celeris-1 Decision60.5%1,329 of 2,19885.2%559 of 65649.9%770 of 1,542Not measuredNot runNot runNot run$0.015$150.401 s
Jev 1.14Artificial Analysis: Jev 1.1459.2%1,301 of 2,19885.1%558 of 65648.2%743 of 1,54243.3%330 of 763Not runNot runNot run$0.029$290.211 s
Mercury DecideArtificial Analysis: Mercury Decide58.3%1,281 of 2,19884.5%554 of 65647.1%727 of 1,542Not measuredNot runNot runNot run$0.008$80.313 s
ClefArtificial Analysis: Clef57.3%1,259 of 2,19885.1%558 of 65645.5%701 of 1,542Not measured7.4%71 “Bilmiyorum” of 965 not right96.8%2,127 of 2,198 answered42.0%$0.110$1100.205 s
d1Artificial Analysis: d157.0%1,253 of 2,19882.8%543 of 65646.0%710 of 1,542Not measuredNot runNot runNot run$0.007$70.305 sits repeat runs
Elephant-1Artificial Analysis: Elephant 9B52.4%1,151 of 2,19880.5%528 of 65640.4%623 of 1,542Not measured34.8%405 “Bilmiyorum” of 1,163 not right81.6%1,793 of 2,198 answered42.3%$0.014$140.074 s
Perplexity Decider Flash v1.2Artificial Analysis: Decider Flash v1.247.8%1,050 of 2,19872.7%477 of 65637.2%573 of 1,542Not measured14.7%180 “Bilmiyorum” of 1,224 not right91.8%2,018 of 2,198 answered51.7%$0.004$40.132 s
Microsoft-Decision-1Artificial Analysis: Microsoft-Decision-146.5%1,021 of 2,19873.5%482 of 65635.0%539 of 1,542Not measuredNot runNot runNot run$0.014$140.286 s

Artificial Analysis's post of 22 October 2026 ranks 14 decision models on its Decision Index v1.0, a weighted average of four of its own evaluations (AA-Omniscience-Decision, HLE-MultiChoice, When2Call and WildGuardTest). 13 of them are on our leaderboard; Clef-Omni is not. This page gathers our numbers for those 13, from our published results, and opens with our reply charts. It ranks nothing new and changes no number on the other pages.

Download: results.json (the Text and Hallucination tabs' numbers) and vision.json (the Page images tab's).