English reading

The reading questions in English

Every setup in English

ModelEnglish accuracyfirst English run, or the mean of a setup's three seed runs; higher is betterDecision Score (AA formula)first English run; higher is betterTurkish accuracyText tab run, same 547; for contextEnglish non-hallucinationof answers not right, % “I don't know” · higher is better; answered and error rate under itEnglish cost$ per 1,000 questionsper 1M questions, first English run; lower is betterEnglish latencymedian seconds per request, first English run; lower is better
Claude Fable 5.2 max · chat Closed · Flagship98.0%536 of 547 rightnot applicable97.8%535 of 547 right28.6%4 “I don't know” of 14 not rightanswered 99.3%error rate 1.8%52.94252,942subscription, priced as the API74.016 questions per request, 1 in flight
Gemini 4 Hydrogen high · chat Closed · Workhorse96.7%529 of 547 rightnot applicable97.1%531 of 547 right20.0%4 “I don't know” of 20 not rightanswered 99.3%error rate 2.9%4.9274,927subscription, priced as the API28.416 questions per request, 1 in flight
GPT-6 Astra low · chat Closed · Flagship96.3%527 of 547 rightnot applicable96.7%529 of 547 right18.2%4 “I don't know” of 22 not rightanswered 99.3%error rate 3.3%5.9185,918subscription, priced as the API11.516 questions per request, 1 in flight
GPT-6 Astra high · chat Closed · Flagship96.2%526 of 547 rightnot applicable96.9%530 of 547 right19.0%4 “I don't know” of 21 not rightanswered 99.3%error rate 3.1%6.3346,334subscription, priced as the API15.616 questions per request, 1 in flight
GPT-6.1 Terra high · chat Closed · Workhorse96.2%526 of 547 rightnot applicable96.3%527 of 547 right16.7%4 “I don't know” of 24 not rightanswered 99.3%error rate 3.7%0.509509subscription, priced as the API7.9416 questions per request, 1 in flight
GPT-6.1 Terra low · chat Closed · Workhorse96.2%526 of 547 rightnot applicable95.6%523 of 547 right15.4%4 “I don't know” of 26 not rightanswered 99.3%error rate 4.1%0.491491subscription, priced as the API7.2916 questions per request, 1 in flight
Claude Fable 5.1 high · chat Closed · Flagship96.0%525 of 547 rightnot applicable95.4%522 of 547 right7.4%2 “I don't know” of 27 not rightanswered 99.6%error rate 4.6%11.53111,531subscription, priced as the API15.616 questions per request, 1 in flight
Gemini 3.8 Flash high · chat Closed · Workhorse96.0%525 of 547 rightnot applicable96.9%530 of 547 right10.0%2 “I don't know” of 20 not rightanswered 99.6%error rate 3.3%3.6953,695at list prices104.616 questions per request, 4 in flight
Gemini 4 Argon low · chat Closed · Flagship96.0%525 of 547 rightnot applicable96.9%530 of 547 right18.5%5 “I don't know” of 27 not rightanswered 99.1%error rate 4.1%1.9001,900at list prices6.0316 questions per request, 1 in flight
GPT-6 Astra medium · chat Closed · Flagship96.0%525 of 547 rightnot applicable96.7%529 of 547 right16.7%4 “I don't know” of 24 not rightanswered 99.3%error rate 3.7%4.6964,696subscription, priced as the API10.716 questions per request, 1 in flight
GPT-6.1 Sol low · chat Closed · Workhorse95.8%524 of 547 rightnot applicable95.8%524 of 547 right14.8%4 “I don't know” of 27 not rightanswered 99.3%error rate 4.2%1.2191,219subscription, priced as the API10.316 questions per request, 1 in flight
GPT-6 Astra low · chat · 1 per request Closed · Flagship95.6%523 of 547 rightnot applicable97.1%531 of 547 right14.8%4 “I don't know” of 27 not rightanswered 99.3%error rate 4.2%28.65328,653subscription, priced as the API6.431 question per request, 1 in flight
GPT-6.1 Sol high · chat Closed · Workhorse95.6%523 of 547 rightnot applicable96.2%526 of 547 right16.7%4 “I don't know” of 24 not rightanswered 99.3%error rate 3.7%1.4271,427subscription, priced as the API16.916 questions per request, 1 in flight
Gemini 3.8 Flash low · chat · 1 per request Closed · Workhorse95.2%521 of 547 rightnot applicable96.2%526 of 547 right7.7%2 “I don't know” of 26 not rightanswered 99.6%error rate 4.4%0.858858at list prices2.851 question per request, 1 in flight
GPT-6.1 Sol low · chat · 1 per request Closed · Workhorse95.2%521 of 547 rightnot applicable95.6%523 of 547 right12.9%4 “I don't know” of 31 not rightanswered 99.3%error rate 5.0%6.1136,113subscription, priced as the API5.591 question per request, 1 in flight
Claude Opus 5.5 low · chat · 1 per request Closed · Flagship94.7%518 of 547 rightnot applicable95.4%522 of 547 right0.0%0 “I don't know” of 31 not rightanswered 100.0%error rate 5.7%10.09410,094subscription, priced as the API3.501 question per request, 1 in flight
Claude Opus 5.5 high · chat Closed · Flagship94.5%517 of 547 rightnot applicable95.6%523 of 547 right6.5%2 “I don't know” of 31 not rightanswered 99.6%error rate 5.3%4.5194,519subscription, priced as the API11.316 questions per request, 1 in flight
Qwen4-Max high · chat Closed · Workhorse94.5%517 of 547 rightnot applicable94.7%518 of 547 right0.0%0 “I don't know” of 29 not rightanswered 100.0%error rate 5.3%1.1191,119at list prices29.516 questions per request, 4 in flight
Gemini 3.8 Flash low · chat Closed · Workhorse94.3%516 of 547 rightnot applicable95.2%521 of 547 right3.4%1 “I don't know” of 29 not rightanswered 99.8%error rate 5.1%0.288288at list prices4.1916 questions per request, 1 in flight
Gemini 4 Carbon high · chat Closed · Flagship94.3%516 of 547 rightnot applicable97.6%534 of 547 right25.0%4 “I don't know” of 16 not rightanswered 99.3%error rate 2.2%38.05338,053subscription, priced as the API287.816 questions per request, 1 in flight
KnowLine-31B-Gen1.5 Open · 31B93.6%512 of 547 right94.33chosen-answer Brier 0.049494.3%516 of 547 right0.0%0 “I don't know” of 37 not rightanswered 100.0%error rate 6.8%No list price1.121 question per request, 1 in flight
Grok 4.7 high · chat Closed · Workhorse93.4%511 of 547 rightnot applicable95.1%520 of 547 right27.9%12 “I don't know” of 43 not rightanswered 97.8%error rate 5.8%6.8516,851subscription, priced as the API197.216 questions per request, 4 in flight
Claude Opus 5.5 low · chat Closed · Flagship93.1%509 of 547 rightnot applicable95.1%520 of 547 right4.7%2 “I don't know” of 43 not rightanswered 99.6%error rate 7.5%1.0871,087subscription, priced as the API7.6416 questions per request, 1 in flight
Copernicus-31B Open · 31B92.9%508 of 547 right93.66chosen-answer Brier 0.055694.5%517 of 547 right0.0%0 “I don't know” of 39 not rightanswered 100.0%error rate 7.1%0.142142from GPU time0.1881 question per request, 1 in flight
Claude Haiku 5.5 max · chat Closed · Efficient92.7%507 of 547 rightnot applicable89.4%489 of 547 right7.8%4 “I don't know” of 51 not rightanswered 96.3%error rate 8.9%1.2611,261at list prices148.916 questions per request, 4 in flight
DeepSeek-V4.1-Flash reasoning on · chat · 1017 Open · 552B MoE · 16B active92.5%506 of 547 rightnot applicable94.5%517 of 547 right7.1%3 “I don't know” of 42 not rightanswered 99.5%error rate 7.2%0.421421at the maker's list price189.416 questions per request, 8 in flight
REAX 1.3 Open · 57B92.5%506 of 547 right93.75chosen-answer Brier 0.050094.0%514 of 547 right6.7%3 “I don't know” of 45 not rightanswered 99.5%error rate 7.7%0.169169from GPU time0.2291 question per request, 1 in flight
Kimi K3.1 Pro low · chat Open · 2.8T MoE · 104B active92.3%505 of 547 rightnot applicable95.2%521 of 547 right7.3%3 “I don't know” of 41 not rightanswered 99.5%error rate 7.0%0.984984at list prices16.616 questions per request, 4 in flight
KnowLine-31B-Gen2 Open · 31B92.3%505 of 547 right93.37chosen-answer Brier 0.055894.5%517 of 547 right0.0%0 “I don't know” of 40 not rightanswered 100.0%error rate 7.3%0.127127from GPU time0.1711 question per request, 1 in flight
Karpo Decision 31B Open · 31B92.1%504 of 547 right93.23chosen-answer Brier 0.056894.9%519 of 547 right0.0%0 “I don't know” of 42 not rightanswered 100.0%error rate 7.7%0.145145from GPU time0.2071 question per request, 1 in flight
Claude Sonnet 5.5 high · chat · 1 per request Closed · Workhorse91.6%501 of 547 rightnot applicable94.3%516 of 547 right0.0%0 “I don't know” of 38 not rightanswered 100.0%error rate 6.9%5.5135,513subscription, priced as the API3.021 question per request, 1 in flight
Kimi K3.1 Pro no reasoning · chat Open · 2.8T MoE · 104B active91.4%500 of 547 rightnot applicable94.0%514 of 547 right7.5%4 “I don't know” of 53 not rightanswered 99.3%error rate 9.0%0.735735at list prices6.3116 questions per request, 4 in flight
Qwen3.5-397B-A17B high · chat Open · 397B MoE · 17B active91.4%500 of 547 rightnot applicable91.4%500 of 547 right6.2%3 “I don't know” of 48 not rightanswered 99.5%error rate 8.3%1.7421,742at list prices162.916 questions per request, 4 in flight
Gemini Decision 1 20261111 Closed · Efficient91.0%498 of 547 right92.31chosen-answer Brier 0.064292.9%508 of 547 right9.1%5 “I don't know” of 55 not rightanswered 99.1%error rate 9.2%0.04039.51at list prices0.2351 question per request, 1 in flight
Logits Sigma-1 Duo Open · 57B91.0%498 of 547 right92.55chosen-answer Brier 0.059593.4%511 of 547 right4.3%2 “I don't know” of 46 not rightanswered 99.6%error rate 8.1%0.191191from GPU time0.2631 question per request, 1 in flight
GLiDE 27B Open · 27B90.9%497 of 547 right92.36chosen-answer Brier 0.061391.2%499 of 547 right37.9%22 “I don't know” of 58 not rightanswered 96.0%error rate 6.9%No list price0.5451 question per request, 1 in flight
d1-ultra Closed · Efficient90.7%496 of 547 right91.92chosen-answer Brier 0.068493.6%512 of 547 right56.7%51 “I don't know” of 90 not rightanswered 90.7%error rate 7.9%0.01413.98at list prices0.2201 question per request, 1 in flight
Sieve 31B v3 Open · 31B90.7%496 of 547 right92.51chosen-answer Brier 0.056593.4%511 of 547 right2.2%1 “I don't know” of 46 not rightanswered 99.8%error rate 8.2%No list price1.061 question per request, 1 in flight
Turion Jev 31B Open · 31B90.7%496 of 547 right91.69chosen-answer Brier 0.072993.6%512 of 547 right8.5%4 “I don't know” of 47 not rightanswered 99.3%error rate 7.9%No list price0.4451 question per request, 1 in flight
Gemma 4 31B reasoning off · chat Open · 31B90.5%495 of 547 rightnot applicable92.5%506 of 547 right3.6%2 “I don't know” of 55 not rightanswered 99.6%error rate 9.7%0.05352.81at a host's list price for the same weights3.1916 questions per request, 1 in flight
Perplexity Decider 27B v1.2 API · with web Open · 27B90.5%495 of 547 right91.39chosen-answer Brier 0.077190.7%496 of 547 right15.2%10 “I don't know” of 66 not rightanswered 98.2%error rate 10.4%≤ 1.036≤ 1,036at list prices0.4841 question per request, 1 in flight
GLiDE v1.2 Open · 27B90.3%494 of 547 right91.96chosen-answer Brier 0.063990.5%495 of 547 right45.5%30 “I don't know” of 66 not rightanswered 94.5%error rate 7.0%0.01514.55at the maker's list price0.1041 question per request, 1 in flight
d1.2-ultra Closed · Efficient90.3%494 of 547 right91.84chosen-answer Brier 0.066490.9%497 of 547 right89.3%109 “I don't know” of 122 not rightanswered 80.1%error rate 3.0%0.01413.98at list prices0.2211 question per request, 1 in flight
Mammoth-1 Open · 27B90.1%493 of 547 right91.75chosen-answer Brier 0.066392.9%508 of 547 right50.7%37 “I don't know” of 73 not rightanswered 93.2%error rate 7.1%0.04040.07at the maker's list price0.1891 question per request, 1 in flight
Gemma 4 31B reasoning off · chat · 1 per request Open · 31B89.9%492 of 547 rightnot applicable90.3%494 of 547 right0.0%0 “I don't know” of 59 not rightanswered 100.0%error rate 10.8%0.07070.17at a host's list price for the same weights0.2991 question per request, 1 in flight
d3 Open · 26B89.8%491 of 547 right91.65chosen-answer Brier 0.064691.0%498 of 547 right1.7%1 “I don't know” of 59 not rightanswered 99.8%error rate 10.6%0.05555.35from GPU time0.0741 question per request, 1 in flight
Drex Max Open · 27B89.6%490 of 547 right91.50chosen-answer Brier 0.065891.4%500 of 547 right3.2%2 “I don't know” of 62 not rightanswered 99.6%error rate 11.0%0.02423.56at the maker's list price0.0811 question per request, 1 in flight
GLM-5.3-Flash reasoning on · chat Open · 320B MoE · 18B active89.6%490 of 547 rightnot applicable90.1%493 of 547 right1.6%1 “I don't know” of 62 not rightanswered 99.8%error rate 11.2%0.08181.38at the maker's list price33.116 questions per request, 8 in flight
Muse Ember max · chat Closed · Efficient89.6%490 of 547 rightnot applicable90.9%497 of 547 right5.3%3 “I don't know” of 57 not rightanswered 99.5%error rate 9.9%0.245245at list prices11.616 questions per request, 4 in flight
Perplexity Decider 27B v1.3 Open · 27B89.6%490 of 547 right91.22chosen-answer Brier 0.071490.7%496 of 547 right8.8%5 “I don't know” of 57 not rightanswered 99.1%error rate 9.6%0.08382.63from GPU time0.1151 question per request, 1 in flight
d3-ultra Open · 26B89.4%489 of 547 right91.18chosen-answer Brier 0.070491.4%500 of 547 right8.2%5 “I don't know” of 61 not rightanswered 99.1%error rate 10.3%0.106106from GPU time0.1491 question per request, 1 in flight
d1.2 Closed · Efficient89.0%487 of 547 right90.51chosen-answer Brier 0.080188.8%486 of 547 right61.5%59 “I don't know” of 96 not rightanswered 89.2%error rate 7.6%0.0076.66at list prices0.2191 question per request, 1 in flight
MiMo-V2.6-Flash high · chat Open · 309B MoE · 15B active89.0%487 of 547 rightnot applicable86.8%475 of 547 right1.7%1 “I don't know” of 59 not rightanswered 99.8%error rate 10.6%0.298298at list prices6.671 question per request, 8 in flight
Qwen3.8-27B low · chat Open · 27B89.0%487 of 547 rightnot applicable90.7%496 of 547 right3.6%2 “I don't know” of 55 not rightanswered 99.6%error rate 9.7%0.853853at the maker's list price71.716 questions per request, 4 in flight
Perplexity Decider 27B v1.2 DGX Spark Open · 27B88.7%485 of 547 right90.75chosen-answer Brier 0.071791.4%500 of 547 right3.0%2 “I don't know” of 66 not rightanswered 99.6%error rate 11.7%0.0087.85at the maker's list price0.1261 question per request, 1 in flight
Drex Max v1.1 Closed · Efficient (provisional)88.5%484 of 547 right90.81chosen-answer Brier 0.068590.1%493 of 547 right51.2%43 “I don't know” of 84 not rightanswered 92.1%error rate 8.1%0.02323.38at list prices0.3351 question per request, 1 in flight
Perplexity Decider 27B v1 Open · 27B88.5%484 of 547 right90.64chosen-answer Brier 0.072089.0%487 of 547 rightNot run0.01615.59at list prices0.2471 question per request, 1 in flight
Perplexity Decider 27B v1.2 API Open · 27B88.5%484 of 547 right90.64chosen-answer Brier 0.072191.0%498 of 547 right4.5%3 “I don't know” of 66 not rightanswered 99.5%error rate 11.6%0.0087.79at list prices0.2561 question per request, 1 in flight
Inkling-Small high · chat · Jul 2026 Open · 276B MoE · 12B active88.3%483 of 547 rightnot applicable89.4%489 of 547 right0.0%0 “I don't know” of 60 not rightanswered 100.0%error rate 11.0%0.425425at list prices25.316 questions per request, 4 in flight
d1.1 Closed · Efficient88.3%483 of 547 right90.14chosen-answer Brier 0.080288.3%483 of 547 rightNot run0.0076.66at list prices0.1731 question per request, 1 in flight
GLM-5.3 low · chat Open · Workhorse87.8%480 of 547 rightnot applicable89.2%488 of 547 right3.4%2 “I don't know” of 58 not rightanswered 99.6%error rate 10.3%0.408408at list prices6.8416 questions per request, 4 in flight
Jeffan-9B-1211 Open · 9.0B87.6%479 of 547 right89.05chosen-answer Brier 0.094686.5%473 of 547 right0.0%0 “I don't know” of 71 not rightanswered 100.0%error rate 13.0%0.04645.85from GPU time0.0671 question per request, 1 in flight
GLM-5.3-Flash reasoning off · chat Open · 320B MoE · 18B active87.2%477 of 547 rightnot applicable84.3%461 of 547 right5.6%4 “I don't know” of 71 not rightanswered 99.3%error rate 12.3%0.04646.07at the maker's list price3.3716 questions per request, 1 in flight
DeepSeek-V4.1-Flash reasoning off · chat · 1017 Open · 552B MoE · 16B active87.0%476 of 547 rightnot applicable89.9%492 of 547 right5.4%4 “I don't know” of 74 not rightanswered 99.3%error rate 12.9%0.07979.31at the maker's list price6.5916 questions per request, 1 in flight
Qwen3.5-4B reasoning on · chat Open · 4.0B87.0%476 of 547 rightnot applicable82.8%453 of 547 right2.4%2 “I don't know” of 85 not rightanswered 99.6%error rate 15.2%1.3901,390from GPU time58.31 question per request, 32 in flight
Qwen3.8 Flash no reasoning · chat Closed · Efficient87.0%476 of 547 rightnot applicable88.8%486 of 547 right5.6%4 “I don't know” of 71 not rightanswered 99.3%error rate 12.3%0.05049.69at list prices10.116 questions per request, 4 in flight
d1 Closed · Efficient86.8%475 of 547 right89.26chosen-answer Brier 0.083285.0%465 of 547 right2.9%2 “I don't know” of 69 not rightanswered 99.6%error rate 12.3%0.0076.66at list prices0.1801 question per request, 1 in flight
StartLux-Decision-v1.1-9B-Instant Open · 9.0B86.8%475 of 547 right88.59chosen-answer Brier 0.096583.0%454 of 547 right2.6%2 “I don't know” of 76 not rightanswered 99.6%error rate 13.6%No list price0.1561 question per request, 1 in flight
Gemma 4 12B Unified default reasoning · chat Open · 12B86.8%475 of 547 rightnot applicableQueuedits Text tab run in logic's GPU 1 queue (Ümit #6527, logic #6955)5.5%4 “I don't know” of 73 not rightanswered 98.9%error rate 12.8%0.154154at a host's list price for the same weights15.81 question per request, 32 in flight
Clef Open · 27B86.7%474 of 547 right89.19chosen-answer Brier 0.082888.1%482 of 547 right1.3%1 “I don't know” of 76 not rightanswered 99.8%error rate 13.7%0.111111at the maker's list price0.2201 question per request, 1 in flight
KnowLine-12B-Gen1 Open · 12B86.7%474 of 547 right88.66chosen-answer Brier 0.093387.6%479 of 547 right1.3%1 “I don't know” of 77 not rightanswered 99.8%error rate 13.9%0.05756.86from GPU time0.0781 question per request, 1 in flight
GLiDE 9B Open · 7.9B86.3%472 of 547 right88.60chosen-answer Brier 0.090984.3%461 of 547 right16.7%14 “I don't know” of 84 not rightanswered 97.4%error rate 13.1%No list price0.2071 question per request, 1 in flight
GPT-6 Luna low · chat Closed · Efficient86.3%472 of 547 rightnot applicable88.7%485 of 547 right6.8%5 “I don't know” of 74 not rightanswered 99.1%error rate 12.7%0.05857.70subscription, priced as the API7.7216 questions per request, 1 in flight
OneJev-27B Open · 27B86.3%472 of 547 right88.51chosen-answer Brier 0.092787.8%480 of 547 right4.0%3 “I don't know” of 75 not rightanswered 99.5%error rate 13.2%0.106106from GPU time0.1301 question per request, 1 in flight
Claude Sonnet 5.5 high · chat Closed · Workhorse86.1%471 of 547 rightnot applicable87.6%479 of 547 right4.8%3 “I don't know” of 62 not rightanswered 99.5%error rate 10.8%1.9981,998subscription, priced as the API3.8516 questions per request, 1 in flight
Jev 1.14 Closed · Efficient86.1%471 of 547 right89.16chosen-answer Brier 0.077887.2%477 of 547 right4.0%3 “I don't know” of 75 not rightanswered 99.5%error rate 13.2%0.02726.64at list prices0.1811 question per request, 1 in flight
Cerno 9B v1.2 Open · 9.0B85.9%470 of 547 right88.42chosen-answer Brier 0.090783.5%457 of 547 right2.5%2 “I don't know” of 79 not rightanswered 99.6%error rate 14.1%No list price0.2111 question per request, 1 in flight
Drex 2.0 Open · 35B MoE · 3.0B active85.9%470 of 547 right88.15chosen-answer Brier 0.096384.8%464 of 547 rightNot run0.01919.42at list prices0.2831 question per request, 1 in flight
Jev 1.13 Closed · Efficient85.9%470 of 547 right88.86chosen-answer Brier 0.082186.1%471 of 547 rightNot run0.02726.64at list prices0.1731 question per request, 1 in flight
KnowLine-9B-Gen1 Open · 9.0B85.9%470 of 547 right88.32chosen-answer Brier 0.092882.4%451 of 547 right1.3%1 “I don't know” of 77 not rightanswered 99.8%error rate 13.9%0.04241.55from GPU time0.0591 question per request, 1 in flight
SubQ Snap 1.1 Closed · Efficient85.9%470 of 547 right88.70chosen-answer Brier 0.085289.2%488 of 547 right25.0%21 “I don't know” of 84 not rightanswered 96.2%error rate 12.0%0.01615.59at list prices0.3811 question per request, 1 in flight
SubQ Snap Preview Closed · Efficient85.9%470 of 547 right88.70chosen-answer Brier 0.085289.9%492 of 547 rightNot run0.01919.48at list prices0.3811 question per request, 1 in flight
Adjudex 9B v5 Open · 9.0B85.7%469 of 547 right88.35chosen-answer Brier 0.090583.9%459 of 547 right0.0%0 “I don't know” of 77 not rightanswered 100.0%error rate 14.1%0.02524.93from GPU time0.0341 question per request, 1 in flight
Mercury Decide Closed · Efficient85.7%469 of 547 right88.15chosen-answer Brier 0.094387.0%476 of 547 right13.4%11 “I don't know” of 82 not rightanswered 98.0%error rate 13.2%0.0087.79at list prices0.3101 question per request, 1 in flight
Microsoft-Decision-1 Closed · Efficient85.7%469 of 547 right88.10chosen-answer Brier 0.095476.4%418 of 547 right3.8%3 “I don't know” of 80 not rightanswered 99.5%error rate 14.2%0.01414.01at list prices0.3151 question per request, 1 in flight
Elephant-1 Open · 9.0B85.4%467 of 547 right87.99chosen-answer Brier 0.093983.5%457 of 547 right25.6%23 “I don't know” of 90 not rightanswered 95.8%error rate 12.8%0.01414.02at the maker's list price0.0701 question per request, 1 in flight
Open-Jev-27B-v1.1 Open · 27B85.4%467 of 547 right77.18chosen-answer Brier 0.310286.5%473 of 547 right17.4%16 “I don't know” of 92 not rightanswered 97.1%error rate 14.3%0.351351from GPU time0.4951 question per request, 1 in flight
KnowLine-9B-Gen2 Open · 9.0B85.2%466 of 547 right87.38chosen-answer Brier 0.104482.8%453 of 547 right1.3%1 “I don't know” of 79 not rightanswered 99.8%error rate 14.3%0.03939.48from GPU time0.0561 question per request, 1 in flight
Qwen3.8-27B stated confidence · reasoning off · chat Open · 27B84.8%464 of 547 rightnot applicable86.3%472 of 547 right3.7%3 “I don't know” of 81 not rightanswered 99.5%error rate 14.3%0.212212at the maker's list price0.3901 question per request, 1 in flight
Qwen3.8-27B yes/no scoring Open · 27B84.8%464 of 547 rightnot applicable84.6%463 of 547 rightNot runyes/no scoring cannot offer it0.741741at the maker's list price0.4131 question per request, 1 in flight
MiniMax-M3.1-Flash high · chat Open · 428B MoE · 23B active84.5%462 of 547 rightnot applicable86.3%472 of 547 right3.4%3 “I don't know” of 88 not rightanswered 99.5%error rate 15.6%0.193193at list prices26.716 questions per request, 4 in flight
celeris-1-decision Closed · Efficient84.1%460 of 547 right87.29chosen-answer Brier 0.095287.4%478 of 547 right71.3%92 “I don't know” of 129 not rightanswered 83.2%error rate 8.1%0.01616.09at list prices0.3901 question per request, 1 in flight
GPT-6 Luna Decisions API Closed · Efficient84.1%460 of 547 rightunavailable86.7%474 of 547 right9.4%10 “I don't know” of 106 not rightanswered 98.2%error rate 17.9%0.04141.09at list prices0.3231 question per request, 4 in flight
Claude Sonnet 5.5 low · chat Closed · Workhorse83.9%459 of 547 rightnot applicable84.8%464 of 547 right4.4%4 “I don't know” of 90 not rightanswered 99.3%error rate 15.8%0.354354subscription, priced as the API3.6816 questions per request, 1 in flight
Cygnet Open · 12B83.9%459 of 547 right87.03chosen-answer Brier 0.098485.6%468 of 547 right11.2%10 “I don't know” of 89 not rightanswered 98.2%error rate 14.7%0.04141.42at a host's list price for the same weights0.0431 question per request, 1 in flight
Qwen3.8-27B vote share · reasoning off · chat Open · 27B83.9%459 of 547 rightnot applicable85.9%470 of 547 right3.4%3 “I don't know” of 87 not rightanswered 99.5%error rate 15.4%0.645645at the maker's list price1.701 question per request, 1 in flight
Winnow-12B Open · 12B83.9%459 of 547 right86.18chosen-answer Brier 0.115685.0%465 of 547 right15.0%15 “I don't know” of 100 not rightanswered 97.3%error rate 16.0%0.05049.59from GPU time0.0701 question per request, 1 in flight
Perplexity Decider Flash v1.3 Open · 4.0B83.7%458 of 547 right86.96chosen-answer Brier 0.098078.8%431 of 547 right7.2%7 “I don't know” of 97 not rightanswered 98.7%error rate 16.7%0.02323.17from GPU time0.0331 question per request, 1 in flight
Valen-Decision-1.5-9B Open · 9.0B83.7%458 of 547 right86.43chosen-answer Brier 0.108779.0%432 of 547 right1.2%1 “I don't know” of 85 not rightanswered 99.8%error rate 15.4%0.04140.55from GPU time0.0591 question per request, 1 in flight
celeris-1.1-decision Closed · Efficient83.7%458 of 547 right87.14chosen-answer Brier 0.0946Not runEnglish only, no Text tab run74.8%98 “I don't know” of 131 not rightanswered 82.1%error rate 7.3%0.0021.61at list prices0.3851 question per request, 1 in flight
Dexy-Decision-9B-V2 Open · 9.0B83.5%457 of 547 right87.01chosen-answer Brier 0.095383.4%456 of 547 right1.1%1 “I don't know” of 91 not rightanswered 99.8%error rate 16.5%0.02525.41from GPU time0.0361 question per request, 1 in flight
vera-medium-omni Open · 4.2B83.5%457 of 547 right86.34chosen-answer Brier 0.108679.5%435 of 547 right7.4%7 “I don't know” of 94 not rightanswered 98.7%error rate 16.1%0.03736.63from GPU time0.0541 question per request, 1 in flight
StartLux-Decision-v1.1-4B-Instant Open · 4.0B82.8%453 of 547 right86.35chosen-answer Brier 0.101178.4%429 of 547 right0.0%0 “I don't know” of 94 not rightanswered 100.0%error rate 17.2%No list price0.0751 question per request, 1 in flight
LFM2.5-24B-A2B default reasoning · chat Open · 24B MoE · 2.5B active82.6%452 of 547 rightnot applicableQueuedits Text tab run not scheduled yet: needs the model served again (Ümit #6527)13.2%14 “I don't know” of 106 not rightanswered 97.4%error rate 17.3%1.7641,764at the maker's list price74.21 question per request, 32 in flight
Gemma 4 E4B default reasoning · chat Open · 8.0B82.4%451 of 547 rightnot applicable80.6%441 of 547 right9.8%10 “I don't know” of 102 not rightanswered 97.8%error rate 17.2%0.199199from GPU time8.391 question per request, 32 in flight
Clef-Flash Open · 9.0B82.3%450 of 547 right85.43chosen-answer Brier 0.114177.7%425 of 547 right5.3%5 “I don't know” of 95 not rightanswered 99.1%error rate 16.6%0.01817.53at the maker's list price0.1401 question per request, 1 in flight
Verdin-4B-v1.1 Open · 4.0B82.3%450 of 547 right85.97chosen-answer Brier 0.103277.5%424 of 547 right2.1%2 “I don't know” of 96 not rightanswered 99.6%error rate 17.2%0.03434.28from GPU time0.0521 question per request, 1 in flight
Perplexity Decider Flash v1.2 Open · 4.0B81.9%448 of 547 right86.04chosen-answer Brier 0.098276.6%419 of 547 right4.8%5 “I don't know” of 105 not rightanswered 99.1%error rate 18.5%0.0043.93at the maker's list price0.0371 question per request, 1 in flight
KnowLine 4B Gen5.5 Open · 4.0B80.8%442 of 547 right84.57chosen-answer Brier 0.116778.6%430 of 547 right0.9%1 “I don't know” of 106 not rightanswered 99.8%error rate 19.2%No list price0.0991 question per request, 1 in flight
Chinese Decision Pointer 4B Open · 4.0B80.4%440 of 547 right84.66chosen-answer Brier 0.111175.7%414 of 547 right3.7%4 “I don't know” of 108 not rightanswered 99.3%error rate 19.2%0.02727.21from GPU time0.0381 question per request, 1 in flight
Qwen3.5-4B reasoning off · chat Open · 4.0B80.4%440 of 547 rightnot applicable72.0%394 of 547 right3.6%4 “I don't know” of 110 not rightanswered 99.3%error rate 19.5%0.309309from GPU time9.741 question per request, 32 in flight
Qwen3-14B reasoning off · chat Open · 15B79.7%436 of 547 rightnot applicable76.8%420 of 547 right1.8%2 “I don't know” of 110 not rightanswered 99.6%error rate 19.8%0.03131.17at a host's list price for the same weights2.8616 questions per request, 1 in flight
MiniCPM5-2B reasoning on · chat Open · 2.5B78.1%76.6%–79.0% min–max of 3 seedsseed 0: run 1051, 419 of 547 right; seed 1: run 1118, 431 of 547 right; seed 2: run 1119, 432 of 547 right. Baseline server capacity not retained; repeats use max-num-seqs 16.not applicableNot runEnglish only, no Text tab run19.7%28 “I don't know” of 142 not rightanswered 94.1%error rate 22.1%from run 10520.714714from GPU time17.51 question per request, 32 in flight
Metask-Jev-4B Open · 4.5B77.1%422 of 547 right82.21chosen-answer Brier 0.127367.1%367 of 547 right1.6%2 “I don't know” of 125 not rightanswered 99.6%error rate 22.6%0.05959.11from GPU time0.0851 question per request, 1 in flight
Gemma 4 E2B reasoning on · chat Open · 5.1B76.2%417 of 547 rightnot applicableNot runEnglish only, no Text tab run11.3%17 “I don't know” of 150 not rightanswered 96.3%error rate 25.2%0.187187from GPU time7.611 question per request, 32 in flight
Qwen3.5-2B reasoning on · chat Open · 2.0B75.9%415 of 547 rightnot applicableNot runEnglish only, no Text tab run9.9%14 “I don't know” of 142 not rightanswered 97.1%error rate 24.1%1.3611,361from GPU time54.41 question per request, 32 in flight
Gemma 4 E2B default reasoning · chat Open · 5.1B74.6%408 of 547 rightnot applicableQueuedits Text tab run in logic's GPU 1 queue (Ümit #6527, logic #6955)11.5%17 “I don't know” of 148 not rightanswered 96.5%error rate 24.8%0.115115from GPU time4.841 question per request, 32 in flight
Granite-4.2-3B reasoning on · chat Open · 3.0B74.4%407 of 547 rightnot applicablePausedits Text tab run its Text run stopped at 741 of 2,198 so that other setups go first on logic's GPU 1; it resumes when logic loads Granite again (logic #7481, #7599)16.1%23 “I don't know” of 143 not rightanswered 95.8%error rate 22.9%2.7172,717from GPU time40.51 question per request, 32 in flight
LFM2.5-2.6B default reasoning · chat Open · 2.7B73.9%404 of 547 rightnot applicable47.9%262 of 547 right15.7%21 “I don't know” of 134 not rightanswered 95.6%error rate 21.6%0.964964at the maker's list price15.01 question per request, 32 in flight
Erk-14B reasoning off · chat Open · 14B72.0%394 of 547 rightnot applicable75.3%412 of 547 right2.1%3 “I don't know” of 144 not rightanswered 99.5%error rate 25.9%0.123123from GPU time2.8616 questions per request, 1 in flight
LFM3-1.7B reasoning off · chat Open · 1.7B71.8%70.9%–72.9% min–max of 3 seedsseed 0: run 1223, 399 of 547 right; seed 1: run 1302, 388 of 547 right; seed 2: run 1303, 391 of 547 rightnot applicable40.2%220 of 547 right11.6%18 “I don't know” of 155 not rightanswered 96.7%error rate 25.9%from run 12250.355355at the maker's list price4.691 question per request, 32 in flight
KnowLine-2B-Gen1 Open · 2.0B71.3%390 of 547 right77.82chosen-answer Brier 0.156758.0%317 of 547 right22.0%38 “I don't know” of 173 not rightanswered 93.1%error rate 26.5%No list price0.0621 question per request, 1 in flight
LFM3-1.7B reasoning on · chat Open · 1.7B71.1%70.4%–71.7% min–max of 3 seedsseed 0: run 1091, 392 of 547 right; seed 1: run 1115, 390 of 547 right; seed 2: run 1116, 385 of 547 rightnot applicable48.1%263 of 547 right15.6%26 “I don't know” of 167 not rightanswered 95.2%error rate 27.1%from run 10620.387387at the maker's list price5.241 question per request, 32 in flight
LFM2.5-8B-A1B default reasoning · chat Open · 8.3B MoE · 1.5B active70.4%385 of 547 rightnot applicable27.1%148 of 547 right26.4%51 “I don't know” of 193 not rightanswered 90.5%error rate 28.7%0.517517from GPU time19.41 question per request, 32 in flight
GLiDE 2B Open · 1.9B70.0%383 of 547 right76.88chosen-answer Brier 0.162654.3%297 of 547 right26.3%47 “I don't know” of 179 not rightanswered 91.4%error rate 26.4%No list price0.0351 question per request, 1 in flight
StartLux-Decision-v1.1-2B-Instant Open · 2.0B69.7%381 of 547 right76.66chosen-answer Brier 0.163355.8%305 of 547 right16.9%32 “I don't know” of 189 not rightanswered 94.1%error rate 30.5%No list price0.0401 question per request, 1 in flight
MiniCPM5-2B reasoning off · chat Open · 2.5B67.8%66.7%–68.9% min–max of 3 seedsseed 0: run 1094, 370 of 547 right; seed 1: run 1300, 377 of 547 right; seed 2: run 1301, 365 of 547 right. Seeds 1 and 2 on logic's load of 15 November (vLLM v0.30.0, max-num-seqs 16), which logic gives as run 1094's setup (#9335); the server's settings are not recorded with a run.not applicable7.1%39 of 547 right16.8%33 “I don't know” of 197 not rightanswered 89.2%error rate 33.6%from run 10950.429429from GPU time7.291 question per request, 32 in flight
Qwen3.5-2B reasoning off · chat Open · 2.0B67.3%368 of 547 rightnot applicable36.7%201 of 547 right10.3%18 “I don't know” of 174 not rightanswered 96.3%error rate 29.6%0.280280from GPU time9.271 question per request, 32 in flight
d1-2B Open · 1.9B65.6%359 of 547 right74.16chosen-answer Brier 0.173141.9%229 of 547 right60.9%151 “I don't know” of 248 not rightanswered 72.4%error rate 24.5%0.0088.08from GPU time0.0111 question per request, 1 in flight
GLiDE 0.8B Open · 0.8B60.7%332 of 547 right70.59chosen-answer Brier 0.195146.1%252 of 547 right21.7%49 “I don't know” of 226 not rightanswered 91.0%error rate 35.5%No list price0.0231 question per request, 1 in flight
StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B59.6%326 of 547 right70.09chosen-answer Brier 0.194237.3%204 of 547 right5.2%12 “I don't know” of 229 not rightanswered 97.8%error rate 40.6%No list price0.0281 question per request, 1 in flight
GLiDE 0.4B Open · 0.4B59.4%325 of 547 right69.84chosen-answer Brier 0.197325.8%141 of 547 right53.1%153 “I don't know” of 288 not rightanswered 72.0%error rate 34.3%No list price0.0141 question per request, 1 in flight
OneJev1.6-0.8B Open · 0.8B59.4%325 of 547 right69.82chosen-answer Brier 0.197842.8%234 of 547 right28.4%74 “I don't know” of 261 not rightanswered 86.5%error rate 39.5%No list price0.0181 question per request, 1 in flight
KnowLine-0.8B-Gen1 Open · 0.8B58.9%322 of 547 right69.65chosen-answer Brier 0.195640.2%220 of 547 right10.7%26 “I don't know” of 243 not rightanswered 95.2%error rate 41.7%No list price0.0331 question per request, 1 in flight
GLiClass v4 Edge Open · 0.4B58.3%319 of 547 right69.05chosen-answer Brier 0.202126.9%147 of 547 right21.0%52 “I don't know” of 248 not rightanswered 90.5%error rate 39.6%No list price0.0171 question per request, 1 in flight
Decider 1 Closed · Efficient55.8%305 of 547 right67.47chosen-answer Brier 0.208236.6%200 of 547 rightNot run0.01514.95at list prices0.6991 question per request, 1 in flight
Raon-1 nano reasoning off · chat Open · 3.1B53.7%294 of 547 rightnot applicableQueuedits Text tab run asked on its next load in logic's GPU 1 queue, whichever way its smoke came out (run 1154: 3 of 20 replies at the 32,768-token limit, 5 without a letter; Ümit #6515 via Zeki #8211)26.2%71 “I don't know” of 271 not rightanswered 86.8%error rate 42.1%0.235235from GPU time3.741 question per request, 32 in flight
Strands Decider 2B Open · 2.0B45.5%249 of 547 right64.08chosen-answer Brier 0.173639.1%214 of 547 right0.7%2 “I don't know” of 303 not rightanswered 99.6%error rate 55.2%0.01818.15from GPU time0.0261 question per request, 1 in flight
Laya Open · 0.4B22.9%125 of 547 right51.57chosen-answer Brier 0.197120.5%112 of 547 right18.4%83 “I don't know” of 450 not rightanswered 84.8%error rate 79.1%0.0087.84from GPU time0.0111 question per request, 1 in flight
GLiNER2.5-multi-Decide Open · 0.3B21.2%116 of 547 right47.58chosen-answer Brier 0.260420.7%113 of 547 right16.2%70 “I don't know” of 433 not rightanswered 87.2%error rate 76.1%0.0076.73from GPU time0.0091 question per request, 1 in flight
GLiNER2.5 Multi Open · 0.3B16.3%89 of 547 rightnot applicable16.8%92 of 547 right19.6%92 “I don't know” of 470 not rightanswered 83.2%error rate 83.1%0.0088.39at the maker's list price0.0091 question per request, 1 in flight
Laya Multilingual Open · 0.3B15.4%84 of 547 right39.32chosen-answer Brier 0.367117.0%93 of 547 right42.6%208 “I don't know” of 488 not rightanswered 62.0%error rate 82.6%0.01312.62from GPU time0.0181 question per request, 1 in flight

Ranked by English accuracy, highest first: right answers of the 547 kept reading questions in each setup's first English run (an invalid answer counts as not right). Turkish accuracy: the setup's Text tab run on the same 547 questions, for context (Not run: an English-only setup, with no Text tab run; Paused: its Text tab run is stopped for now, as its row says; Queued: its Text tab run has not started yet, as its row says); the paired comparison is on English and Turkish. English non-hallucination: “I don't know” among the answers that were not right, in the setup's first English run with it offered as a sixth option; Running: a chat model's run still running; Not run: a decision model not run with it yet, or an English-only setup. English cost: $ per 1,000 questionsper 1M questions, with its basis under it; English latency: the median time per request, with the request size and the requests in flight under it, each from the first English run (Cost and Latency below). No list price: a setup on our own hardware whose maker lists no price for the same model. MiniCPM5-2B, LFM3-1.7B, LFM3-1.7B and MiniCPM5-2B are run with 3 sampling seeds: until all are in, the number is the first English run's, named with the seeds in so far; then it is the mean of the runs' accuracies, ranked by it, with their lowest and highest (min–max, a range, not a confidence interval) under it. The paired comparison with Turkish and the “I don't know” runs stay on their own runs. Each name opens its model page's English section (an English-only setup has no model page: its name opens its row here); its tooltip gives its settings and English run.

Decision Score (AA formula): 50 × (1 + accuracy − chosen-answer Brier) on the first English run, the chosen-answer Brier being the mean of (p − c)², p the probability given to the chosen answer and c 1 if it is right, else 0. GPT-6 Luna · Decisions API: unavailable, 2 of 547 answers without a probability for the chosen answer.

Artificial Analysis's formula applied to our questions. Not their Index. Decision models only: chat models and the I don't know runs are excluded.

Costs: API list prices and recorded token use. Setups on our own GPUs: the maker's list price for the same model (each value's tooltip names it); our hardware cost is not measured. On our GPUs, estimated at a public host's list price for the same weights (the host, the date read and any precision difference are on each setup's page): an API-equivalent estimate, not measured spending. On our GPUs with no public price for the same weights, estimated from GPU time: the run's request time x the GPUs its server held x Runpod's on-demand price for an RTX PRO 6000, $2.49 per GPU-hour (read 5 November 2026): a cloud-equivalent estimate, not measured spending. No list price: on our own GPUs, with no estimate. ≤ Upper-bound estimate: one $0.001 web search counted per request; Perplexity bills only completed searches.

Accuracy in English · 547 machine-translated questions

Share of the 547 kept reading questions answered right in English, then the right answers of 547: each of the 142 setups' first English run, or the mean of a setup's three seed runs · higher is better

  1. Claude Fable 5.2 max · chat98.0% · 536
  2. Gemini 4 Hydrogen high · chat96.7% · 529
  3. GPT-6 Astra low · chat96.3% · 527
  4. GPT-6 Astra high · chat96.2% · 526
  5. GPT-6.1 Terra high · chat96.2% · 526
  6. GPT-6.1 Terra low · chat96.2% · 526
  7. Claude Fable 5.1 high · chat96.0% · 525
  8. Gemini 3.8 Flash high · chat96.0% · 525
  9. Gemini 4 Argon low · chat96.0% · 525
  10. GPT-6 Astra medium · chat96.0% · 525
  11. GPT-6.1 Sol low · chat95.8% · 524
  12. GPT-6 Astra low · chat · 1 per request95.6% · 523
  13. GPT-6.1 Sol high · chat95.6% · 523
  14. Gemini 3.8 Flash low · chat · 1 per request95.2% · 521
  15. GPT-6.1 Sol low · chat · 1 per request95.2% · 521
  16. Claude Opus 5.5 low · chat · 1 per request94.7% · 518
  17. Claude Opus 5.5 high · chat94.5% · 517
  18. Qwen4-Max high · chat94.5% · 517
  19. Gemini 3.8 Flash low · chat94.3% · 516
  20. Gemini 4 Carbon high · chat94.3% · 516
  21. KnowLine-31B-Gen1.593.6% · 512
  22. Grok 4.7 high · chat93.4% · 511
  23. Claude Opus 5.5 low · chat93.1% · 509
  24. Copernicus-31B92.9% · 508
  25. Claude Haiku 5.5 max · chat92.7% · 507
  26. DeepSeek-V4.1-Flash reasoning on · chat · 101792.5% · 506
  27. REAX 1.392.5% · 506
  28. Kimi K3.1 Pro low · chat92.3% · 505
  29. KnowLine-31B-Gen292.3% · 505
  30. Karpo Decision 31B92.1% · 504
  31. Claude Sonnet 5.5 high · chat · 1 per request91.6% · 501
  32. Kimi K3.1 Pro no reasoning · chat91.4% · 500
  33. Qwen3.5-397B-A17B high · chat91.4% · 500
  34. Gemini Decision 1 2026111191.0% · 498
  35. Logits Sigma-1 Duo91.0% · 498
  36. GLiDE 27B90.9% · 497
  37. d1-ultra90.7% · 496
  38. Sieve 31B v390.7% · 496
  39. Turion Jev 31B90.7% · 496
  40. Gemma 4 31B reasoning off · chat90.5% · 495
  41. Perplexity Decider 27B v1.2 API · with web90.5% · 495
  42. GLiDE v1.290.3% · 494
  43. d1.2-ultra90.3% · 494
  44. Mammoth-190.1% · 493
  45. Gemma 4 31B reasoning off · chat · 1 per request89.9% · 492
  46. d389.8% · 491
  47. Drex Max89.6% · 490
  48. GLM-5.3-Flash reasoning on · chat89.6% · 490
  49. Muse Ember max · chat89.6% · 490
  50. Perplexity Decider 27B v1.389.6% · 490
  51. d3-ultra89.4% · 489
  52. d1.289.0% · 487
  53. MiMo-V2.6-Flash high · chat89.0% · 487
  54. Qwen3.8-27B low · chat89.0% · 487
  55. Perplexity Decider 27B v1.2 DGX Spark88.7% · 485
  56. Drex Max v1.188.5% · 484
  57. Perplexity Decider 27B v188.5% · 484
  58. Perplexity Decider 27B v1.2 API88.5% · 484
  59. Inkling-Small high · chat · Jul 202688.3% · 483
  60. d1.188.3% · 483
  61. GLM-5.3 low · chat87.8% · 480
  62. Jeffan-9B-121187.6% · 479
  63. GLM-5.3-Flash reasoning off · chat87.2% · 477
  64. DeepSeek-V4.1-Flash reasoning off · chat · 101787.0% · 476
  65. Qwen3.5-4B reasoning on · chat87.0% · 476
  66. Qwen3.8 Flash no reasoning · chat87.0% · 476
  67. d186.8% · 475
  68. StartLux-Decision-v1.1-9B-Instant86.8% · 475
  69. Gemma 4 12B Unified default reasoning · chat86.8% · 475
  70. Clef86.7% · 474
  71. KnowLine-12B-Gen186.7% · 474
  72. GLiDE 9B86.3% · 472
  73. GPT-6 Luna low · chat86.3% · 472
  74. OneJev-27B86.3% · 472
  75. Claude Sonnet 5.5 high · chat86.1% · 471
  76. Jev 1.1486.1% · 471
  77. Cerno 9B v1.285.9% · 470
  78. Drex 2.085.9% · 470
  79. Jev 1.1385.9% · 470
  80. KnowLine-9B-Gen185.9% · 470
  81. SubQ Snap 1.185.9% · 470
  82. SubQ Snap Preview85.9% · 470
  83. Adjudex 9B v585.7% · 469
  84. Mercury Decide85.7% · 469
  85. Microsoft-Decision-185.7% · 469
  86. Elephant-185.4% · 467
  87. Open-Jev-27B-v1.185.4% · 467
  88. KnowLine-9B-Gen285.2% · 466
  89. Qwen3.8-27B stated confidence · reasoning off · chat84.8% · 464
  90. Qwen3.8-27B yes/no scoring84.8% · 464
  91. MiniMax-M3.1-Flash high · chat84.5% · 462
  92. celeris-1-decision84.1% · 460
  93. GPT-6 Luna Decisions API84.1% · 460
  94. Claude Sonnet 5.5 low · chat83.9% · 459
  95. Cygnet83.9% · 459
  96. Qwen3.8-27B vote share · reasoning off · chat83.9% · 459
  97. Winnow-12B83.9% · 459
  98. Perplexity Decider Flash v1.383.7% · 458
  99. Valen-Decision-1.5-9B83.7% · 458
  100. celeris-1.1-decision83.7% · 458
  101. Dexy-Decision-9B-V283.5% · 457
  102. vera-medium-omni83.5% · 457
  103. StartLux-Decision-v1.1-4B-Instant82.8% · 453
  104. LFM2.5-24B-A2B default reasoning · chat82.6% · 452
  105. Gemma 4 E4B default reasoning · chat82.4% · 451
  106. Clef-Flash82.3% · 450
  107. Verdin-4B-v1.182.3% · 450
  108. Perplexity Decider Flash v1.281.9% · 448
  109. KnowLine 4B Gen5.580.8% · 442
  110. Chinese Decision Pointer 4B80.4% · 440
  111. Qwen3.5-4B reasoning off · chat80.4% · 440
  112. Qwen3-14B reasoning off · chat79.7% · 436
  113. MiniCPM5-2B reasoning on · chat78.1% · mean of 3
  114. Metask-Jev-4B77.1% · 422
  115. Gemma 4 E2B reasoning on · chat76.2% · 417
  116. Qwen3.5-2B reasoning on · chat75.9% · 415
  117. Gemma 4 E2B default reasoning · chat74.6% · 408
  118. Granite-4.2-3B reasoning on · chat74.4% · 407
  119. LFM2.5-2.6B default reasoning · chat73.9% · 404
  120. Erk-14B reasoning off · chat72.0% · 394
  121. LFM3-1.7B reasoning off · chat71.8% · mean of 3
  122. KnowLine-2B-Gen171.3% · 390
  123. LFM3-1.7B reasoning on · chat71.1% · mean of 3
  124. LFM2.5-8B-A1B default reasoning · chat70.4% · 385
  125. GLiDE 2B70.0% · 383
  126. StartLux-Decision-v1.1-2B-Instant69.7% · 381
  127. MiniCPM5-2B reasoning off · chat67.8% · mean of 3
  128. Qwen3.5-2B reasoning off · chat67.3% · 368
  129. d1-2B65.6% · 359
  130. GLiDE 0.8B60.7% · 332
  131. StartLux-Decision-v1.1-0.8B-Instant59.6% · 326
  132. GLiDE 0.4B59.4% · 325
  133. OneJev1.6-0.8B59.4% · 325
  134. KnowLine-0.8B-Gen158.9% · 322
  135. GLiClass v4 Edge58.3% · 319
  136. Decider 155.8% · 305
  137. Raon-1 nano reasoning off · chat53.7% · 294
  138. Strands Decider 2B45.5% · 249
  139. Laya22.9% · 125
  140. GLiNER2.5-multi-Decide21.2% · 116
  141. GLiNER2.5 Multi16.3% · 89
  142. Laya Multilingual15.4% · 84

Claude Gemini GPT Qwen KnowLine Grok Copernicus DeepSeek REAX Kimi Karpo Logits Fastino Liquid Sieve Turion Jev Gemma Perplexity Mammoth vLLM-SR Drex GLM Muse MiMo Inkling Jeffan StartLux Clef OneJev Jev Cerno SubQ Adjudex Inception Microsoft Elephant Open-Jev MiniMax Celeris Cygnet Winnow Valen Dexy vera Verdin Chinese Decision Pointer MiniCPM Metask Granite Erk GLiClass meraGPT Raon Strands Laya

Non-hallucination

Holding back by confidence

Confidence at the threshold is not measured in English.

Decision models

Not measured in English.

Chat models

Not measured in English.

Holding back with an “I don't know” option in English

“I don't know” among the answers that were not right, with it offered as a sixth option in English, on the 547 kept questions; beside it the share answered (right + wrong of 547) and the error rate among the answers (wrong of right + wrong) · higher non-hallucination is better, read with the share answered; a model that always chose the option would reach 100% with no right answers

Decision models

Each of the 70 decision models' first English run with it.

  1. d1.2-ultra89.3%80.1%3.0%
  2. celeris-1.1-decision74.8%82.1%7.3%
  3. celeris-1-decision71.3%83.2%8.1%
  4. d1.261.5%89.2%7.6%
  5. d1-2B60.9%72.4%24.5%
  6. d1-ultra56.7%90.7%7.9%
  7. GLiDE 0.4B53.1%72.0%34.3%
  8. Drex Max v1.151.2%92.1%8.1%
  9. Mammoth-150.7%93.2%7.1%
  10. GLiDE v1.245.5%94.5%7.0%
  11. Laya Multilingual42.6%62.0%82.6%
  12. GLiDE 27B37.9%96.0%6.9%
  13. OneJev1.6-0.8B28.4%86.5%39.5%
  14. GLiDE 2B26.3%91.4%26.4%
  15. Elephant-125.6%95.8%12.8%
  16. SubQ Snap 1.125.0%96.2%12.0%
  17. KnowLine-2B-Gen122.0%93.1%26.5%
  18. GLiDE 0.8B21.7%91.0%35.5%
  19. GLiClass v4 Edge21.0%90.5%39.6%
  20. Laya18.4%84.8%79.1%
  21. Open-Jev-27B-v1.117.4%97.1%14.3%
  22. StartLux-Decision-v1.1-2B-Instant16.9%94.1%30.5%
  23. GLiDE 9B16.7%97.4%13.1%
  24. GLiNER2.5-multi-Decide16.2%87.2%76.1%
  25. Perplexity Decider 27B v1.2 API · with web15.2%98.2%10.4%
  26. Winnow-12B15.0%97.3%16.0%
  27. Mercury Decide13.4%98.0%13.2%
  28. Cygnet11.2%98.2%14.7%
  29. KnowLine-0.8B-Gen110.7%95.2%41.7%
  30. GPT-6 Luna Decisions API9.4%98.2%17.9%
  31. Gemini Decision 1 202611119.1%99.1%9.2%
  32. Perplexity Decider 27B v1.38.8%99.1%9.6%
  33. Turion Jev 31B8.5%99.3%7.9%
  34. d3-ultra8.2%99.1%10.3%
  35. vera-medium-omni7.4%98.7%16.1%
  36. Perplexity Decider Flash v1.37.2%98.7%16.7%
  37. REAX 1.36.7%99.5%7.7%
  38. Clef-Flash5.3%99.1%16.6%
  39. StartLux-Decision-v1.1-0.8B-Instant5.2%97.8%40.6%
  40. Perplexity Decider Flash v1.24.8%99.1%18.5%
  41. Perplexity Decider 27B v1.2 API4.5%99.5%11.6%
  42. Logits Sigma-1 Duo4.3%99.6%8.1%
  43. Jev 1.144.0%99.5%13.2%
  44. OneJev-27B4.0%99.5%13.2%
  45. Microsoft-Decision-13.8%99.5%14.2%
  46. Chinese Decision Pointer 4B3.7%99.3%19.2%
  47. Drex Max3.2%99.6%11.0%
  48. Perplexity Decider 27B v1.2 DGX Spark3.0%99.6%11.7%
  49. d12.9%99.6%12.3%
  50. StartLux-Decision-v1.1-9B-Instant2.6%99.6%13.6%
  51. Cerno 9B v1.22.5%99.6%14.1%
  52. Sieve 31B v32.2%99.8%8.2%
  53. Verdin-4B-v1.12.1%99.6%17.2%
  54. d31.7%99.8%10.6%
  55. Metask-Jev-4B1.6%99.6%22.6%
  56. Clef1.3%99.8%13.7%
  57. KnowLine-12B-Gen11.3%99.8%13.9%
  58. KnowLine-9B-Gen11.3%99.8%13.9%
  59. KnowLine-9B-Gen21.3%99.8%14.3%
  60. Valen-Decision-1.5-9B1.2%99.8%15.4%
  61. Dexy-Decision-9B-V21.1%99.8%16.5%
  62. KnowLine 4B Gen5.50.9%99.8%19.2%
  63. Strands Decider 2B0.7%99.6%55.2%
  64. Adjudex 9B v50.0%100.0%14.1%
  65. Copernicus-31B0.0%100.0%7.1%
  66. Jeffan-9B-12110.0%100.0%13.0%
  67. Karpo Decision 31B0.0%100.0%7.7%
  68. KnowLine-31B-Gen1.50.0%100.0%6.8%
  69. KnowLine-31B-Gen20.0%100.0%7.3%
  70. StartLux-Decision-v1.1-4B-Instant0.0%100.0%17.2%

Chose “I don't know” on fewer than 1% of the 547 questions: KnowLine-31B-Gen1.5, 0 times; Copernicus-31B, 0 times; REAX 1.3, 3 times; KnowLine-31B-Gen2, 0 times; Karpo Decision 31B, 0 times; Gemini Decision 1 · 20261111, 5 times; Logits Sigma-1 Duo, 2 times; Sieve 31B v3, 1 time; Turion Jev 31B, 4 times; d3, 1 time; Drex Max, 2 times; Perplexity Decider 27B v1.3, 5 times; d3-ultra, 5 times; Perplexity Decider 27B v1.2 · DGX Spark, 2 times; Perplexity Decider 27B v1.2 · API, 3 times; Jeffan-9B-1211, 0 times; d1, 2 times; StartLux-Decision-v1.1-9B-Instant, 2 times; Clef, 1 time; KnowLine-12B-Gen1, 1 time; OneJev-27B, 3 times; Jev 1.14, 3 times; Cerno 9B v1.2, 2 times; KnowLine-9B-Gen1, 1 time; Adjudex 9B v5, 0 times; Microsoft-Decision-1, 3 times; KnowLine-9B-Gen2, 1 time; Valen-Decision-1.5-9B, 1 time; Dexy-Decision-9B-V2, 1 time; StartLux-Decision-v1.1-4B-Instant, 0 times; Clef-Flash, 5 times; Verdin-4B-v1.1, 2 times; Perplexity Decider Flash v1.2, 5 times; KnowLine 4B Gen5.5, 1 time; Chinese Decision Pointer 4B, 4 times; Metask-Jev-4B, 2 times; Strands Decider 2B, 2 times. Their rate here is low (9.1% or less) because they almost never chose the option; confidence is not measured in English; the Turkish reference below shows their Text tab runs at the threshold.

Liquid Celeris Fastino Drex Mammoth Laya OneJev Elephant SubQ KnowLine GLiClass Open-Jev StartLux Perplexity Winnow Inception Cygnet GPT Gemini Turion Jev vLLM-SR vera REAX Clef Logits Jev Microsoft Chinese Decision Pointer Cerno Sieve Verdin Metask Valen Dexy Strands Adjudex Copernicus Jeffan Karpo Decision models not run with “I don't know” in English yet: Perplexity Decider 27B v1, d1.1, Drex 2.0, Jev 1.13, SubQ Snap Preview, Decider 1.

Chat models

Each of the 65 chat model setups' first English run with it.

  1. Claude Fable 5.2 max · chat28.6%99.3%1.8%
  2. Grok 4.7 high · chat27.9%97.8%5.8%
  3. LFM2.5-8B-A1B default reasoning · chat26.4%90.5%28.7%
  4. Raon-1 nano reasoning off · chat26.2%86.8%42.1%
  5. Gemini 4 Carbon high · chat25.0%99.3%2.2%
  6. Gemini 4 Hydrogen high · chat20.0%99.3%2.9%
  7. MiniCPM5-2B reasoning on · chat19.7%94.1%22.1%
  8. GLiNER2.5 Multi19.6%83.2%83.1%
  9. GPT-6 Astra high · chat19.0%99.3%3.1%
  10. Gemini 4 Argon low · chat18.5%99.1%4.1%
  11. GPT-6 Astra low · chat18.2%99.3%3.3%
  12. MiniCPM5-2B reasoning off · chat16.8%89.2%33.6%
  13. GPT-6 Astra medium · chat16.7%99.3%3.7%
  14. GPT-6.1 Sol high · chat16.7%99.3%3.7%
  15. GPT-6.1 Terra high · chat16.7%99.3%3.7%
  16. Granite-4.2-3B reasoning on · chat16.1%95.8%22.9%
  17. LFM2.5-2.6B default reasoning · chat15.7%95.6%21.6%
  18. LFM3-1.7B reasoning on · chat15.6%95.2%27.1%
  19. GPT-6.1 Terra low · chat15.4%99.3%4.1%
  20. GPT-6 Astra low · chat · 1 per request14.8%99.3%4.2%
  21. GPT-6.1 Sol low · chat14.8%99.3%4.2%
  22. LFM2.5-24B-A2B default reasoning · chat13.2%97.4%17.3%
  23. GPT-6.1 Sol low · chat · 1 per request12.9%99.3%5.0%
  24. LFM3-1.7B reasoning off · chat11.6%96.7%25.9%
  25. Gemma 4 E2B default reasoning · chat11.5%96.5%24.8%
  26. Gemma 4 E2B reasoning on · chat11.3%96.3%25.2%
  27. Qwen3.5-2B reasoning off · chat10.3%96.3%29.6%
  28. Gemini 3.8 Flash high · chat10.0%99.6%3.3%
  29. Qwen3.5-2B reasoning on · chat9.9%97.1%24.1%
  30. Gemma 4 E4B default reasoning · chat9.8%97.8%17.2%
  31. Claude Haiku 5.5 max · chat7.8%96.3%8.9%
  32. Gemini 3.8 Flash low · chat · 1 per request7.7%99.6%4.4%
  33. Kimi K3.1 Pro no reasoning · chat7.5%99.3%9.0%
  34. Claude Fable 5.1 high · chat7.4%99.6%4.6%
  35. Kimi K3.1 Pro low · chat7.3%99.5%7.0%
  36. DeepSeek-V4.1-Flash reasoning on · chat · 10177.1%99.5%7.2%
  37. GPT-6 Luna low · chat6.8%99.1%12.7%
  38. Claude Opus 5.5 high · chat6.5%99.6%5.3%
  39. Qwen3.5-397B-A17B high · chat6.2%99.5%8.3%
  40. GLM-5.3-Flash reasoning off · chat5.6%99.3%12.3%
  41. Qwen3.8 Flash no reasoning · chat5.6%99.3%12.3%
  42. Gemma 4 12B Unified default reasoning · chat5.5%98.9%12.8%
  43. DeepSeek-V4.1-Flash reasoning off · chat · 10175.4%99.3%12.9%
  44. Muse Ember max · chat5.3%99.5%9.9%
  45. Claude Sonnet 5.5 high · chat4.8%99.5%10.8%
  46. Claude Opus 5.5 low · chat4.7%99.6%7.5%
  47. Claude Sonnet 5.5 low · chat4.4%99.3%15.8%
  48. Qwen3.8-27B stated confidence · reasoning off · chat3.7%99.5%14.3%
  49. Gemma 4 31B reasoning off · chat3.6%99.6%9.7%
  50. Qwen3.5-4B reasoning off · chat3.6%99.3%19.5%
  51. Qwen3.8-27B low · chat3.6%99.6%9.7%
  52. Gemini 3.8 Flash low · chat3.4%99.8%5.1%
  53. GLM-5.3 low · chat3.4%99.6%10.3%
  54. Qwen3.8-27B vote share · reasoning off · chat3.4%99.5%15.4%
  55. MiniMax-M3.1-Flash high · chat3.4%99.5%15.6%
  56. Qwen3.5-4B reasoning on · chat2.4%99.6%15.2%
  57. Erk-14B reasoning off · chat2.1%99.5%25.9%
  58. Qwen3-14B reasoning off · chat1.8%99.6%19.8%
  59. MiMo-V2.6-Flash high · chat1.7%99.8%10.6%
  60. GLM-5.3-Flash reasoning on · chat1.6%99.8%11.2%
  61. Claude Opus 5.5 low · chat · 1 per request0.0%100.0%5.7%
  62. Claude Sonnet 5.5 high · chat · 1 per request0.0%100.0%6.9%
  63. Gemma 4 31B reasoning off · chat · 1 per request0.0%100.0%10.8%
  64. Inkling-Small high · chat · Jul 20260.0%100.0%11.0%
  65. Qwen4-Max high · chat0.0%100.0%5.3%

Claude Grok Liquid Raon Gemini MiniCPM Fastino GPT Granite Gemma Qwen Kimi DeepSeek GLM Muse MiniMax Erk MiMo Inkling Not run, as yes/no scoring cannot offer it: Qwen3.8-27B · yes/no scoring.

Decision Score (AA formula), decision models

ModelDecision Score (AA formula)AccuracyChosen-answer Brier
KnowLine-31B-Gen1.5 Open · 31B94.3393.60%0.0494
Copernicus-31B Open · 31B93.6692.87%0.0556
REAX 1.3 Open · 57B93.7592.50%0.0500
KnowLine-31B-Gen2 Open · 31B93.3792.32%0.0558
Karpo Decision 31B Open · 31B93.2392.14%0.0568
Gemini Decision 1 20261111 Closed · Efficient92.3191.04%0.0642
Logits Sigma-1 Duo Open · 57B92.5591.04%0.0595
GLiDE 27B Open · 27B92.3690.86%0.0613
d1-ultra Closed · Efficient91.9290.68%0.0684
Sieve 31B v3 Open · 31B92.5190.68%0.0565
Turion Jev 31B Open · 31B91.6990.68%0.0729
Perplexity Decider 27B v1.2 API · with web Open · 27B91.3990.49%0.0771
GLiDE v1.2 Open · 27B91.9690.31%0.0639
d1.2-ultra Closed · Efficient91.8490.31%0.0664
Mammoth-1 Open · 27B91.7590.13%0.0663
d3 Open · 26B91.6589.76%0.0646
Drex Max Open · 27B91.5089.58%0.0658
Perplexity Decider 27B v1.3 Open · 27B91.2289.58%0.0714
d3-ultra Open · 26B91.1889.40%0.0704
d1.2 Closed · Efficient90.5189.03%0.0801
Perplexity Decider 27B v1.2 DGX Spark Open · 27B90.7588.67%0.0717
Drex Max v1.1 Closed · Efficient (provisional)90.8188.48%0.0685
Perplexity Decider 27B v1 Open · 27B90.6488.48%0.0720
Perplexity Decider 27B v1.2 API Open · 27B90.6488.48%0.0721
d1.1 Closed · Efficient90.1488.30%0.0802
Jeffan-9B-1211 Open · 9.0B89.0587.57%0.0946
d1 Closed · Efficient89.2686.84%0.0832
StartLux-Decision-v1.1-9B-Instant Open · 9.0B88.5986.84%0.0965
Clef Open · 27B89.1986.65%0.0828
KnowLine-12B-Gen1 Open · 12B88.6686.65%0.0933
GLiDE 9B Open · 7.9B88.6086.29%0.0909
OneJev-27B Open · 27B88.5186.29%0.0927
Jev 1.14 Closed · Efficient89.1686.11%0.0778
Cerno 9B v1.2 Open · 9.0B88.4285.92%0.0907
Drex 2.0 Open · 35B MoE · 3.0B active88.1585.92%0.0963
Jev 1.13 Closed · Efficient88.8685.92%0.0821
KnowLine-9B-Gen1 Open · 9.0B88.3285.92%0.0928
SubQ Snap 1.1 Closed · Efficient88.7085.92%0.0852
SubQ Snap Preview Closed · Efficient88.7085.92%0.0852
Adjudex 9B v5 Open · 9.0B88.3585.74%0.0905
Mercury Decide Closed · Efficient88.1585.74%0.0943
Microsoft-Decision-1 Closed · Efficient88.1085.74%0.0954
Elephant-1 Open · 9.0B87.9985.37%0.0939
Open-Jev-27B-v1.1 Open · 27B77.1885.37%0.3102
KnowLine-9B-Gen2 Open · 9.0B87.3885.19%0.1044
celeris-1-decision Closed · Efficient87.2984.10%0.0952
GPT-6 Luna Decisions API Closed · Efficientunavailableunavailableunavailable
Cygnet Open · 12B87.0383.91%0.0984
Winnow-12B Open · 12B86.1883.91%0.1156
Perplexity Decider Flash v1.3 Open · 4.0B86.9683.73%0.0980
Valen-Decision-1.5-9B Open · 9.0B86.4383.73%0.1087
celeris-1.1-decision Closed · Efficient87.1483.73%0.0946
Dexy-Decision-9B-V2 Open · 9.0B87.0183.55%0.0953
vera-medium-omni Open · 4.2B86.3483.55%0.1086
StartLux-Decision-v1.1-4B-Instant Open · 4.0B86.3582.82%0.1011
Clef-Flash Open · 9.0B85.4382.27%0.1141
Verdin-4B-v1.1 Open · 4.0B85.9782.27%0.1032
Perplexity Decider Flash v1.2 Open · 4.0B86.0481.90%0.0982
KnowLine 4B Gen5.5 Open · 4.0B84.5780.80%0.1167
Chinese Decision Pointer 4B Open · 4.0B84.6680.44%0.1111
Metask-Jev-4B Open · 4.5B82.2177.15%0.1273
KnowLine-2B-Gen1 Open · 2.0B77.8271.30%0.1567
GLiDE 2B Open · 1.9B76.8870.02%0.1626
StartLux-Decision-v1.1-2B-Instant Open · 2.0B76.6669.65%0.1633
d1-2B Open · 1.9B74.1665.63%0.1731
GLiDE 0.8B Open · 0.8B70.5960.69%0.1951
StartLux-Decision-v1.1-0.8B-Instant Open · 0.8B70.0959.60%0.1942
GLiDE 0.4B Open · 0.4B69.8459.41%0.1973
OneJev1.6-0.8B Open · 0.8B69.8259.41%0.1978
KnowLine-0.8B-Gen1 Open · 0.8B69.6558.87%0.1956
GLiClass v4 Edge Open · 0.4B69.0558.32%0.2021
Decider 1 Closed · Efficient67.4755.76%0.2082
Strands Decider 2B Open · 2.0B64.0845.52%0.1736
Laya Open · 0.4B51.5722.85%0.1971
GLiNER2.5-multi-Decide Open · 0.3B47.5821.21%0.2604
Laya Multilingual Open · 0.3B39.3215.36%0.3671

Decision Score (AA formula): 50 × (1 + accuracy − chosen-answer Brier) on the first English run, the chosen-answer Brier being the mean of (p − c)², p the probability given to the chosen answer and c 1 if it is right, else 0. GPT-6 Luna · Decisions API: unavailable, 2 of 547 answers without a probability for the chosen answer.

Artificial Analysis's formula applied to our questions. Not their Index. Decision models only: chat models and the I don't know runs are excluded.

Turkish reference (the Text tab runs)

The answers each setup gave at 99% confidence or more, in its Text tab run on the Turkish questions, for reference, the bar their share of the 2,198 questions; how many of them were wrong; and its ranking score (AUROC: how often a right answer has higher confidence than a wrong one, 0.5 chance, 1 perfect). No wrong answer at 99% first, by answers kept, then by the share wrong among the kept answers; a setup with no answers at 99% last · more kept and fewer wrong is better

Decision models

Confidence: option probabilities.

  1. Karpo Decision 31B50500.879
  2. Perplexity Decider 27B v1.2 DGX Spark36000.837
  3. Perplexity Decider 27B v1.2 API32700.843
  4. Gemini Decision 1 2026111131800.862
  5. Mammoth-128500.833
  6. Perplexity Decider 27B v1.327600.825
  7. d325000.835
  8. GLiDE 27B24700.843
  9. GLiDE v1.224000.839
  10. REAX 1.324000.868
  11. Drex Max v1.122000.834
  12. Elephant-115500.788
  13. KnowLine-31B-Gen1.514300.867
  14. Copernicus-31B10700.876
  15. d1-ultra9000.850
  16. d1.18700.806
  17. KnowLine-31B-Gen28100.867
  18. KnowLine-9B-Gen16200.785
  19. Logits Sigma-1 Duo4200.855
  20. Turion Jev 31B4200.872
  21. KnowLine 4B Gen5.53900.751
  22. d1.23300.807
  23. Drex 2.03300.804
  24. Perplexity Decider Flash v1.32900.742
  25. GLiDE 9B2700.789
  26. Verdin-4B-v1.12700.753
  27. Perplexity Decider 27B v12100.827
  28. StartLux-Decision-v1.1-9B-Instant2000.783
  29. GLiDE 2B1900.640
  30. d1.2-ultra1100.825
  31. StartLux-Decision-v1.1-4B-Instant900.763
  32. Valen-Decision-1.5-9B700.767
  33. vera-medium-omni400.746
  34. Cygnet200.802
  35. OneJev-27B200.818
  36. Drex Max44310.826
  37. d3-ultra38410.833
  38. Perplexity Decider 27B v1.2 API · with web35110.826
  39. Sieve 31B v332910.873
  40. Jev 1.1331210.807
  41. SubQ Snap 1.129910.836
  42. d127410.792
  43. Jev 1.1426010.790
  44. Winnow-12B16310.789
  45. Dexy-Decision-9B-V213810.773
  46. GPT-6 Luna Decisions API36830.820
  47. Adjudex 9B v513220.785
  48. Chinese Decision Pointer 4B5110.729
  49. Mercury Decide451130.808
  50. Microsoft-Decision-16130.749
  51. GLiDE 0.8B1810.620
  52. Laya Multilingual110.454
  53. celeris-1-decisionNo answers at this threshold––0.805
  54. Cerno 9B v1.2No answers at this threshold––0.777
  55. ClefNo answers at this threshold––0.785
  56. Clef-FlashNo answers at this threshold––0.740
  57. d1-2BNo answers at this threshold––0.637
  58. Decider 1No answers at this threshold––0.592
  59. GLiClass v4 EdgeNo answers at this threshold––0.558
  60. GLiDE 0.4BNo answers at this threshold––0.516
  61. GLiNER2.5-multi-DecideNo answers at this threshold––0.537
  62. Jeffan-9B-1211No answers at this threshold––0.798
  63. KnowLine-0.8B-Gen1No answers at this threshold––0.634
  64. KnowLine-12B-Gen1No answers at this threshold––0.810
  65. KnowLine-2B-Gen1No answers at this threshold––0.661
  66. KnowLine-9B-Gen2No answers at this threshold––0.784
  67. LayaNo answers at this threshold––0.479
  68. Metask-Jev-4BNo answers at this threshold––0.725
  69. OneJev1.6-0.8BNo answers at this threshold––0.605
  70. Open-Jev-27B-v1.1No answers at this threshold––0.823
  71. Perplexity Decider Flash v1.2No answers at this threshold––0.757
  72. StartLux-Decision-v1.1-0.8B-InstantNo answers at this threshold––0.606
  73. StartLux-Decision-v1.1-2B-InstantNo answers at this threshold––0.663
  74. Strands Decider 2BNo answers at this threshold––0.647
  75. SubQ Snap PreviewNo answers at this threshold––0.825

Karpo Perplexity Gemini Mammoth vLLM-SR Fastino REAX Drex Elephant KnowLine Copernicus Liquid Logits Turion Jev Verdin StartLux Valen vera Cygnet OneJev Sieve Jev SubQ Winnow Dexy GPT Adjudex Chinese Decision Pointer Inception Microsoft Laya Celeris Cerno Clef meraGPT GLiClass Jeffan Metask Open-Jev Strands

Chat models

Confidence: the confidence the model states for 39; option probabilities for 9; agreement across ten answers for 1.

  1. Inkling-Small high · chat · Jul 20263400.778
  2. GLM-5.3-Flash reasoning on · chat900.760
  3. Claude Fable 5.2 max · chat400.896
  4. GLM-5.3-Flash reasoning off · chat400.733
  5. Qwen4-Max high · chat300.857
  6. Claude Opus 5.5 low · chat · 1 per request100.882
  7. GLM-5.3 low · chat100.739
  8. Muse Ember max · chat100.800
  9. Gemini 4 Carbon high · chat1,79750.892
  10. GPT-6 Astra low · chat1,72550.918
  11. Gemini 4 Hydrogen high · chat1,43750.890
  12. GPT-6 Astra high · chat1,79370.890
  13. Gemini 3.8 Flash low · chat · 1 per request1,36560.842
  14. GPT-6 Astra medium · chat1,74980.897
  15. Gemini 3.8 Flash high · chat1,845100.833
  16. GPT-6.1 Sol high · chat1,826100.879
  17. Qwen3.8-27B yes/no scoringoption probabilities28920.802
  18. GPT-6.1 Sol low · chat · 1 per request1,741140.859
  19. GPT-6.1 Sol low · chat1,610130.859
  20. GPT-6 Astra low · chat · 1 per request1,839150.861
  21. Gemini 4 Argon low · chat1,842230.855
  22. GPT-6.1 Terra high · chat1,363190.812
  23. Gemini 3.8 Flash low · chat976150.766
  24. GPT-6.1 Terra low · chat1,306270.820
  25. DeepSeek-V4.1-Flash reasoning off · chat · 1017option probabilities696160.828
  26. Erk-14B reasoning off · chatoption probabilities10150.754
  27. MiMo-V2.6-Flash high · chat1810.709
  28. DeepSeek-V4.1-Flash reasoning on · chat · 1017option probabilities2,1641490.614
  29. Kimi K3.1 Pro low · chat1,4361140.750
  30. Kimi K3.1 Pro no reasoning · chat442360.728
  31. Qwen3.5-397B-A17B high · chat7880.744
  32. Qwen3.8-27B stated confidence · reasoning off · chat810.515
  33. MiniMax-M3.1-Flash high · chat375540.692
  34. Qwen3.8-27B vote share · reasoning off · chatagreement across ten answers8751290.777
  35. Qwen3.8 Flash no reasoning · chat5590.660
  36. GPT-6 Luna low · chat8301370.685
  37. Gemma 4 31B reasoning off · chatoption probabilities1,7823320.833
  38. Qwen3.8-27B low · chatoption probabilities2,1455060.636
  39. Gemma 4 31B reasoning off · chat · 1 per requestoption probabilities1,8264870.820
  40. Qwen3-14B reasoning off · chatoption probabilities9753420.734
  41. Claude Fable 5.1 high · chatNo answers at this threshold––0.858
  42. Claude Haiku 5.5 max · chatNo answers at this threshold––0.819
  43. Claude Opus 5.5 high · chatNo answers at this threshold––0.837
  44. Claude Opus 5.5 low · chatNo answers at this threshold––0.840
  45. Claude Sonnet 5.5 high · chatNo answers at this threshold––0.781
  46. Claude Sonnet 5.5 high · chat · 1 per requestNo answers at this threshold––0.838
  47. Claude Sonnet 5.5 low · chatNo answers at this threshold––0.786
  48. GLiNER2.5 Multioption probabilities · No answers at this threshold––0.451
  49. Grok 4.7 high · chatNo answers at this threshold––0.849

Inkling GLM Claude Qwen Muse Gemini GPT DeepSeek Erk MiMo Kimi MiniMax Gemma Fastino Grok Confidence not measured (a plain answer line, no confidence): Qwen3.5-4B · reasoning on · chat, Gemma 4 E4B · default reasoning · chat, Qwen3.5-4B · reasoning off · chat, LFM2.5-2.6B · default reasoning · chat, LFM3-1.7B · reasoning off · chat, LFM3-1.7B · reasoning on · chat, LFM2.5-8B-A1B · default reasoning · chat, MiniCPM5-2B · reasoning off · chat and Qwen3.5-2B · reasoning off · chat.

Cost per 1,000 English questionsper 1M English questions

Estimated dollars per 1,000 questionsper 1M questions: each of the 142 setups' first English run, ordered by English cost · lower is better

  1. celeris-1.1-decision$0.002$1.61
  2. Perplexity Decider Flash v1.2$0.004$3.93
  3. d1$0.007$6.66
  4. d1.1$0.007$6.66
  5. d1.2$0.007$6.66
  6. GLiNER2.5-multi-Decide$0.007$6.73
  7. Mercury Decide$0.008$7.79
  8. Perplexity Decider 27B v1.2 API$0.008$7.79
  9. Laya$0.008$7.84
  10. Perplexity Decider 27B v1.2 DGX Spark$0.008$7.85
  11. d1-2B$0.008$8.08
  12. GLiNER2.5 Multi$0.008$8.39
  13. Laya Multilingual$0.013$12.62
  14. d1-ultra$0.014$13.98
  15. d1.2-ultra$0.014$13.98
  16. Microsoft-Decision-1$0.014$14.01
  17. Elephant-1$0.014$14.02
  18. GLiDE v1.2$0.015$14.55
  19. Decider 1$0.015$14.95
  20. Perplexity Decider 27B v1$0.016$15.59
  21. SubQ Snap 1.1$0.016$15.59
  22. celeris-1-decision$0.016$16.09
  23. Clef-Flash$0.018$17.53
  24. Strands Decider 2B$0.018$18.15
  25. Drex 2.0$0.019$19.42
  26. SubQ Snap Preview$0.019$19.48
  27. Perplexity Decider Flash v1.3$0.023$23.17
  28. Drex Max v1.1$0.023$23.38
  29. Drex Max$0.024$23.56
  30. Adjudex 9B v5$0.025$24.93
  31. Dexy-Decision-9B-V2$0.025$25.41
  32. Jev 1.13$0.027$26.64
  33. Jev 1.14$0.027$26.64
  34. Chinese Decision Pointer 4B$0.027$27.21
  35. Qwen3-14B reasoning off · chat$0.031$31.17
  36. Verdin-4B-v1.1$0.034$34.28
  37. vera-medium-omni$0.037$36.63
  38. KnowLine-9B-Gen2$0.039$39.48
  39. Gemini Decision 1 20261111$0.040$39.51
  40. Mammoth-1$0.040$40.07
  41. Valen-Decision-1.5-9B$0.041$40.55
  42. GPT-6 Luna Decisions API$0.041$41.09
  43. Cygnet$0.041$41.42
  44. KnowLine-9B-Gen1$0.042$41.55
  45. Jeffan-9B-1211$0.046$45.85
  46. GLM-5.3-Flash reasoning off · chat$0.046$46.07
  47. Winnow-12B$0.050$49.59
  48. Qwen3.8 Flash no reasoning · chat$0.050$49.69
  49. Gemma 4 31B reasoning off · chat$0.053$52.81
  50. d3$0.055$55.35
  51. KnowLine-12B-Gen1$0.057$56.86
  52. GPT-6 Luna low · chat$0.058$57.70
  53. Metask-Jev-4B$0.059$59.11
  54. Gemma 4 31B reasoning off · chat · 1 per request$0.070$70.17
  55. DeepSeek-V4.1-Flash reasoning off · chat · 1017$0.079$79.31
  56. GLM-5.3-Flash reasoning on · chat$0.081$81.38
  57. Perplexity Decider 27B v1.3$0.083$82.63
  58. OneJev-27B$0.106$106
  59. d3-ultra$0.106$106
  60. Clef$0.111$111
  61. Gemma 4 E2B default reasoning · chat$0.115$115
  62. Erk-14B reasoning off · chat$0.123$123
  63. KnowLine-31B-Gen2$0.127$127
  64. Copernicus-31B$0.142$142
  65. Karpo Decision 31B$0.145$145
  66. Gemma 4 12B Unified default reasoning · chat$0.154$154
  67. REAX 1.3$0.169$169
  68. Gemma 4 E2B reasoning on · chat$0.187$187
  69. Logits Sigma-1 Duo$0.191$191
  70. MiniMax-M3.1-Flash high · chat$0.193$193
  71. Gemma 4 E4B default reasoning · chat$0.199$199
  72. Qwen3.8-27B stated confidence · reasoning off · chat$0.212$212
  73. Raon-1 nano reasoning off · chat$0.235$235
  74. Muse Ember max · chat$0.245$245
  75. Qwen3.5-2B reasoning off · chat$0.280$280
  76. Gemini 3.8 Flash low · chat$0.288$288
  77. MiMo-V2.6-Flash high · chat$0.298$298
  78. Qwen3.5-4B reasoning off · chat$0.309$309
  79. Open-Jev-27B-v1.1$0.351$351
  80. Claude Sonnet 5.5 low · chat$0.354$354
  81. LFM3-1.7B reasoning off · chat$0.355$355
  82. LFM3-1.7B reasoning on · chat$0.387$387
  83. GLM-5.3 low · chat$0.408$408
  84. DeepSeek-V4.1-Flash reasoning on · chat · 1017$0.421$421
  85. Inkling-Small high · chat · Jul 2026$0.425$425
  86. MiniCPM5-2B reasoning off · chat$0.429$429
  87. GPT-6.1 Terra low · chat$0.491$491
  88. GPT-6.1 Terra high · chat$0.509$509
  89. LFM2.5-8B-A1B default reasoning · chat$0.517$517
  90. Qwen3.8-27B vote share · reasoning off · chat$0.645$645
  91. MiniCPM5-2B reasoning on · chat$0.714$714
  92. Kimi K3.1 Pro no reasoning · chat$0.735$735
  93. Qwen3.8-27B yes/no scoring$0.741$741
  94. Qwen3.8-27B low · chat$0.853$853
  95. Gemini 3.8 Flash low · chat · 1 per request$0.858$858
  96. LFM2.5-2.6B default reasoning · chat$0.964$964
  97. Kimi K3.1 Pro low · chat$0.984$984
  98. Perplexity Decider 27B v1.2 API · with web≤ $1.036≤ $1,036
  99. Claude Opus 5.5 low · chat$1.087$1,087
  100. Qwen4-Max high · chat$1.119$1,119
  101. GPT-6.1 Sol low · chat$1.219$1,219
  102. Claude Haiku 5.5 max · chat$1.261$1,261
  103. Qwen3.5-2B reasoning on · chat$1.361$1,361
  104. Qwen3.5-4B reasoning on · chat$1.390$1,390
  105. GPT-6.1 Sol high · chat$1.427$1,427
  106. Qwen3.5-397B-A17B high · chat$1.742$1,742
  107. LFM2.5-24B-A2B default reasoning · chat$1.764$1,764
  108. Gemini 4 Argon low · chat$1.900$1,900
  109. Claude Sonnet 5.5 high · chat$1.998$1,998
  110. Granite-4.2-3B reasoning on · chat$2.717$2,717
  111. Gemini 3.8 Flash high · chat$3.695$3,695
  112. Claude Opus 5.5 high · chat$4.519$4,519
  113. GPT-6 Astra medium · chat$4.696$4,696
  114. Gemini 4 Hydrogen high · chat$4.927$4,927
  115. Claude Sonnet 5.5 high · chat · 1 per request$5.513$5,513
  116. GPT-6 Astra low · chat$5.918$5,918
  117. GPT-6.1 Sol low · chat · 1 per request$6.113$6,113
  118. GPT-6 Astra high · chat$6.334$6,334
  119. Grok 4.7 high · chat$6.851$6,851
  120. Claude Opus 5.5 low · chat · 1 per request$10.094$10,094
  121. Claude Fable 5.1 high · chat$11.531$11,531
  122. GPT-6 Astra low · chat · 1 per request$28.653$28,653
  123. Gemini 4 Carbon high · chat$38.053$38,053
  124. Claude Fable 5.2 max · chat$52.942$52,942
  125. Cerno 9B v1.2no list price
  126. GLiClass v4 Edgeno list price
  127. GLiDE 0.4Bno list price
  128. GLiDE 0.8Bno list price
  129. GLiDE 27Bno list price
  130. GLiDE 2Bno list price
  131. GLiDE 9Bno list price
  132. KnowLine 4B Gen5.5no list price
  133. KnowLine-0.8B-Gen1no list price
  134. KnowLine-2B-Gen1no list price
  135. KnowLine-31B-Gen1.5no list price
  136. OneJev1.6-0.8Bno list price
  137. Sieve 31B v3no list price
  138. StartLux-Decision-v1.1-0.8B-Instantno list price
  139. StartLux-Decision-v1.1-2B-Instantno list price
  140. StartLux-Decision-v1.1-4B-Instantno list price
  141. StartLux-Decision-v1.1-9B-Instantno list price
  142. Turion Jev 31Bno list price

Celeris Perplexity Liquid Fastino Inception Laya Microsoft Elephant meraGPT SubQ Clef Strands Drex Adjudex Dexy Jev Chinese Decision Pointer Qwen Verdin vera KnowLine Gemini Mammoth Valen GPT Cygnet Jeffan GLM Winnow Gemma vLLM-SR Metask DeepSeek OneJev Erk Copernicus Karpo REAX Logits MiniMax Raon Muse MiMo Open-Jev Claude Inkling MiniCPM Kimi Granite Grok Cerno GLiClass Sieve StartLux Turion Jev Costs: API list prices and recorded token use. Setups on our own GPUs: the maker's list price for the same model (each value's tooltip names it); our hardware cost is not measured. On our GPUs, estimated at a public host's list price for the same weights (the host, the date read and any precision difference are on each setup's page): an API-equivalent estimate, not measured spending. On our GPUs with no public price for the same weights, estimated from GPU time: the run's request time x the GPUs its server held x Runpod's on-demand price for an RTX PRO 6000, $2.49 per GPU-hour (read 5 November 2026): a cloud-equivalent estimate, not measured spending. No list price: on our own GPUs, with no estimate. ≤ Upper-bound estimate: one $0.001 web search counted per request; Perplexity bills only completed searches. Log scale. No list price: Cerno 9B v1.2, GLiClass v4 Edge, GLiDE 0.4B, GLiDE 0.8B, GLiDE 27B, GLiDE 2B, GLiDE 9B, KnowLine 4B Gen5.5, KnowLine-0.8B-Gen1, KnowLine-2B-Gen1, KnowLine-31B-Gen1.5, OneJev1.6-0.8B, Sieve 31B v3, StartLux-Decision-v1.1-0.8B-Instant, StartLux-Decision-v1.1-2B-Instant, StartLux-Decision-v1.1-4B-Instant, StartLux-Decision-v1.1-9B-Instant and Turion Jev 31B, on our own hardware; the maker lists no price for the same model.

Median latency per request on the 547 English questions

Median time per request: each of the 142 setups' first English run, ordered by English latency · lower is better

  1. GLiNER2.5-multi-Decide0.009 s
  2. GLiNER2.5 Multi0.009 s
  3. Laya0.011 s
  4. d1-2B0.011 s
  5. GLiDE 0.4B0.014 s
  6. GLiClass v4 Edge0.017 s
  7. OneJev1.6-0.8B0.018 s
  8. Laya Multilingual0.018 s
  9. GLiDE 0.8B0.023 s
  10. Strands Decider 2B0.026 s
  11. StartLux-Decision-v1.1-0.8B-Instant0.028 s
  12. Perplexity Decider Flash v1.30.033 s
  13. KnowLine-0.8B-Gen10.033 s
  14. Adjudex 9B v50.034 s
  15. GLiDE 2B0.035 s
  16. Dexy-Decision-9B-V20.036 s
  17. Perplexity Decider Flash v1.20.037 s
  18. Chinese Decision Pointer 4B0.038 s
  19. StartLux-Decision-v1.1-2B-Instant0.040 s
  20. Cygnet0.043 s
  21. Verdin-4B-v1.10.052 s
  22. vera-medium-omni0.054 s
  23. KnowLine-9B-Gen20.056 s
  24. KnowLine-9B-Gen10.059 s
  25. Valen-Decision-1.5-9B0.059 s
  26. KnowLine-2B-Gen10.062 s
  27. Jeffan-9B-12110.067 s
  28. Elephant-10.070 s
  29. Winnow-12B0.070 s
  30. d30.074 s
  31. StartLux-Decision-v1.1-4B-Instant0.075 s
  32. KnowLine-12B-Gen10.078 s
  33. Drex Max0.081 s
  34. Metask-Jev-4B0.085 s
  35. KnowLine 4B Gen5.50.099 s
  36. GLiDE v1.20.104 s
  37. Perplexity Decider 27B v1.30.115 s
  38. Perplexity Decider 27B v1.2 DGX Spark0.126 s
  39. OneJev-27B0.130 s
  40. Clef-Flash0.140 s
  41. d3-ultra0.149 s
  42. StartLux-Decision-v1.1-9B-Instant0.156 s
  43. KnowLine-31B-Gen20.171 s
  44. Jev 1.130.173 s
  45. d1.10.173 s
  46. d10.180 s
  47. Jev 1.140.181 s
  48. Copernicus-31B0.188 s
  49. Mammoth-10.189 s
  50. GLiDE 9B0.207 s
  51. Karpo Decision 31B0.207 s
  52. Cerno 9B v1.20.211 s
  53. d1.20.219 s
  54. d1-ultra0.220 s
  55. Clef0.220 s
  56. d1.2-ultra0.221 s
  57. REAX 1.30.229 s
  58. Gemini Decision 1 202611110.235 s
  59. Perplexity Decider 27B v10.247 s
  60. Perplexity Decider 27B v1.2 API0.256 s
  61. Logits Sigma-1 Duo0.263 s
  62. Drex 2.00.283 s
  63. Gemma 4 31B reasoning off · chat · 1 per request0.299 s
  64. Mercury Decide0.310 s
  65. Microsoft-Decision-10.315 s
  66. GPT-6 Luna Decisions API0.323 s
  67. Drex Max v1.10.335 s
  68. SubQ Snap 1.10.381 s
  69. SubQ Snap Preview0.381 s
  70. celeris-1.1-decision0.385 s
  71. celeris-1-decision0.390 s
  72. Qwen3.8-27B stated confidence · reasoning off · chat0.390 s
  73. Qwen3.8-27B yes/no scoring0.413 s
  74. Turion Jev 31B0.445 s
  75. Perplexity Decider 27B v1.2 API · with web0.484 s
  76. Open-Jev-27B-v1.10.495 s
  77. GLiDE 27B0.545 s
  78. Decider 10.699 s
  79. Sieve 31B v31.06 s
  80. KnowLine-31B-Gen1.51.12 s
  81. Qwen3.8-27B vote share · reasoning off · chat1.70 s
  82. Gemini 3.8 Flash low · chat · 1 per request2.85 s
  83. Erk-14B reasoning off · chat2.86 s
  84. Qwen3-14B reasoning off · chat2.86 s
  85. Claude Sonnet 5.5 high · chat · 1 per request3.02 s
  86. Gemma 4 31B reasoning off · chat3.19 s
  87. GLM-5.3-Flash reasoning off · chat3.37 s
  88. Claude Opus 5.5 low · chat · 1 per request3.50 s
  89. Claude Sonnet 5.5 low · chat3.68 s
  90. Raon-1 nano reasoning off · chat3.74 s
  91. Claude Sonnet 5.5 high · chat3.85 s
  92. Gemini 3.8 Flash low · chat4.19 s
  93. LFM3-1.7B reasoning off · chat4.69 s
  94. Gemma 4 E2B default reasoning · chat4.84 s
  95. LFM3-1.7B reasoning on · chat5.24 s
  96. GPT-6.1 Sol low · chat · 1 per request5.59 s
  97. Gemini 4 Argon low · chat6.03 s
  98. Kimi K3.1 Pro no reasoning · chat6.31 s
  99. GPT-6 Astra low · chat · 1 per request6.43 s
  100. DeepSeek-V4.1-Flash reasoning off · chat · 10176.59 s
  101. MiMo-V2.6-Flash high · chat6.67 s
  102. GLM-5.3 low · chat6.84 s
  103. MiniCPM5-2B reasoning off · chat7.29 s
  104. GPT-6.1 Terra low · chat7.29 s
  105. Gemma 4 E2B reasoning on · chat7.61 s
  106. Claude Opus 5.5 low · chat7.64 s
  107. GPT-6 Luna low · chat7.72 s
  108. GPT-6.1 Terra high · chat7.94 s
  109. Gemma 4 E4B default reasoning · chat8.39 s
  110. Qwen3.5-2B reasoning off · chat9.27 s
  111. Qwen3.5-4B reasoning off · chat9.74 s
  112. Qwen3.8 Flash no reasoning · chat10.1 s
  113. GPT-6.1 Sol low · chat10.3 s
  114. GPT-6 Astra medium · chat10.7 s
  115. Claude Opus 5.5 high · chat11.3 s
  116. GPT-6 Astra low · chat11.5 s
  117. Muse Ember max · chat11.6 s
  118. LFM2.5-2.6B default reasoning · chat15.0 s
  119. GPT-6 Astra high · chat15.6 s
  120. Claude Fable 5.1 high · chat15.6 s
  121. Gemma 4 12B Unified default reasoning · chat15.8 s
  122. Kimi K3.1 Pro low · chat16.6 s
  123. GPT-6.1 Sol high · chat16.9 s
  124. MiniCPM5-2B reasoning on · chat17.5 s
  125. LFM2.5-8B-A1B default reasoning · chat19.4 s
  126. Inkling-Small high · chat · Jul 202625.3 s
  127. MiniMax-M3.1-Flash high · chat26.7 s
  128. Gemini 4 Hydrogen high · chat28.4 s
  129. Qwen4-Max high · chat29.5 s
  130. GLM-5.3-Flash reasoning on · chat33.1 s
  131. Granite-4.2-3B reasoning on · chat40.5 s
  132. Qwen3.5-2B reasoning on · chat54.4 s
  133. Qwen3.5-4B reasoning on · chat58.3 s
  134. Qwen3.8-27B low · chat71.7 s
  135. Claude Fable 5.2 max · chat74.0 s
  136. LFM2.5-24B-A2B default reasoning · chat74.2 s
  137. Gemini 3.8 Flash high · chat104.6 s
  138. Claude Haiku 5.5 max · chat148.9 s
  139. Qwen3.5-397B-A17B high · chat162.9 s
  140. DeepSeek-V4.1-Flash reasoning on · chat · 1017189.4 s
  141. Grok 4.7 high · chat197.2 s
  142. Gemini 4 Carbon high · chat287.8 s

Fastino Laya Liquid GLiClass OneJev Strands StartLux Perplexity KnowLine Adjudex Dexy Chinese Decision Pointer Cygnet Verdin vera Valen Jeffan Elephant Winnow vLLM-SR Drex Metask Clef Jev Copernicus Mammoth Karpo Cerno REAX Gemini Logits Gemma Inception Microsoft GPT SubQ Celeris Qwen Turion Jev Open-Jev meraGPT Sieve Erk Claude GLM Raon Kimi DeepSeek MiMo MiniCPM Muse Inkling MiniMax Granite Grok Time per request as measured from our computer during each run; API times include the network. English runs: 1 question per request for 104 runs and 16 for 38; 1 request in flight for 108 runs, 32 for 17, 4 for 14 and 8 for 3. Of the 85 self-hosted setups, KnowLine-31B-Gen1.5, GLiDE 27B, Sieve 31B v3, Turion Jev 31B, StartLux-Decision-v1.1-9B-Instant, GLiDE 9B, Cerno 9B v1.2, StartLux-Decision-v1.1-4B-Instant, KnowLine 4B Gen5.5, KnowLine-2B-Gen1, GLiDE 2B, StartLux-Decision-v1.1-2B-Instant, GLiDE 0.8B, StartLux-Decision-v1.1-0.8B-Instant, GLiDE 0.4B, OneJev1.6-0.8B, KnowLine-0.8B-Gen1 and GLiClass v4 Edge ran their English runs on an NVIDIA DGX Spark, the others on our RTX PRO 6000 GPUs (Our method). Clef and Clef-Flash ran their English runs at the same time on one GPU. Log scale.

Limit: a model translated the questions and no person checked them (Our method).

More about this page

Calibration of holding back by confidence at the 99% threshold is not measured in English.