Turkish vs English
The Text tab's reading questions, in Turkish and in English
547 of the Text tab's 656 reading questions, translated into English by a model, asked of 17 of the 20 decision models we chose (which ones) as on the Text tab and compared with the same models' Turkish runs on the same questions. No person checked the translations: a limit of this test (Our method). This tab is separate from the leaderboard; it ranks nothing on the Text tab and changes none of its numbers.
Four of the 17 models compared scored higher on the English translation than on the Turkish original; 13 scored lower; one of the 17 differences is significant after Holm's correction: Microsoft-Decision-1 (higher in English) (Paired test).
Turkish vs English
| Model | Accuracy | Answered with “I don't know” | Wrong of answers with “I don't know” | Non-hallucination with “I don't know” | Turkish with “Bilmiyorum”wrong of answers, lower is better; answered and non-hallucination under it | Rank by accuracy | Languages the model card listsand its source | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Turkishhigher is better | Englishhigher is better | DifferenceEnglish minus Turkish, in points; positive = higher in English | Turkish(right + wrong) / 547; higher is better | English(right + wrong) / 547; higher is better | Turkishwrong / (right + wrong); lower is better | Englishwrong / (right + wrong); lower is better | Turkish“I don't know” / (“I don't know” + wrong); higher is better | English“I don't know” / (“I don't know” + wrong); higher is better | DifferenceEnglish minus Turkish non-hallucination, in points; positive = higher in English | Turkishamong the 17; lower is better | Englishamong the 17; lower is better | |||
| Liquid d1-ultra | 93.6% | 90.7% | -2.9 | 95.1%520 of 547 answered | 90.7%496 of 547 answered | 5.4%28 of 520 | 7.9%39 of 496 | 49.1%27 “I don't know” of 55 not right | 56.7%51 “I don't know” of 90 not right | +7.6 | 5.0%26 of 517answered 94.5%, non-hallucination 53.6% | 1 | 1 | not stateddocs.liquid.ai |
| Mammoth-1 | 92.9% | 90.1% | -2.7 | 95.4%522 of 547 answered | 93.2%510 of 547 answered | 6.1%32 of 522 | 7.1%36 of 510 | 43.9%25 “I don't know” of 57 not right | 50.7%37 “I don't know” of 73 not right | +6.8 | 6.3%33 of 523answered 95.6%, non-hallucination 42.1% | 2 | 3 | English (the card's language: en tag; its text names no language)huggingface.co |
| Drex Max v1 | 91.4% | 89.6% | -1.8 | 99.8%546 of 547 answered | Not measured | 9.0%49 of 546 | Not measured | 2.0%1 “I don't know” of 50 not right | Not measured | Not measured | 9.7%53 of 545answered 99.6%, non-hallucination 3.6% | 3 | 5 | not statedhuggingface.co |
| Perplexity Decider 27B v1.2 | 91.4% | 88.7% | -2.7 | 99.8%546 of 547 answered | 99.6%545 of 547 answered | 9.3%51 of 546 | 11.7%64 of 545 | 1.9%1 “I don't know” of 52 not right | 3.0%2 “I don't know” of 66 not right | +1.1 | 9.1%50 of 547answered 100.0%, non-hallucination 0.0% | 3 | 7 | not statedhuggingface.co |
| vLLM-SR d3-ultra | 91.4% | 89.4% | -2.0 | 99.6%545 of 547 answered | Not measured | 9.4%51 of 545 | Not measured | 3.8%2 “I don't know” of 53 not right | Not measured | Not measured | 8.4%46 of 546answered 99.8%, non-hallucination 2.1% | 3 | 6 | not statedhuggingface.co |
| vLLM-SR d3 | 91.0% | 89.8% | -1.3 | 100.0%547 of 547 answered | Not measured | 9.5%52 of 547 | Not measured | 0.0%0 “I don't know” of 52 not right | Not measured | Not measured | 9.7%53 of 547answered 100.0%, non-hallucination 0.0% | 6 | 4 | not statedhuggingface.co |
| Fastino GLiDE v1.2 | 90.5% | 90.3% | -0.2 | 98.4%538 of 547 answered | 94.5%517 of 547 answered | 8.7%47 of 538 | 7.0%36 of 517 | 16.1%9 “I don't know” of 56 not right | 45.5%30 “I don't know” of 66 not right | +29.4 | 8.7%47 of 541answered 98.9%, non-hallucination 11.3% | 7 | 2 | not statedhuggingface.co |
| Drex Max 1.1 | 90.1% | 88.5% | -1.6 | 96.7%529 of 547 answered | 92.1%504 of 547 answered | 9.1%48 of 529 | 8.1%41 of 504 | 27.3%18 “I don't know” of 66 not right | 51.2%43 “I don't know” of 84 not right | +23.9 | 9.5%51 of 538answered 98.4%, non-hallucination 15.0% | 8 | 8 | not statedconsole.nace.ai |
| SubQ Snap 1.1 | 89.2% | 85.9% | -3.3 | 96.5%528 of 547 answered | 96.2%526 of 547 answered | 8.5%45 of 528 | 12.0%63 of 526 | 29.7%19 “I don't know” of 64 not right | 25.0%21 “I don't know” of 84 not right | -4.7 | 9.7%51 of 528answered 96.5%, non-hallucination 27.1% | 9 | 11 | not stateddocs.subq.ai |
| Celeris-1 Decision | 87.4% | 84.1% | -3.3 | Not measured | 83.2%455 of 547 answered | Not measured | 8.1%37 of 455 | Not measured | 71.3%92 “I don't know” of 129 not right | Not measured | Not measured | 10 | 15 | not stateddocs.celeris.ai |
| Jev 1.14.0 | 87.2% | 86.1% | -1.1 | Not measured | 99.5%544 of 547 answered | Not measured | 13.2%72 of 544 | Not measured | 4.0%3 “I don't know” of 75 not right | Not measured | 12.7%69 of 544answered 99.5%, non-hallucination 4.2% | 11 | 10 | "English is the primary training language and where accuracy is currently best. Other languages, including CJK scripts, are handled but not equally well"docs.typesafe.ai |
| Inception Mercury Decide | 87.0% | 85.7% | -1.3 | Not measured | 98.0%536 of 547 answered | Not measured | 13.2%71 of 536 | Not measured | 13.4%11 “I don't know” of 82 not right | Not measured | 13.2%71 of 539answered 98.5%, non-hallucination 10.1% | 12 | 12 | not statedopenrouter.ai |
| GPT-6 Luna (Decisions API) | 86.7% | 84.1% | -2.6 | Not measured | 98.2%537 of 547 answered | Not measured | 17.9%96 of 537 | Not measured | 9.4%10 “I don't know” of 106 not right | Not measured | 14.4%78 of 541answered 98.9%, non-hallucination 7.1% | 13 | 15 | not stateddevelopers.openai.com |
| Liquid d1 | 85.0% | 86.8% | +1.8 | Not measured | 99.6%545 of 547 answered | Not measured | 12.3%67 of 545 | Not measured | 2.9%2 “I don't know” of 69 not right | Not measured | 16.3%89 of 546answered 99.8%, non-hallucination 1.1% | 14 | 9 | not stateddocs.liquid.ai |
| Elephant-1 | 83.5% | 85.4% | +1.8 | 95.6%523 of 547 answered | 95.8%524 of 547 answered | 15.7%82 of 523 | 12.8%67 of 524 | 22.6%24 “I don't know” of 106 not right | 25.6%23 “I don't know” of 90 not right | +2.9 | 15.4%82 of 532answered 97.3%, non-hallucination 15.5% | 15 | 14 | English (the card's language: en tag; its text names no language)huggingface.co |
| Perplexity Decider Flash v1.2 | 76.6% | 81.9% | +5.3 | Not measured | 99.1%542 of 547 answered | Not measured | 18.5%100 of 542 | Not measured | 4.8%5 “I don't know” of 105 not right | Not measured | 24.6%133 of 541answered 98.9%, non-hallucination 4.3% | 16 | 17 | not statedhuggingface.co |
| Microsoft-Decision-1 | 76.4% | 85.7% | +9.3 * | Not measured | 99.5%544 of 547 answered | Not measured | 14.2%77 of 544 | Not measured | 3.8%3 “I don't know” of 80 not right | Not measured | 16.1%88 of 545answered 99.6%, non-hallucination 2.2% | 17 | 12 | not statedopenrouter.ai |
Ordered by Turkish accuracy, highest first. * marks a difference that is significant after Holm's correction: its paired test (the exact McNemar test, with p adjusted by Holm's method over the models compared) has a Holm-adjusted p below 0.05. Accuracy: right answers over the 547 kept reading questions, the first English run against the Text tab run (Accuracy, Paired test). The “I don't know” columns: each model's first English run with “I don't know” against its Turkish run with the option also written “I don't know” (the Hallucination tab's option-word run, counted on these questions), so only the language differs (Only the language differs). Answered: (right + wrong) / 547; Wrong of answers: wrong / (right + wrong); Non-hallucination: “I don't know” / (“I don't know” + wrong); the difference's paired test is of wrong answers, with p adjusted by Holm's method over the seven models with both runs. Turkish with “Bilmiyorum”: the Turkish side of the published comparison with “Bilmiyorum” (With “I don't know”) or, for a model outside it, its Hallucination tab run counted on these questions; its wrong of answers with its answered share and non-hallucination rate under it. Not measured: no such run. Rank: by accuracy among the 17 compared models in each language, 1 the most accurate, tied models sharing a rank. Languages: what each model's card or its maker's docs list, with the source (Languages the model cards list). The full tables are under Info.
Accuracy in Turkish and in English
Share of the 547 kept reading questions answered right: each model's Text tab run and its first English run, ordered by English accuracy · higher is better
Turkish (the Text tab run), the upper bar English (first run), the lower bar
Right only in English, or only in Turkish?
Each model's first English run against its Text tab run on the 547 kept questions: the questions only one of the two got right, by the exact McNemar test with p adjusted by Holm's method over the 17 models · the longer bar is the language with more right answers
Right only in English Right only in Turkish * A significant difference (Holm-adjusted p below 0.05): Microsoft-Decision-1 (p < 0.0001). The other 16: p = 0.068, p = 0.11, p = 0.46, p = 0.46, p = 0.46, p = 0.46, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1, p = 1 and p = 1.
Liquid d1-ultra against each model in English
In English, against Liquid d1-ultra, eight are too close to call and eight are behind it (of the 16 other models, after Holm's correction).
| Model | Runfirst English run | Only d1-ultra rightthe other not right | Only the other rightd1-ultra not right | pexact McNemar | Holm pover the 16 | Verdictby Holm p, 0.05 |
|---|---|---|---|---|---|---|
| Fastino GLiDE v1.2local · 1 question per request | run 521 | 27 | 25 | 0.89 | 1 | too close to call |
| Mammoth-11 question per request | run 503 | 23 | 20 | 0.76 | 1 | too close to call |
| vLLM-SR d31 question per request | run 554 | 23 | 18 | 0.53 | 1 | too close to call |
| Drex Max v11 question per request | run 552 | 27 | 21 | 0.47 | 1 | too close to call |
| vLLM-SR d3-ultratransformers, rev 52ba9e7 · 1 question per request | run 550 | 31 | 24 | 0.42 | 1 | too close to call |
| Drex Max 1.11 question per request | run 399 | 36 | 24 | 0.16 | 1 | too close to call |
| Perplexity Decider 27B v1.2local · 1 question per request | run 527 | 29 | 18 | 0.14 | 1 | too close to call |
| Liquid d11 question per request | run 430 | 40 | 19 | 0.0086 | 0.069 | too close to call |
| Jev 1.14.01 question per request | run 410 | 44 | 19 | 0.0022 | 0.02 | behind d1-ultra |
| Inception Mercury Decidevia OpenRouter · 1 question per request | run 398 | 46 | 19 | 0.0011 | 0.012 | behind d1-ultra |
| Microsoft-Decision-1via OpenRouter · 1 question per request | run 412 | 46 | 19 | 0.0011 | 0.012 | behind d1-ultra |
| SubQ Snap 1.11 question per request | run 413 | 42 | 16 | 0.00086 | 0.01 | behind d1-ultra |
| Elephant-11 question per request | run 499 | 49 | 20 | 0.00064 | 0.0083 | behind d1-ultra |
| GPT-6 Luna (Decisions API)Decisions API, beta · 1 question per request | run 397 | 54 | 18 | < 0.0001 | 0.00036 | behind d1-ultra |
| Celeris-1 Decision1 question per request | run 546 | 50 | 14 | < 0.0001 | 0.00011 | behind d1-ultra |
| Perplexity Decider Flash v1.2local · 1 question per request | run 533 | 64 | 16 | < 0.0001 | < 0.0001 | behind d1-ultra |
Ordered by p, highest first. On the 547 kept reading questions, Liquid d1-ultra's first English run (run 411) against each other model's first English run: the exact McNemar test on the questions only one of the two got right, with p adjusted by Holm's method over the 16 models. Verdict: too close to call when the Holm-adjusted p is 0.05 or more; otherwise behind d1-ultra when Liquid d1-ultra has more questions only it got right, ahead of it when the other model has more. Each model's name links its Text tab run, as in the tables above; Paired test compares each model with itself in Turkish.
With an “I don't know” option
The same 547 kept questions with a sixth option, “I don't know” in English and “Bilmiyorum” in Turkish, as on the Hallucination tab, asked of the same models (how).
Eleven of the 13 models had a higher non-hallucination rate in English than in Turkish; two lower; one of the 13 differences in wrong answers is significant after Holm's correction: Perplexity Decider Flash v1.2 (more wrong answers in Turkish) (Paired test). Celeris-1 Decision has an English rate of 71.3% (92 “I don't know” of 129 not right) with an attempt rate of 83.2% (455 answered) but no Turkish run with “Bilmiyorum” on these questions, so no comparison or test. Read each rate with its attempt rate: a model that always chose the option would reach 100%.
With the option written “I don't know” in Turkish too, so that only the language differs, the comparison is in the next section.
Non-hallucination rate in Turkish and in English
The option among the answers that were not right, on the 547 kept questions: each model's first run in each language, ordered by the English rate · higher is better, read with the attempt rate (under the chart and in each bar's tooltip)
Turkish, with “Bilmiyorum”, the upper bar English, with “I don't know”, the lower bar Attempt rate, Turkish · English: Liquid d1-ultra 94.5% · 90.7%; Drex Max 1.1 98.4% · 92.1%; Mammoth-1 95.6% · 93.2%; Fastino GLiDE v1.2 98.9% · 94.5%; Elephant-1 97.3% · 95.8%; SubQ Snap 1.1 96.5% · 96.2%; Inception Mercury Decide 98.5% · 98.0%; GPT-6 Luna (Decisions API) 98.9% · 98.2%; Perplexity Decider Flash v1.2 98.9% · 99.1%; Jev 1.14.0 99.5% · 99.5%; Microsoft-Decision-1 99.6% · 99.5%; Perplexity Decider 27B v1.2 100.0% · 99.6%; Liquid d1 99.8% · 99.6%. A model that always chose the option would reach 100% with no right answers. Not charted, with no Turkish run with “Bilmiyorum” on these questions: Celeris-1 Decision (in the table under Info).
Wrong only in English, or only in Turkish?
Each model's first run in each language, both with the option, on the 547 kept questions: the questions only one of the two got wrong, by the exact McNemar test with p adjusted by Holm's method over the 13 models · the shorter bar is the language with fewer wrong answers
Wrong only in English Wrong only in Turkish * A significant difference (Holm-adjusted p below 0.05): Perplexity Decider Flash v1.2 (p = 0.022). The other 12: p = 0.098, p = 0.21, p = 0.49, p = 0.49, p = 0.88, p = 0.88, p = 1, p = 1, p = 1, p = 1, p = 1 and p = 1. Not charted, with no Turkish run with “Bilmiyorum” on these questions: Celeris-1 Decision (in the table under Info).
Only the language differs: “I don't know” in both
The same 547 kept questions with the sixth option written “I don't know” in English and in Turkish too, with the same sentence added to the instructions: each model's first English run with the option, as in the section above, against its Turkish run from the option's word check of the Hallucination tab, counted on these questions. Only the language differs: of the question and its options, of the word in the instructions that names it and of the options heading (how).
Seven of the 17 models have both runs. Among them, wrong of answers was lower in English for three and higher for four; the non-hallucination rate was lower in English for one and higher for six. None of the seven differences in wrong answers is significant after Holm's correction over the seven, so each could be chance; the smallest adjusted p is SubQ Snap 1.1's, p = 0.21 (Paired test). Turkish only, with no English run with “I don't know”: vLLM-SR d3, Drex Max v1 and vLLM-SR d3-ultra. English only, with no Turkish run with the option written “I don't know”: Liquid d1, Jev 1.14.0, Inception Mercury Decide, Microsoft-Decision-1, Celeris-1 Decision, GPT-6 Luna (Decisions API) and Perplexity Decider Flash v1.2 (in the main table and the table under Info, with no test).
Wrong of answers in Turkish and in English
Wrong / (right + wrong), with the option written “I don't know” in both languages, on the 547 kept questions: each model's first run in each language, ordered by the Turkish share, lowest first · lower is better, read with the answered share (under the chart and in each bar's tooltip)
Turkish, with “I don't know”, the upper bar English, with “I don't know”, the lower bar Answered, Turkish · English: Liquid d1-ultra 95.1% · 90.7%; Mammoth-1 95.4% · 93.2%; SubQ Snap 1.1 96.5% · 96.2%; Fastino GLiDE v1.2 98.4% · 94.5%; Drex Max 1.1 96.7% · 92.1%; Perplexity Decider 27B v1.2 99.8% · 99.6%; Elephant-1 95.6% · 95.8%. A model that answered fewer questions can have fewer wrong answers among them. Not charted, with one side only: vLLM-SR d3, Drex Max v1, vLLM-SR d3-ultra, Liquid d1, Jev 1.14.0, Inception Mercury Decide, Microsoft-Decision-1, Celeris-1 Decision, GPT-6 Luna (Decisions API) and Perplexity Decider Flash v1.2 (in the table under Info).
Wrong only in English, or only in Turkish?
Each model's first run in each language, both with the option written “I don't know”, on the 547 kept questions: the questions only one of the two got wrong, by the exact McNemar test with p adjusted by Holm's method over the 7 models with both runs · the shorter bar is the language with fewer wrong answers
Wrong only in English Wrong only in Turkish No difference is significant (Holm-adjusted p below 0.05). Adjusted: p = 0.21, p = 0.26, p = 0.36, p = 0.44, p = 0.44, p = 0.76 and p = 0.76. Not charted, with one side only: vLLM-SR d3, Drex Max v1, vLLM-SR d3-ultra, Liquid d1, Jev 1.14.0, Inception Mercury Decide, Microsoft-Decision-1, Celeris-1 Decision, GPT-6 Luna (Decisions API) and Perplexity Decider Flash v1.2 (in the table under Info).
The option's word: “I don't know” instead of “Bilmiyorum”
The Hallucination tab's 2,198 Turkish questions, with the sixth option “Bilmiyorum” and again with “I don't know”, and only the sentence that names it changed: a model that skipped “Bilmiyorum” because it did not recognise the Turkish word should choose “I don't know” more often. Each of the 11 models' first run with each word, in the Hallucination tab's order; the full table and method are on the Hallucination tab.
| Model | Answered with “Bilmiyorum”(right + wrong) / questions; higher is better | Answered with “I don't know”(right + wrong) / questions; higher is better | Wrong of answers with “Bilmiyorum”wrong / (right + wrong); lower is better | Wrong of answers with “I don't know”wrong / (right + wrong); lower is better | Non-hallucination with “Bilmiyorum”“Bilmiyorum” / (“Bilmiyorum” + wrong); higher is better | Non-hallucination with “I don't know”“I don't know” / (“I don't know” + wrong); higher is better | Difference“I don't know” minus “Bilmiyorum” in non-hallucination, in points; positive = higher with the English word | Holm-adjusted pexact McNemar test of wrong answers |
|---|---|---|---|---|---|---|---|---|
| Liquid d1-ultra | 71.9%1,581 of 2,198 answered | 72.0%1,582 of 2,198 answered | 13.0%205 of 1,581 | 13.7%216 of 1,582 | 75.1%1,376 right, 205 wrong, 617 “Bilmiyorum” | 74.0%1,366 right, 216 wrong, 616 “I don't know” | -1.0 | p = 1 |
| Mammoth-1 | 68.6%1,508 of 2,198 answered | 65.7%1,445 of 2,198 answered | 20.8%314 of 1,508 | 19.1%276 of 1,445 | 68.7%1,194 right, 314 wrong, 690 “Bilmiyorum” | 73.2%1,169 right, 276 wrong, 753 “I don't know” | +4.5 * | p < 0.0001 |
| Fastino GLiDE | 76.3%1,677 of 2,198 answered | 73.6%1,617 of 2,198 answered | 21.2%356 of 1,677 | 20.7%335 of 1,617 | 59.4%1,321 right, 356 wrong, 521 “Bilmiyorum” | 63.4%1,282 right, 335 wrong, 581 “I don't know” | +4.0 | p = 0.44 |
| SubQ Snap 1.1 | 80.1%1,761 of 2,198 answered | 77.3%1,700 of 2,198 answered | 27.8%490 of 1,761 | 26.7%454 of 1,700 | 47.1%1,271 right, 490 wrong, 437 “Bilmiyorum” | 52.3%1,246 right, 454 wrong, 498 “I don't know” | +5.2 | p = 0.19 |
| Fastino GLiDE v1.2 | 85.4%1,877 of 2,198 answered | 80.6%1,772 of 2,198 answered | 26.2%491 of 1,877 | 24.1%427 of 1,772 | 39.5%1,386 right, 491 wrong, 321 “Bilmiyorum” | 49.9%1,345 right, 427 wrong, 426 “I don't know” | +10.4 * | p < 0.0001 |
| Drex Max 1.1 | 85.4%1,878 of 2,198 answered | 77.9%1,712 of 2,198 answered | 27.5%517 of 1,878 | 24.1%413 of 1,712 | 38.2%1,361 right, 517 wrong, 320 “Bilmiyorum” | 54.1%1,299 right, 413 wrong, 486 “I don't know” | +15.8 * | p < 0.0001 |
| Elephant-1 | 81.6%1,793 of 2,198 answered | 72.5%1,594 of 2,198 answered | 42.3%758 of 1,793 | 39.0%622 of 1,594 | 34.8%1,035 right, 758 wrong, 405 “Bilmiyorum” | 49.3%972 right, 622 wrong, 604 “I don't know” | +14.4 * | p < 0.0001 |
| vLLM-SR d3-ultra | 99.5%2,188 of 2,198 answered | 99.6%2,189 of 2,198 answered | 32.2%704 of 2,188 | 32.2%704 of 2,189 | 1.4%1,484 right, 704 wrong, 10 “Bilmiyorum” | 1.3%1,485 right, 704 wrong, 9 “I don't know” | -0.1 | p = 1 |
| Drex Max v1 | 99.9%2,196 of 2,198 answered | 99.9%2,196 of 2,198 answered | 32.4%711 of 2,196 | 32.4%712 of 2,196 | 0.3%1,485 right, 711 wrong, 2 “Bilmiyorum” | 0.3%1,484 right, 712 wrong, 2 “I don't know” | -0.0 | p = 1 |
| Perplexity Decider 27B v1.2 | 99.9%2,196 of 2,198 answered | 99.8%2,194 of 2,198 answered | 32.7%717 of 2,196 | 33.2%728 of 2,194 | 0.3%1,479 right, 717 wrong, 2 “Bilmiyorum” | 0.5%1,466 right, 728 wrong, 4 “I don't know” | +0.3 | p = 1 |
| vLLM-SR d3 | 100.0%2,198 of 2,198 answered | 99.9%2,196 of 2,198 answered | 32.5%715 of 2,198 | 32.5%714 of 2,196 | 0.0%1,483 right, 715 wrong, 0 “Bilmiyorum” | 0.3%1,482 right, 714 wrong, 2 “I don't know” | +0.3 | p = 1 |
Answered: (right + wrong) / 2,198 questions. Wrong of answers: wrong / (right + wrong). Both are worked out from the published counts; under each non-hallucination rate, of the 2,198 questions, the right answers, the wrong ones and the option. * marks a difference whose paired test (the exact McNemar test on the questions only one of the two runs got wrong, with p adjusted by Holm's method over the models) has a Holm-adjusted p below 0.05.
Risk and coverage
How often each model answers and how often its answers are wrong, on the same 547 kept questions in both languages: with the option, as the model chose it, and without it, by holding back the answers below a confidence score.
Method
Answered = right + wrong (an invalid answer or one without a score is never answered). Coverage = answered / 547. Error rate among answered (risk) = wrong / answered. Abstained: for a run with the option, the times it chose the option; for a confidence point, 547 minus answered minus invalid. Each confidence curve has one point per distinct score of the run on these questions (answer when the score, the recorded probability of the chosen option, is at least that score): answers with equal scores enter together, so coverage moves in jumps and every point is one the run achieved; a model that reports its probabilities with few decimals has many equal scores, so its curve has fewer points, moves in larger steps and can start well above 0% coverage (each curve is drawn from its first point). Every series counts these questions only (a run on the full bank is restricted to them) and is each model's first run. Turkish curves: the Text tab run; English curves: the first English run. Turkish with “Bilmiyorum” is the Turkish side of the English “I don't know” comparison or, for a model outside it, its Hallucination tab run, restricted to these; Turkish with “I don't know” is the option's word control on the Hallucination tab's questions, restricted to these; a model without such a run is Not measured.
All series are counted on the same kept reading questions; a full-bank run is restricted to them; first runs only. Answered = right + wrong (an invalid answer or one without a score is never answered). Coverage = answered / kept questions. Risk = error rate among answered = wrong / answered. Abstained: for a native point, the times the model chose the option; for a curve point, kept questions - answered - invalid. A native point is the run with its abstention option (Turkish "Bilmiyorum", Turkish "I don't know" as the wording control, English "I don't know"); Turkish "Bilmiyorum" is the Turkish side of the English "I don't know" comparison or, for a model outside it, its Hallucination tab run; a missing one is Not measured. A curve is one five-option run held back by its recorded score (the probability of the chosen option): one point per distinct score s, answering when the score is at least s; answers with equal scores enter together, so coverage moves in jumps and every point is an achieved one; all distinct points kept, ordered by coverage.
| Model | Turkish with “Bilmiyorum”coverage · error rate among answered | Turkish with “I don't know”coverage · error rate among answered | English with “I don't know”coverage · error rate among answered |
|---|---|---|---|
| Liquid d1-ultra | 94.5% · 5.0%491 right, 26 wrong, 30 abstained · run 350, full bank restricted | 95.1% · 5.4%492 right, 28 wrong, 27 abstained · run 511, full bank restricted | 90.7% · 7.9%457 right, 39 wrong, 51 abstained · run 449 |
| Fastino GLiDE v1.2 | 98.9% · 8.7%494 right, 47 wrong, 6 abstained · run 369, full bank restricted | 98.4% · 8.7%491 right, 47 wrong, 9 abstained · run 525, full bank restricted | 94.5% · 7.0%481 right, 36 wrong, 30 abstained · run 523 |
| Mammoth-1 | 95.6% · 6.3%490 right, 33 wrong, 24 abstained · run 386, full bank restricted | 95.4% · 6.1%490 right, 32 wrong, 25 abstained · run 519, full bank restricted | 93.2% · 7.1%474 right, 36 wrong, 37 abstained · run 505 |
| vLLM-SR d3 | 100.0% · 9.7%494 right, 53 wrong, 0 abstained · run 358, full bank restricted | 100.0% · 9.5%495 right, 52 wrong, 0 abstained · run 544, full bank restricted | Not measured |
| Drex Max v1 | 99.6% · 9.7%492 right, 53 wrong, 2 abstained · run 367, full bank restricted | 99.8% · 9.0%497 right, 49 wrong, 1 abstained · run 542, full bank restricted | Not measured |
| vLLM-SR d3-ultra | 99.8% · 8.4%500 right, 46 wrong, 1 abstained · run 363, full bank restricted | 99.6% · 9.4%494 right, 51 wrong, 2 abstained · run 540, full bank restricted | Not measured |
| Perplexity Decider 27B v1.2 | 100.0% · 9.1%497 right, 50 wrong, 0 abstained · run 372, full bank restricted | 99.8% · 9.3%495 right, 51 wrong, 1 abstained · run 531, full bank restricted | 99.6% · 11.7%481 right, 64 wrong, 2 abstained · run 529 |
| Drex Max 1.1 | 98.4% · 9.5%487 right, 51 wrong, 9 abstained · run 352, full bank restricted | 96.7% · 9.1%481 right, 48 wrong, 18 abstained · run 513, full bank restricted | 92.1% · 8.1%463 right, 41 wrong, 43 abstained · run 447 |
| Liquid d1 | 99.8% · 16.3%457 right, 89 wrong, 1 abstained · run 472 | Not measured | 99.6% · 12.3%478 right, 67 wrong, 2 abstained · run 445 |
| Jev 1.14.0 | 99.5% · 12.7%475 right, 69 wrong, 3 abstained · run 470 | Not measured | 99.5% · 13.2%472 right, 72 wrong, 3 abstained · run 446 |
| SubQ Snap 1.1 | 96.5% · 9.7%477 right, 51 wrong, 19 abstained · run 392, full bank restricted | 96.5% · 8.5%483 right, 45 wrong, 19 abstained · run 512, full bank restricted | 96.2% · 12.0%463 right, 63 wrong, 21 abstained · run 448 |
| Inception Mercury Decide | 98.5% · 13.2%468 right, 71 wrong, 8 abstained · run 489 | Not measured | 98.0% · 13.2%465 right, 71 wrong, 11 abstained · run 450 |
| Microsoft-Decision-1 | 99.6% · 16.1%457 right, 88 wrong, 2 abstained · run 487 | Not measured | 99.5% · 14.2%467 right, 77 wrong, 3 abstained · run 451 |
| Elephant-1 | 97.3% · 15.4%450 right, 82 wrong, 15 abstained · run 382, full bank restricted | 95.6% · 15.7%441 right, 82 wrong, 24 abstained · run 515, full bank restricted | 95.8% · 12.8%457 right, 67 wrong, 23 abstained · run 501 |
| Celeris-1 Decision | Not measured | Not measured | 83.2% · 8.1%418 right, 37 wrong, 92 abstained · run 548 |
| GPT-6 Luna (Decisions API) | 98.9% · 14.4%463 right, 78 wrong, 6 abstained · run 453 | Not measured | 98.2% · 17.9%441 right, 96 wrong, 10 abstained · run 444 |
| Perplexity Decider Flash v1.2 | 98.9% · 24.6%408 right, 133 wrong, 6 abstained · run 537 | Not measured | 99.1% · 18.5%442 right, 100 wrong, 5 abstained · run 535 |
Each cell: coverage · error rate among answered, then the counts and the run.
Risk and coverage in Turkish
The 547 kept questions in Turkish: each model's five-option run answering only at or above a confidence score (the line), and its run with the option (the marker: “Bilmiyorum” “I don't know”) · lower is better at the same coverage
- Liquid d1-ultra: confidence, run 199 · “Bilmiyorum”, run 350 · “I don't know”, run 511
- Fastino GLiDE v1.2: confidence, run 248 · “Bilmiyorum”, run 369 · “I don't know”, run 525
- Mammoth-1: confidence, run 374 · “Bilmiyorum”, run 386 · “I don't know”, run 519
- vLLM-SR d3: confidence, run 259 · “Bilmiyorum”, run 358 · “I don't know”, run 544
- Drex Max v1: confidence, run 322 · “Bilmiyorum”, run 367 · “I don't know”, run 542
- vLLM-SR d3-ultra: confidence, run 269 · “Bilmiyorum”, run 363 · “I don't know”, run 540
- Perplexity Decider 27B v1.2: confidence, run 217 · “Bilmiyorum”, run 372 · “I don't know”, run 531
- Drex Max 1.1: confidence, run 326 · “Bilmiyorum”, run 352 · “I don't know”, run 513
- Liquid d1: confidence, run 52 · “Bilmiyorum”, run 472
- Jev 1.14.0: confidence, run 194 · “Bilmiyorum”, run 470
- SubQ Snap 1.1: confidence, run 346 · “Bilmiyorum”, run 392 · “I don't know”, run 512
- Inception Mercury Decide: confidence, run 204 · “Bilmiyorum”, run 489
- Microsoft-Decision-1: confidence, run 205 · “Bilmiyorum”, run 487
- Elephant-1: confidence, run 373 · “Bilmiyorum”, run 382 · “I don't know”, run 515
- Celeris-1 Decision: confidence, run 150
- GPT-6 Luna (Decisions API): confidence, run 143 · “Bilmiyorum”, run 453
- Perplexity Decider Flash v1.2: confidence, run 223 · “Bilmiyorum”, run 537
Risk and coverage in English
The 547 kept questions in English: each model's five-option run answering only at or above a confidence score (the line), and its run with the option (the marker: “I don't know”) · lower is better at the same coverage
- Liquid d1-ultra: confidence, run 411 · “I don't know”, run 449
- Fastino GLiDE v1.2: confidence, run 521 · “I don't know”, run 523
- Mammoth-1: confidence, run 503 · “I don't know”, run 505
- vLLM-SR d3: confidence, run 554
- Drex Max v1: confidence, run 552
- vLLM-SR d3-ultra: confidence, run 550
- Perplexity Decider 27B v1.2: confidence, run 527 · “I don't know”, run 529
- Drex Max 1.1: confidence, run 399 · “I don't know”, run 447
- Liquid d1: confidence, run 430 · “I don't know”, run 445
- Jev 1.14.0: confidence, run 410 · “I don't know”, run 446
- SubQ Snap 1.1: confidence, run 413 · “I don't know”, run 448
- Inception Mercury Decide: confidence, run 398 · “I don't know”, run 450
- Microsoft-Decision-1: confidence, run 412 · “I don't know”, run 451
- Elephant-1: confidence, run 499 · “I don't know”, run 501
- Celeris-1 Decision: confidence, run 546 · “I don't know”, run 548
- GPT-6 Luna (Decisions API): confidence, run 397 · “I don't know”, run 444
- Perplexity Decider Flash v1.2: confidence, run 533 · “I don't know”, run 535
Info The table, the models not compared and our method, then the same with “I don't know”
About this test
The 13 decision models of Artificial Analysis's Decision Index (its post of 22 October 2026) that we benchmark, and the ten most accurate decision models on the Text tab on 2 November 2026, seven of them not in that index: 20 in all. 17 of them are compared here; the other three are listed under the results with their reasons.
The Text tab's reading questions (units 1 to 6; the grammar questions are left out) were translated into English by Claude Opus 5.5. The same model flagged the questions whose answer depends on Turkish wording; the 109 flagged questions are left out and the other 547 kept. Each of the 17 compared models answered the kept questions in English and is compared with its own Text tab run on the same 547 questions.
Download: results.json (the comparison's numbers, under english_reading), answers.json (each English run's answers, under english_runs), english-reading.json (the translator, its prompts, the flags, the question units and each question's hash of its English text; no question text) and kept-question-ids.txt (the 547 kept questions' IDs, one per line).
Results
Ordered by accuracy in English, highest first. Each row compares the model's first English run with its Text tab run, out of the 547 kept questions; an invalid answer counts as not right. Where a side has more than one run, its accuracy shows the range over its runs and its runs show how many answers changed between them. The paired counts, p and the Holm-adjusted p (over the 17 models) are those of the exact McNemar test; * marks a Holm-adjusted p below 0.05.
| Model | English runsthe first is compared; answers changed between runs where more than one | Turkish runsthe Text tab run, then its repeats; answers changed between them | Turkish accuracythe Text tab run on the kept questions; range over its runs; higher is better | English accuracythe first English run; range over its runs; higher is better | DifferenceEnglish minus Turkish, in points | Right only in Englishquestions the English run got right and the Turkish run did not | Right only in Turkishquestions the Turkish run got right and the English run did not | pexact McNemar | Holm padjusted over the 17 models |
|---|---|---|---|---|---|---|---|---|---|
| Liquid d1-ultra1 question per request | runs 411, 418, 422, 426, 42828 answers changed between runs | runs 199 to 20318 answers changed between runs | 93.6%512 of 547 rightrange 92.7% to 93.6% | 90.7%496 of 547 rightrange 90.1% to 91.0% | -2.9 | 8 | 24 | 0.007 | 0.11 |
| Fastino GLiDE v1.2local · 1 question per request | run 521 | runs 248, 250 to 2530 answers changed between runs | 90.5%495 of 547 rightthe same in every run | 90.3%494 of 547 right | -0.2 | 21 | 22 | 1 | 1 |
| Mammoth-11 question per request | run 503 | runs 374, 377, 380, 383 and 3840 answers changed between runs | 92.9%508 of 547 rightthe same in every run | 90.1%493 of 547 right | -2.7 | 15 | 30 | 0.036 | 0.46 |
| vLLM-SR d31 question per request | run 554 | runs 259 to 2630 answers changed between runs | 91.0%498 of 547 rightthe same in every run | 89.8%491 of 547 right | -1.3 | 15 | 22 | 0.32 | 1 |
| Drex Max v11 question per request | run 552 | runs 322 to 324, 327 and 3280 answers changed between runs | 91.4%500 of 547 rightthe same in every run | 89.6%490 of 547 right | -1.8 | 19 | 29 | 0.19 | 1 |
| vLLM-SR d3-ultratransformers, rev 52ba9e7 · 1 question per request | run 550 | runs 269 to 2730 answers changed between runs | 91.4%500 of 547 rightthe same in every run | 89.4%489 of 547 right | -2.0 | 22 | 33 | 0.18 | 1 |
| Perplexity Decider 27B v1.2local · 1 question per request | run 527 | runs 217 to 2210 answers changed between runs | 91.4%500 of 547 rightthe same in every run | 88.7%485 of 547 right | -2.7 | 15 | 30 | 0.036 | 0.46 |
| Drex Max 1.11 question per request | run 399 | runs 326, 329 to 3320 answers changed between runs | 90.1%493 of 547 rightthe same in every run | 88.5%484 of 547 right | -1.6 | 18 | 27 | 0.23 | 1 |
| Liquid d11 question per request | runs 430, 432 to 43550 answers changed between runs | runs 52, 66, 94, 105, 10819 answers changed between runs | 85.0%465 of 547 rightrange 85.0% to 85.7% | 86.8%475 of 547 rightrange 85.7% to 87.9% | +1.8 | 36 | 26 | 0.25 | 1 |
| Jev 1.14.01 question per request | runs 410, 416, 421, 423, 42514 answers changed between runs | runs 194 to 19817 answers changed between runs | 87.2%477 of 547 rightrange 86.5% to 87.2% | 86.1%471 of 547 rightrange 86.1% to 86.5% | -1.1 | 17 | 23 | 0.43 | 1 |
| SubQ Snap 1.11 question per request | run 413 | runs 346, 387 to 3900 answers changed between runs | 89.2%488 of 547 rightthe same in every run | 85.9%470 of 547 right | -3.3 | 23 | 41 | 0.033 | 0.46 |
| Inception Mercury Decidevia OpenRouter · 1 question per request | runs 398, 400 to 40320 answers changed between runs | runs 204, 210, 212, 214, 21619 answers changed between runs | 87.0%476 of 547 rightrange 86.7% to 87.4% | 85.7%469 of 547 rightrange 85.0% to 85.9% | -1.3 | 28 | 35 | 0.45 | 1 |
| Microsoft-Decision-1via OpenRouter · 1 question per request | runs 412, 420, 424, 427, 43119 answers changed between runs | runs 205, 209, 211, 213, 21536 answers changed between runs | 76.4%418 of 547 rightrange 76.2% to 77.1% | 85.7%469 of 547 rightrange 85.0% to 86.1% | +9.3 | 74 | 23 | < 0.0001 | < 0.0001 * |
| Elephant-11 question per request | run 499 | runs 373, 375 and 376, 378 and 3790 answers changed between runs | 83.5%457 of 547 rightthe same in every run | 85.4%467 of 547 right | +1.8 | 39 | 29 | 0.27 | 1 |
| Celeris-1 Decision1 question per request | run 546 | runs 150 and 151, 156 to 15819 answers changed between runs | 87.4%478 of 547 rightrange 86.7% to 87.4% | 84.1%460 of 547 right | -3.3 | 24 | 42 | 0.036 | 0.46 |
| GPT-6 Luna (Decisions API)Decisions API, beta · 1 question per request | run 397 | run 143 | 86.7%474 of 547 right, 1 invalid | 84.1%460 of 547 right, 2 invalid | -2.6 | 28 | 42 | 0.12 | 1 |
| Perplexity Decider Flash v1.2local · 1 question per request | run 533 | runs 223 to 2270 answers changed between runs | 76.6%419 of 547 rightthe same in every run | 81.9%448 of 547 right | +5.3 | 63 | 34 | 0.0042 | 0.068 |
Not compared
- Fastino GLiDE: since 10 October, our requests for fastino/GLiDE have been answered by Fastino's glide_v2 engine, not the glide engine of its Turkish run (6 October), so an English run would not measure the same model.
Not run
- Perplexity Decider 27B v1.1: Perplexity's API now answers requests for v1.1 with v1.2, so a new run could not measure v1.1, and no Perplexity API key is available.
- Clef: Clef's own model code rounds each option's probability to 4 decimal places, so its English smoke run (538) failed our probability check, which needs the probabilities to add up to 1 to 7 decimal places; an allowance for that rounding, like the ones for Drex 2.0 and Base Labs' models, awaits a decision.
Our method
The translation
The translator is Claude Opus 5.5 (claude-opus-5-5) at effort high; route: Claude Code CLI, Claude subscription, Claude Code version 2.1.312. Its maker has no decision model on our board. It was told to translate faithfully, keep the option letters and their order, not solve the question and not add or remove information (translation prompt version 2976d0f96197).
The flags
In a separate request, the same model was shown each Turkish question, its English translation and the keyed option, and asked whether answering it correctly depends on Turkish wording, with the reasons (flag prompt version 08a5c70222aa). It flagged 109 of the 656 questions; they are left out, and 547 are kept. By reason, as a flagged question can have several:
- word or phrase meaning: 76
- idiom: 44
- sound: 5
- marked words: 55
- other: 13
No person checked the translations or the flags, and this is a limit of the method: a translation error, or a question whose answer still depends on Turkish wording that the model did not flag, would stay among the 547 kept questions and could move a model's accuracy in English. Both prompts in full are in english-reading.json and runner/translate_reading.py.
The structural check
A script (runner/translate_reading.py, its check step) then compared each translation with its Turkish original: a problem is a missing translation, option letters other than the original's, an empty text or a different count of underline markup; a warning is a change in the numbering or the numbers, Turkish letters in the English, or an English text under half or over 2.5 times the Turkish length. It found problems in 1 question (3043), already left out by the flags, so it left out no further question; its 159 warnings leave out nothing. In all, the checks left out 109 of the 656 questions and kept 547: their IDs are in kept-question-ids.txt, one per line, and in english-reading.json.
Why the grammar questions are not translated
The Text tab's 1,542 grammar questions test Turkish grammar itself, so a translation would test something else, not the same question in another language. Only the 656 reading questions were translated.
Asking the decision models
Each model is asked exactly as on the Text tab, with its own settings, one question per request, and with its option probabilities, except for one word of the System One instructions, the options heading and the question itself. The Text tab's instructions:
Which option is the correct answer to this Turkish multiple-choice question?
The English runs' instructions:
Which option is the correct answer to this English multiple-choice question?
The options follow the heading “Options:” instead of “Seçenekler:”, and the question and its options are the English translation. Prompt version 47f5776e71db; every English run used the translations with SHA-256 bf0d47cd385a93050c65906977e199ef7e45d5848dbbdef2bc8d124f4d5abf68. Liquid d1 was asked as d1, while its Turkish runs used d1:free, because Liquid's API no longer serves that name; its model list still names d1, released 29 September 2026. Perplexity Decider 27B v1.2, Perplexity Decider Flash v1.2 and Fastino GLiDE v1.2 made their Text tab runs on an NVIDIA DGX Spark and their English runs on our RTX PRO 6000 GPU, with the same checkpoint, server code and settings. Celeris-1 Decision was added on 3 November 2026 with one English run, although its repeated Turkish runs changed answers, for which the rule under Runs gives five.
Runs
One English run per model; five where the model's repeated Turkish runs on the Text tab changed answers. Here: five runs for Liquid d1-ultra, Liquid d1, Jev 1.14.0, Inception Mercury Decide and Microsoft-Decision-1; one for the other 12. The English runs are kept out of the leaderboard and its repeats.
The comparison
On the 547 kept questions only: accuracy is the right answers over 547 for the first English run and for the Text tab run (an invalid answer counts as not right), and the difference is English minus Turkish. The paired test is the exact McNemar test on the questions only one of the two runs got right, with p adjusted by Holm's method over the 17 models compared. Where a side has more than one run, its range is the lowest and highest accuracy over its runs, and its answers changed are the kept questions whose answer differs between them.
Why the translations are not published
The site never publishes question text, Turkish or English: the publisher owns the questions. english-reading.json holds instead each reading question's ID, unit, whether it was kept, its flag reasons and the SHA-256 of its English text, with the translator and both prompts, so that a copy of these translations can be checked against it. Translating again with runner/translate_reading.py need not give the same text word for word.
The instructions of each kind of run
Each run's System One instructions in full, as results.json and answers.json give them. The question and its options follow in the run's language; their text is not published.
Five options, in Turkish (the Text tab)
Which option is the correct answer to this Turkish multiple-choice question?
Five options, in English
Which option is the correct answer to this English multiple-choice question?
With the option, in Turkish (“Bilmiyorum”, the Hallucination tab)
Which option is the correct answer to this Turkish multiple-choice question? If you do not know the answer, choose Bilmiyorum.
With the option, in English (“I don't know”)
Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.
The option's word: the Turkish questions with “I don't know”
Which option is the correct answer to this Turkish multiple-choice question? If you do not know the answer, choose I don't know.
Results with an “I don't know” option
Ordered by the non-hallucination rate in English, highest first. Each row compares the model's first run with “I don't know” in English with its first run with “Bilmiyorum” in Turkish, out of the 547 kept questions. Where a side has more than one run, its rates show the range over its runs. The paired counts, p and the Holm-adjusted p (over the 13 models) are those of the exact McNemar test on wrong answers; * marks a Holm-adjusted p below 0.05.
| Model | English runswith “I don't know”; the first is compared | Turkish runswith “Bilmiyorum”; the first is compared | English answered(right + wrong) / 547; higher is better | Turkish answered(right + wrong) / 547; higher is better | English wrong of answerswrong / (right + wrong); lower is better | Turkish wrong of answerswrong / (right + wrong); lower is better | English non-hallucination“I don't know” / (“I don't know” + wrong); higher is better, read with the answered share | Turkish non-hallucination“Bilmiyorum” / (“Bilmiyorum” + wrong); higher is better, read with the answered share | English accuracyhigher is better | Turkish accuracyhigher is better | English mean probabilityof “I don't know” | Turkish mean probabilityof “Bilmiyorum” | Wrong only in Englishquestions the English run got wrong and the Turkish run did not | Wrong only in Turkishquestions the Turkish run got wrong and the English run did not | pexact McNemar | Holm padjusted over the 13 models |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Celeris-1 Decision1 question per request | run 548a new run | Not measured | 83.2%455 of 547 answered, 92 “I don't know” | Not measured | 8.1%37 of 455 | Not measured | 71.3%92 “I don't know” of 129 not right | Not measured | 76.4%418 of 547 right | Not measured | 12.7% | Not measured | Not measured | Not measured | Not measured | Not measured |
| Liquid d1-ultra1 question per request | runs 449, 456, 462, 466, 474new runs | runs 350, 354, 360, 365, 370the Hallucination tab's full runs, counted on the kept questions | 90.7%496 of 547 answered, 51 “I don't know”range 89.6% to 91.4% | 94.5%517 of 547 answered, 30 “Bilmiyorum”range 94.1% to 95.6% | 7.9%39 of 496 | 5.0%26 of 517 | 56.7%51 “I don't know” of 90 not rightrange 55.3% to 61.3% | 53.6%30 “Bilmiyorum” of 56 not rightrange 44.4% to 59.3% | 83.5%457 of 547 rightrange 82.6% to 84.5% | 89.8%491 of 547 rightrange 89.2% to 90.1% | 25.3%range 25.3% to 25.6% | 22.8%range 22.4% to 23.0% | 20 | 7 | 0.019 | 0.21 |
| Drex Max 1.11 question per request | run 447a new run | run 352the Hallucination tab's full runs, counted on the kept questions | 92.1%504 of 547 answered, 43 “I don't know” | 98.4%538 of 547 answered, 9 “Bilmiyorum” | 8.1%41 of 504 | 9.5%51 of 538 | 51.2%43 “I don't know” of 84 not right | 15.0%9 “Bilmiyorum” of 60 not right | 84.6%463 of 547 right | 89.0%487 of 547 right | 14.3% | 8.4% | 20 | 30 | 0.2 | 1 |
| Mammoth-11 question per request | run 505a new run | run 386the Hallucination tab's full runs, counted on the kept questions | 93.2%510 of 547 answered, 37 “I don't know” | 95.6%523 of 547 answered, 24 “Bilmiyorum” | 7.1%36 of 510 | 6.3%33 of 523 | 50.7%37 “I don't know” of 73 not right | 42.1%24 “Bilmiyorum” of 57 not right | 86.7%474 of 547 right | 89.6%490 of 547 right | 9.8% | 7.4% | 18 | 15 | 0.73 | 1 |
| Fastino GLiDE v1.2local · 1 question per request | run 523a new run | run 369the Hallucination tab's full runs, counted on the kept questions | 94.5%517 of 547 answered, 30 “I don't know” | 98.9%541 of 547 answered, 6 “Bilmiyorum” | 7.0%36 of 517 | 8.7%47 of 541 | 45.5%30 “I don't know” of 66 not right | 11.3%6 “Bilmiyorum” of 53 not right | 87.9%481 of 547 right | 90.3%494 of 547 right | 12.5% | 8.7% | 16 | 27 | 0.13 | 0.88 |
| Elephant-11 question per request | run 501a new run | run 382the Hallucination tab's full runs, counted on the kept questions | 95.8%524 of 547 answered, 23 “I don't know” | 97.3%532 of 547 answered, 15 “Bilmiyorum” | 12.8%67 of 524 | 15.4%82 of 532 | 25.6%23 “I don't know” of 90 not right | 15.5%15 “Bilmiyorum” of 97 not right | 83.5%457 of 547 right | 82.3%450 of 547 right | 6.8% | 5.7% | 31 | 46 | 0.11 | 0.88 |
| SubQ Snap 1.11 question per request | run 448a new run | run 392the Hallucination tab's full runs, counted on the kept questions | 96.2%526 of 547 answered, 21 “I don't know” | 96.5%528 of 547 answered, 19 “Bilmiyorum” | 12.0%63 of 526 | 9.7%51 of 528 | 25.0%21 “I don't know” of 84 not right | 27.1%19 “Bilmiyorum” of 70 not right | 84.6%463 of 547 right | 87.2%477 of 547 right | 11.7% | 8.4% | 39 | 27 | 0.18 | 1 |
| Inception Mercury Decidevia OpenRouter · 1 question per request | runs 450, 460, 469, 478, 484new runs | runs 489, 491, 493, 495, 497new runs on the kept questions | 98.0%536 of 547 answered, 11 “I don't know”range 98.0% to 98.4% | 98.5%539 of 547 answered, 8 “Bilmiyorum”the same in every run | 13.2%71 of 536 | 13.2%71 of 539 | 13.4%11 “I don't know” of 82 not rightrange 11.1% to 13.4% | 10.1%8 “Bilmiyorum” of 79 not rightrange 10.1% to 10.5% | 85.0%465 of 547 rightrange 85.0% to 85.4% | 85.6%468 of 547 rightrange 85.6% to 86.1% | 3.2%range 3.1% to 3.2% | 1.9%range 1.9% to 2.0% | 29 | 29 | 1 | 1 |
| GPT-6 Luna (Decisions API)Decisions API, beta · 1 question per request | run 444a new run | run 453a new run on the kept questions | 98.2%537 of 547 answered, 10 “I don't know” | 98.9%541 of 547 answered, 6 “Bilmiyorum” | 17.9%96 of 537 | 14.4%78 of 541 | 9.4%10 “I don't know” of 106 not right | 7.1%6 “Bilmiyorum” of 84 not right | 80.6%441 of 547 right | 84.6%463 of 547 right | 3.0% | 2.7% | 47 | 29 | 0.05 | 0.49 |
| Perplexity Decider Flash v1.2local · 1 question per request | run 535a new run | run 537a new run on the kept questions | 99.1%542 of 547 answered, 5 “I don't know” | 98.9%541 of 547 answered, 6 “Bilmiyorum” | 18.5%100 of 542 | 24.6%133 of 541 | 4.8%5 “I don't know” of 105 not right | 4.3%6 “Bilmiyorum” of 139 not right | 80.8%442 of 547 right | 74.6%408 of 547 right | 5.5% | 6.3% | 36 | 69 | 0.0017 | 0.022 * |
| Jev 1.14.01 question per request | runs 446, 454, 457, 461, 464new runs | runs 470, 473, 476, 480, 482new runs on the kept questions | 99.5%544 of 547 answered, 3 “I don't know”range 99.3% to 99.5% | 99.5%544 of 547 answered, 3 “Bilmiyorum”the same in every run | 13.2%72 of 544 | 12.7%69 of 544 | 4.0%3 “I don't know” of 75 not rightrange 4.0% to 5.2% | 4.2%3 “Bilmiyorum” of 72 not rightrange 4.0% to 4.5% | 86.3%472 of 547 rightrange 85.9% to 86.7% | 86.8%475 of 547 rightrange 86.3% to 87.8% | 1.6%range 1.6% to 1.6% | 1.7%range 1.6% to 1.7% | 20 | 17 | 0.74 | 1 |
| Microsoft-Decision-1via OpenRouter · 1 question per request | runs 451, 459, 467, 477, 483new runs | runs 487, 490, 492, 494, 496new runs on the kept questions | 99.5%544 of 547 answered, 3 “I don't know”the same in every run | 99.6%545 of 547 answered, 2 “Bilmiyorum”range 99.3% to 99.6% | 14.2%77 of 544 | 16.1%88 of 545 | 3.8%3 “I don't know” of 80 not rightrange 3.8% to 4.0% | 2.2%2 “Bilmiyorum” of 90 not rightrange 2.2% to 4.3% | 85.4%467 of 547 rightrange 85.4% to 86.3% | 83.5%457 of 547 rightrange 82.8% to 83.5% | 3.6%range 3.6% to 3.6% | 2.3%range 2.3% to 2.3% | 24 | 35 | 0.19 | 1 |
| Perplexity Decider 27B v1.2local · 1 question per request | run 529a new run | run 372the Hallucination tab's full runs, counted on the kept questions | 99.6%545 of 547 answered, 2 “I don't know” | 100.0%547 of 547 answered, 0 “Bilmiyorum” | 11.7%64 of 545 | 9.1%50 of 547 | 3.0%2 “I don't know” of 66 not right | 0.0%0 “Bilmiyorum” of 50 not right | 87.9%481 of 547 right | 90.9%497 of 547 right | 2.8% | 0.9% | 29 | 15 | 0.049 | 0.49 |
| Liquid d11 question per request | runs 445, 455, 458, 463, 465new runs | runs 472, 475, 479, 481, 485new runs on the kept questions | 99.6%545 of 547 answered, 2 “I don't know”the same in every run | 99.8%546 of 547 answered, 1 “Bilmiyorum”range 99.8% to 100.0% | 12.3%67 of 545 | 16.3%89 of 546 | 2.9%2 “I don't know” of 69 not rightrange 2.6% to 2.9% | 1.1%1 “Bilmiyorum” of 90 not rightrange 0.0% to 1.1% | 87.4%478 of 547 rightrange 85.9% to 87.4% | 83.5%457 of 547 rightrange 83.0% to 85.0% | 0.9%range 0.9% to 1.0% | 1.3%range 1.1% to 1.3% | 21 | 43 | 0.0081 | 0.098 |
Our method with an “I don't know” option
The option
As on the Hallucination tab, each question gets a sixth option after its own five: F, “I don't know” in English, “Bilmiyorum” in Turkish. It is always the last option, never shuffled and never correct. The English reading instructions get one added sentence: “If you do not know the answer, choose I don't know.” The full instructions:
Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.
Prompt version 413d6775fc58. The question and its options are the English translation, as in the accuracy comparison above; no person checked the translations. The Turkish side uses the Hallucination tab's prompts unchanged.
The Turkish side
The Turkish side: the Hallucination tab's published full runs, counted on the kept questions only, for Liquid d1-ultra (runs 350, 354, 360, 365, 370), Drex Max 1.1 (run 352), Mammoth-1 (run 386), Fastino GLiDE v1.2 (run 369), Elephant-1 (run 382), SubQ Snap 1.1 (run 392) and Perplexity Decider 27B v1.2 (run 372); new runs on the kept questions only, with the Hallucination tab's prompts, for Inception Mercury Decide (runs 489, 491, 493, 495, 497), GPT-6 Luna (Decisions API) (run 453), Perplexity Decider Flash v1.2 (run 537), Jev 1.14.0 (runs 470, 473, 476, 480, 482), Microsoft-Decision-1 (runs 487, 490, 492, 494, 496) and Liquid d1 (runs 472, 475, 479, 481, 485). The new Turkish runs are not on the Hallucination tab, which uses full runs only.
Liquid d1 was asked as d1 in both languages, while its Text tab run used d1:free, because Liquid's API no longer serves that name; its model list still names d1, released 29 September 2026.
Celeris-1 Decision is shown in English only (run 548): no Turkish run with “Bilmiyorum” on these questions; new Turkish runs were outside the request of 3 November 2026. Its Turkish and paired columns read Not measured; it is in no paired test and no Holm adjustment.
Runs
five runs in English and five runs in Turkish for Liquid d1-ultra, Inception Mercury Decide, Jev 1.14.0, Microsoft-Decision-1 and Liquid d1; one run in English and one run in Turkish for Drex Max 1.1, Mammoth-1, Fastino GLiDE v1.2, Elephant-1, SubQ Snap 1.1, GPT-6 Luna (Decisions API), Perplexity Decider Flash v1.2 and Perplexity Decider 27B v1.2. These runs are kept out of the leaderboard, its repeats, the Hallucination tab and the accuracy comparison above.
The rates and how to read them
Each answer is right, wrong, the option or invalid. Out of the 547 kept questions, accuracy is right / 547; Answered is (right + wrong) / 547, the share of questions the model answered (the attempt rate), higher is better; Wrong of answers is wrong / (right + wrong), the share of its answers that were wrong, lower is better, and Not measured when it answered none. These two lead the table, with their counts under them. The non-hallucination rate, after them, is the option / (the option + wrong): among the answers that were not right, the share where the model said it did not know. As the Hallucination tab says, it is not a direct measure of what the model does not know: a model that always chose the option would reach 100% with no right answers, so read it with the attempt rate. With neither a wrong answer nor the option, the rate is undefined (Not measured). The mean probability is the probability the model gave the option, averaged over its answers that report option probabilities. The paired test is the exact McNemar test on the questions only one of the two first runs got wrong, with p adjusted by Holm's method over the 13 models. No new combined score is made from these rates.
Results with “I don't know” in both languages
In the order of the Results table above. Each row compares the model's first English run with “I don't know” with its Turkish run with the option also written “I don't know”, out of the 547 kept questions. The paired counts, p and the Holm-adjusted p (over the 7 models with both runs) are those of the exact McNemar test on wrong answers; * marks a Holm-adjusted p below 0.05. Not measured: no such run.
| Model | Turkish answered(right + wrong) / 547; higher is better | English answered(right + wrong) / 547; higher is better | Turkish wrong of answerswrong / (right + wrong); lower is better | English wrong of answerswrong / (right + wrong); lower is better | Turkish non-hallucination“I don't know” / (“I don't know” + wrong); higher is better, read with the answered share | English non-hallucination“I don't know” / (“I don't know” + wrong); higher is better, read with the answered share | Turkish accuracyhigher is better | English accuracyhigher is better | Turkish mean probabilityof “I don't know” | English mean probabilityof “I don't know” | Turkish runthe option-word run, counted on the kept questions | English runthe first English run with “I don't know” | Wrong only in Englishquestions the English run got wrong and the Turkish run did not | Wrong only in Turkishquestions the Turkish run got wrong and the English run did not | pexact McNemar | Holm padjusted over the 7 models with both runs |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Liquid d1-ultra1 question per request | 95.1%520 of 547 answered, 27 “I don't know” | 90.7%496 of 547 answered, 51 “I don't know” | 5.4%28 of 520 | 7.9%39 of 496 | 49.1%27 “I don't know” of 55 not right | 56.7%51 “I don't know” of 90 not right | 89.9%492 of 547 right | 83.5%457 of 547 right | 23.2% | 25.3% | run 511 | run 449 | 18 | 7 | 0.043 | 0.26 |
| Fastino GLiDE v1.2local · 1 question per request | 98.4%538 of 547 answered, 9 “I don't know” | 94.5%517 of 547 answered, 30 “I don't know” | 8.7%47 of 538 | 7.0%36 of 517 | 16.1%9 “I don't know” of 56 not right | 45.5%30 “I don't know” of 66 not right | 89.8%491 of 547 right | 87.9%481 of 547 right | 9.8% | 12.5% | run 525 | run 523 | 16 | 27 | 0.13 | 0.44 |
| Mammoth-11 question per request | 95.4%522 of 547 answered, 25 “I don't know” | 93.2%510 of 547 answered, 37 “I don't know” | 6.1%32 of 522 | 7.1%36 of 510 | 43.9%25 “I don't know” of 57 not right | 50.7%37 “I don't know” of 73 not right | 89.6%490 of 547 right | 86.7%474 of 547 right | 7.4% | 9.8% | run 519 | run 505 | 19 | 15 | 0.61 | 0.76 |
| vLLM-SR d31 question per request | 100.0%547 of 547 answered, 0 “I don't know” | Not measured | 9.5%52 of 547 | Not measured | 0.0%0 “I don't know” of 52 not right | Not measured | 90.5%495 of 547 right | Not measured | 1.4% | Not measured | run 544 | Not measured | Not measured | Not measured | Not measured | Not measured |
| Drex Max v11 question per request | 99.8%546 of 547 answered, 1 “I don't know” | Not measured | 9.0%49 of 546 | Not measured | 2.0%1 “I don't know” of 50 not right | Not measured | 90.9%497 of 547 right | Not measured | 0.9% | Not measured | run 542 | Not measured | Not measured | Not measured | Not measured | Not measured |
| vLLM-SR d3-ultratransformers, rev 52ba9e7 · 1 question per request | 99.6%545 of 547 answered, 2 “I don't know” | Not measured | 9.4%51 of 545 | Not measured | 3.8%2 “I don't know” of 53 not right | Not measured | 90.3%494 of 547 right | Not measured | 1.3% | Not measured | run 540 | Not measured | Not measured | Not measured | Not measured | Not measured |
| Perplexity Decider 27B v1.2local · 1 question per request | 99.8%546 of 547 answered, 1 “I don't know” | 99.6%545 of 547 answered, 2 “I don't know” | 9.3%51 of 546 | 11.7%64 of 545 | 1.9%1 “I don't know” of 52 not right | 3.0%2 “I don't know” of 66 not right | 90.5%495 of 547 right | 87.9%481 of 547 right | 1.4% | 2.8% | run 531 | run 529 | 29 | 16 | 0.072 | 0.36 |
| Drex Max 1.11 question per request | 96.7%529 of 547 answered, 18 “I don't know” | 92.1%504 of 547 answered, 43 “I don't know” | 9.1%48 of 529 | 8.1%41 of 504 | 27.3%18 “I don't know” of 66 not right | 51.2%43 “I don't know” of 84 not right | 87.9%481 of 547 right | 84.6%463 of 547 right | 10.2% | 14.3% | run 513 | run 447 | 20 | 27 | 0.38 | 0.76 |
| Liquid d11 question per request | Not measured | 99.6%545 of 547 answered, 2 “I don't know” | Not measured | 12.3%67 of 545 | Not measured | 2.9%2 “I don't know” of 69 not right | Not measured | 87.4%478 of 547 right | Not measured | 0.9% | Not measured | run 445 | Not measured | Not measured | Not measured | Not measured |
| Jev 1.14.01 question per request | Not measured | 99.5%544 of 547 answered, 3 “I don't know” | Not measured | 13.2%72 of 544 | Not measured | 4.0%3 “I don't know” of 75 not right | Not measured | 86.3%472 of 547 right | Not measured | 1.6% | Not measured | run 446 | Not measured | Not measured | Not measured | Not measured |
| SubQ Snap 1.11 question per request | 96.5%528 of 547 answered, 19 “I don't know” | 96.2%526 of 547 answered, 21 “I don't know” | 8.5%45 of 528 | 12.0%63 of 526 | 29.7%19 “I don't know” of 64 not right | 25.0%21 “I don't know” of 84 not right | 88.3%483 of 547 right | 84.6%463 of 547 right | 8.9% | 11.7% | run 512 | run 448 | 40 | 22 | 0.03 | 0.21 |
| Inception Mercury Decidevia OpenRouter · 1 question per request | Not measured | 98.0%536 of 547 answered, 11 “I don't know” | Not measured | 13.2%71 of 536 | Not measured | 13.4%11 “I don't know” of 82 not right | Not measured | 85.0%465 of 547 right | Not measured | 3.2% | Not measured | run 450 | Not measured | Not measured | Not measured | Not measured |
| Microsoft-Decision-1via OpenRouter · 1 question per request | Not measured | 99.5%544 of 547 answered, 3 “I don't know” | Not measured | 14.2%77 of 544 | Not measured | 3.8%3 “I don't know” of 80 not right | Not measured | 85.4%467 of 547 right | Not measured | 3.6% | Not measured | run 451 | Not measured | Not measured | Not measured | Not measured |
| Elephant-11 question per request | 95.6%523 of 547 answered, 24 “I don't know” | 95.8%524 of 547 answered, 23 “I don't know” | 15.7%82 of 523 | 12.8%67 of 524 | 22.6%24 “I don't know” of 106 not right | 25.6%23 “I don't know” of 90 not right | 80.6%441 of 547 right | 83.5%457 of 547 right | 7.3% | 6.8% | run 515 | run 501 | 31 | 46 | 0.11 | 0.44 |
| Celeris-1 Decision1 question per request | Not measured | 83.2%455 of 547 answered, 92 “I don't know” | Not measured | 8.1%37 of 455 | Not measured | 71.3%92 “I don't know” of 129 not right | Not measured | 76.4%418 of 547 right | Not measured | 12.7% | Not measured | run 548 | Not measured | Not measured | Not measured | Not measured |
| GPT-6 Luna (Decisions API)Decisions API, beta · 1 question per request | Not measured | 98.2%537 of 547 answered, 10 “I don't know” | Not measured | 17.9%96 of 537 | Not measured | 9.4%10 “I don't know” of 106 not right | Not measured | 80.6%441 of 547 right | Not measured | 3.0% | Not measured | run 444 | Not measured | Not measured | Not measured | Not measured |
| Perplexity Decider Flash v1.2local · 1 question per request | Not measured | 99.1%542 of 547 answered, 5 “I don't know” | Not measured | 18.5%100 of 542 | Not measured | 4.8%5 “I don't know” of 105 not right | Not measured | 80.8%442 of 547 right | Not measured | 5.5% | Not measured | run 535 | Not measured | Not measured | Not measured | Not measured |
Our method with “I don't know” in both languages
What is the same
In both languages the sixth option, F, is written “I don't know”, always the last option, never shuffled and never correct, and the instructions get the same added sentence: “If you do not know the answer, choose I don't know.” Both sides are counted on the same 547 kept questions.
What differs
The language of the question and its options (the English translation, as in the accuracy comparison above, against the Turkish original); the one word of the instructions that names the language of the question; and the options heading, “Seçenekler:” against “Options:”. The instructions of the Turkish runs:
Which option is the correct answer to this Turkish multiple-choice question? If you do not know the answer, choose I don't know.
The instructions of the English runs:
Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.
The runs
The Turkish side is the Hallucination tab's option-word run (the option's word) on all 2,198 Turkish questions, counted on the 547 kept questions only: Liquid d1-ultra (run 511), Fastino GLiDE v1.2 (run 525), Mammoth-1 (run 519), vLLM-SR d3 (run 544), Drex Max v1 (run 542), vLLM-SR d3-ultra (run 540), Perplexity Decider 27B v1.2 (run 531), Drex Max 1.1 (run 513), SubQ Snap 1.1 (run 512) and Elephant-1 (run 515). The English side is the first English run with “I don't know” of the section above: Liquid d1-ultra (run 449), Fastino GLiDE v1.2 (run 523), Mammoth-1 (run 505), Perplexity Decider 27B v1.2 (run 529), Drex Max 1.1 (run 447), Liquid d1 (run 445), Jev 1.14.0 (run 446), SubQ Snap 1.1 (run 448), Inception Mercury Decide (run 450), Microsoft-Decision-1 (run 451), Elephant-1 (run 501), Celeris-1 Decision (run 548), GPT-6 Luna (Decisions API) (run 444) and Perplexity Decider Flash v1.2 (run 535). First runs only: where a model has more runs, the others are not counted here. No run was made for this comparison.
Turkish only, with no English run with “I don't know”: vLLM-SR d3, Drex Max v1 and vLLM-SR d3-ultra. English only, with no Turkish run with the option written “I don't know”: Liquid d1, Jev 1.14.0, Inception Mercury Decide, Microsoft-Decision-1, Celeris-1 Decision, GPT-6 Luna (Decisions API) and Perplexity Decider Flash v1.2. Their other side and paired columns read Not measured; they are in no paired test and no Holm adjustment.
The test
A wrong answer is a valid choice other than “I don't know” that is not right. The paired test is the exact McNemar test on the questions only one of the two runs got wrong, with p adjusted by Holm's method over the seven models with both runs: a family of its own, separate from every other family on the site. The comparison with “Bilmiyorum” in Turkish stays as published in the section above (its table).
Languages the model cards list
Of the 19 models whose cards or docs we checked, only Elephant-1 and Mammoth-1 name a language: English, by the card's language: en tag. Jev's docs call English its primary language. No card names Turkish.
Each model's card at the revision we ran (local weights) or its maker's model page and docs (APIs), searched for language statements and the Hugging Face language: field, on 2026-11-02. “not stated”: neither names a language.
| Model | Languages listed | Source | Checked |
|---|---|---|---|
| Liquid d1-ultra | not stated | docs.liquid.ai | 2026-11-02 |
| Liquid d1 | not stated | docs.liquid.ai | 2026-11-02 |
| SubQ Snap 1.1 | not stated | docs.subq.ai | 2026-11-02 |
| Elephant-1 | English (the card's language: en tag; its text names no language) | huggingface.co | 2026-11-02 |
| Mammoth-1 | English (the card's language: en tag; its text names no language) | huggingface.co | 2026-11-02 |
| Fastino GLiDE v1.2 | not stated | huggingface.co | 2026-11-02 |
| Fastino GLiDE | not stated | docs.fastino.ai | 2026-11-02 |
| Perplexity Decider 27B v1.2 | not stated | huggingface.co | 2026-11-02 |
| Perplexity Decider Flash v1.2 | not stated | huggingface.co | 2026-11-02 |
| Jev 1.14.0 | "English is the primary training language and where accuracy is currently best. Other languages, including CJK scripts, are handled but not equally well" | docs.typesafe.ai | 2026-11-02 |
| Microsoft-Decision-1 | not stated | openrouter.ai | 2026-11-02 |
| Inception Mercury Decide | not stated | openrouter.ai | 2026-11-02 |
| Celeris-1 Decision | not stated | docs.celeris.ai | 2026-11-02 |
| Clef | not stated | huggingface.co | 2026-11-02 |
| GPT-6 Luna (Decisions API) | not stated | developers.openai.com | 2026-11-02 |
| Drex Max 1.1 | not stated | console.nace.ai | 2026-11-02 |
| Drex Max v1 | not stated | huggingface.co | 2026-11-02 |
| vLLM-SR d3 | not stated | huggingface.co | 2026-11-02 |
| vLLM-SR d3-ultra | not stated | huggingface.co | 2026-11-02 |