About
About the Logic & Math bench
What the test is
Our logic and math test: 1,125 logic questions (numerical reasoning (524), verbal reasoning (469), formal logic (132)) and 2,163 math questions in 13 topics, answered by 69 decision models, with chat models for comparison. A decision model scores each answer option and picks one; a chat model writes a reply that ends in a letter. Every question is five-option multiple choice, text only, scored against the answer key of one of six published Turkish question books (not named here), and asked in English translation. About a third of the logic questions and nearly all math questions are past ÖSYM (Turkish national exam) questions, which are public, so models may have seen them. The measures and the method are those of the team's decision-models benchmark (Method).
516 runs with answers so far, 284 of them decision-model runs.
How it was built
The questions were parsed from the question books; set problems and every question that needs a figure are left out. A model translated each one into English, with automatic checks (option keys, digits, LaTeX, tables, labels, leftover untranslated words); a text that still failed is flagged and not asked. Every setup answers every question once, through the same runner as the decision-models benchmark. Sources · Translation · Runs · Repeat checks (Experiments: a second run of a setup, against its first).
How the scores work
Accuracy is the share of questions answered right, with its 95% Wilson interval; non-hallucination is the share of the answers that were not right that were an honest “I don't know” (a sixth option F, in a separate run); confidence counts the answers a decision model was at least 99% sure of and how many of them were wrong; cost and latency are per question. Five points on the front page · every measure in Method.
- Decision Score (AA formula): Artificial Analysis's formula applied to our questions. Not their Index. 50 × (1 + accuracy − chosen-answer Brier), the chosen-answer Brier over the probability of the chosen answer, on decision models' standard runs only. Chat models and the “I don't know” runs are excluded.
The leaderboard's rules
- One row per model: its highest reasoning setting (off < low < medium < high), a model with one setup that setup; a tie by the earliest run, then the name. Perplexity Decider 27B v1.2: its API setup on every part (its other setup is under other settings). The model's other settings are under its row; All setups lists every setup.
- The keys above the table: the part (Logic, Math, or Both: the two pooled, setups that finished both), the models (All, Chat models, Decision models, Small chat models, Small decision models; it opens on the decision models) and All setups (every setup its own row).
- Model size: the parameter count in each row's details box is from the maker's model card or official announcement, or counted from the released checkpoint (its config and tensor shapes); a run's notes win where they state it. Each run's record on Method names the source and the date it was read. A mixture-of-experts model shows its total and active parameters, a Gemma E-model its effective and total parameters. The Class keys: Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price. Open or closed and the list price: the maker's model card or price page (decision models: the price recorded with each setup; chat models through OpenRouter: the pinned provider's price on OpenRouter's endpoint listing). Tiny today: Chinese Decision Pointer 4B, d1-2B, Gemma 4 E2B (chat), Gemma 4 E2B (on · chat · 32k cap · sampled), Gemma 4 E2B (on · chat · 32k cap), Gemma 4 E2B (on · chat), GLiClass v4 Edge, GLiDE 0.4B, GLiDE 0.8B, GLiDE 2B, Granite-4.2-3B (high · chat), KnowLine 4B Gen5.5, KnowLine-0.8B-Gen1, KnowLine-2B-Gen1, LFM2.5-2.6B (on · chat), LFM3-1.7B (off · chat · NVIDIA RTX PRO 6000 Blackwell), LFM3-1.7B (off · chat), LFM3-1.7B (on · chat), Metask-Jev-4B, MiniCPM5-2B (off · chat), MiniCPM5-2B (on · chat), OneJev1.6-0.8B, Perplexity Decider Flash v1.2, Perplexity Decider Flash v1.3, Qwen3.5-2B (off · chat), Qwen3.5-2B (on · chat), Qwen3.5-4B (off · chat), Qwen3.5-4B (on · chat), Raon-1 nano (off · chat), Raon-1 nano (on · chat), StartLux-Decision-v1.1-0.8B-Instant, StartLux-Decision-v1.1-2B-Instant, StartLux-Decision-v1.1-4B-Instant, Strands Decider 2B, vera-medium-omni, Verdin-4B-v1.1. Small today: Adjudex 9B v5, Cerno 9B v1.2, Clef-Flash, Cygnet, Dexy-Decision-9B-V2, Elephant-1, Erk-14B (off · chat), Gemma 4 12B Unified (chat), Gemma 4 E4B (chat), Gemma 4 E4B (on · chat · 32k cap · sampled), Gemma 4 E4B (on · chat · 32k cap), Gemma 4 E4B (on · chat), GLiDE 9B, Jeffan-9B-1211, KnowLine-12B-Gen1, KnowLine-9B-Gen1, KnowLine-9B-Gen2, LFM2.5-8B-A1B (on · chat), Qwen3-14B (off · chat), StartLux-Decision-v1.1-9B-Instant, Valen-Decision-1.5-9B, Winnow-12B. Medium today: Clef, Copernicus-31B, d3, d3-ultra, Drex 2.0, Drex Max, Gemma 4 31B (chat), GLiDE (v1.2), GLiDE 27B, Karpo Decision 31B, KnowLine-31B-Gen1.5, KnowLine-31B-Gen2, LFM2.5-24B-A2B (on · chat), Mammoth-1, OneJev-27B, Open-Jev-27B-v1.1, Perplexity Decider 27B v1.2 (API · with web), Perplexity Decider 27B v1.2 (API), Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell), Perplexity Decider 27B v1.3 (NVIDIA RTX PRO 6000 Blackwell), Qwen3.8-27B (low · chat), Sieve 31B v3, Turion Jev 31B. Large today: DeepSeek-V4-Flash (chat), DeepSeek-V4.1-Flash (high · chat · 1017), GLM-5.3-Flash (high · chat), Inkling 2 Small (high · chat), Inkling-Small (high · chat · Jul 2026), Logits Sigma-1 Duo, MiMo-V2.6-Flash (on · chat), MiniMax-M3.1-Flash (high · chat), Qwen3.5-397B-A17B (on · chat), REAX 1.3. Efficient today: celeris-1-decision, celeris-1.1-decision, d1, d1-ultra, d1.1, Drex Max v1.1, Gemini Decision 1 (20261111), GLiDE (API), GPT-6 Luna (chat), Jev 1.13, Jev 1.14, Mercury Decide, Microsoft-Decision-1, Muse Ember (max · chat), SubQ Snap 1.1, SubQ Snap Preview. Workhorse today: Claude Sonnet 5.5 (chat). Flagship today: Claude Fable 5.1 (chat), Claude Opus 5.5 (chat). Undisclosed today: d1.2, d1.2-ultra, Gemini 4 Carbon (chat), GPT-6 Astra (chat), GPT-6.1 Sol (chat), GPT-6.1 Terra (chat), Grok 4.7 (chat), Qwen3.8 Flash (off · chat), Qwen4-Max (high · chat). No class on record yet: Gemini 4 Hydrogen (chat). Parameters not disclosed by the maker: celeris-1-decision, celeris-1.1-decision, Claude Fable 5.1 (chat), Claude Opus 5.5 (chat), Claude Sonnet 5.5 (chat), d1, d1-ultra, d1.1, d1.2, d1.2-ultra, Drex Max v1.1, Gemini 4 Carbon (chat), Gemini Decision 1 (20261111), GLiDE (API), GPT-6 Luna (chat), Grok 4.7 (chat), Jev 1.13, Jev 1.14, Mercury Decide, Microsoft-Decision-1, Muse Ember (max · chat), Qwen3.8 Flash (off · chat), Qwen4-Max (high · chat), SubQ Snap 1.1, SubQ Snap Preview.
- Every number of a row comes from the one setup the row shows; a row's details box (hover, focus or tap) has its other measures.
- Only finished runs count. In the other part's column, “running”: its run is in progress; a dash: no run yet.
- Ranks show observed scores; small gaps need not be meaningful.
- A setup that runs after the others counts like any other; none is a late entry so far.
- Setups shown once with another:
- SubQ Snap Preview and SubQ Snap 1.1 returned the same answer to every question (and the same probabilities on 3,276 of 3,288); shown once.
- Perplexity Decider 27B v1.2 (API) and Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell): two setups of the same model, one on an NVIDIA RTX PRO 6000 Blackwell (96 GB) and one through Perplexity's API; not verified to be identical weights or settings. Its row is the API setup on every test; Perplexity Decider 27B v1.2 (NVIDIA RTX PRO 6000 Blackwell) is under the row's other settings and All setups.
Cost marks
estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.
More views
Every measure of each part, for all models or one group (the keys on each page):
- Logic Accuracy · Non-hallucination · Cost · Latency · I don't know · Confidence
- Math Accuracy · Non-hallucination · Cost · Latency · I don't know · Confidence
- Both Accuracy · Non-hallucination · Cost · Latency · I don't know · Confidence
How to read the test pages
What the charts and tables of each test page show:
- Logic, Math: Five options each (A–E). Decision-model setups score every option and pick one; chat-model setups write a reply with an answer letter.
- Logic and Math: This test is not a third question set: it pools the 1,125 Logic questions and the 2,163 Math questions into one set of 3,288 questions and scores every model on all of them together. Only models that have finished both Logic and Math are included; the others are listed below as not in Logic and Math yet. Five options each (A–E). Decision-model setups score every option and pick one; chat-model setups write a reply with an answer letter.
- Every setup, every measure (the full table): One row per setup in accuracy order. Tie group: setups that share a letter are not told apart by the exact McNemar test on the questions both answered, Holm-corrected over every pair in this group (tb-next's compact letter display). “Not measured”: chat runs return no option probabilities and have no “I don't know” run yet; a run without a recorded cost shows —.
- Non-hallucination, (a) Probabilistic confidence: Five options, the standard run: how many answers a setup's own probability lets it keep at ≥99% and how many of those were wrong, and how well the probability separates right from wrong answers (AUROC).
- Non-hallucination, (b) Explicit “I don't know” option: A sixth option F on the same questions (mode dont_know runs): non-hallucination = F / (F + wrong), with the attempt rate (right + wrong) / N and the accuracy when answering right / (right + wrong). A setup that works with probabilities may rarely choose F; read it on (a) and on the comparison with its own confidence.
- I don't know: A sixth option F on the same questions (mode dont_know runs): non-hallucination = F / (F + wrong), the attempt rate (right + wrong) / N, the accuracy when answering right / (right + wrong), the counts, and the comparison with the same setup's five-option run held back by its own confidence.
Data
Every aggregate number is in results.json (per test, Logic, Math and Logic and Math, and per setup: accuracy with its Wilson interval, right and N, Decision Score and chosen-answer Brier, AUROC, non-hallucination, cost with its basis and median latency; no question text, question ids or replies) and on these pages: the leaderboard, the full table of each test page (Every setup, every measure), and each run's record (prompt, parameters, cost basis) on Method.