About

About the Logic & Math bench

Method · Data

What the test is

Our logic and math test: 1,125 logic questions (numerical reasoning (524), verbal reasoning (469), formal logic (132)) and 2,163 math questions in 13 topics, answered by 69 decision models, with chat models for comparison. A decision model scores each answer option and picks one; a chat model writes a reply that ends in a letter. Every question is five-option multiple choice, text only, scored against the answer key of one of six published Turkish question books (not named here), and asked in English translation. About a third of the logic questions and nearly all math questions are past ÖSYM (Turkish national exam) questions, which are public, so models may have seen them. The measures and the method are those of the team's decision-models benchmark (Method).

516 runs with answers so far, 284 of them decision-model runs.

How it was built

The questions were parsed from the question books; set problems and every question that needs a figure are left out. A model translated each one into English, with automatic checks (option keys, digits, LaTeX, tables, labels, leftover untranslated words); a text that still failed is flagged and not asked. Every setup answers every question once, through the same runner as the decision-models benchmark. Sources · Translation · Runs · Repeat checks (Experiments: a second run of a setup, against its first).

How the scores work

Accuracy is the share of questions answered right, with its 95% Wilson interval; non-hallucination is the share of the answers that were not right that were an honest “I don't know” (a sixth option F, in a separate run); confidence counts the answers a decision model was at least 99% sure of and how many of them were wrong; cost and latency are per question. Five points on the front page · every measure in Method.

The leaderboard's rules

Cost marks

estimate · RTX PRO 6000 Blackwell / DGX Spark = run on our own hardware; its cost is an estimate, explained on hover, focus or tap of the amount: the machine's wall-clock time on the run, first to last answer while it was running (pauses, estimated from gaps of more than 10 minutes between answers, left out and listed), × $2.49/h per NVIDIA RTX PRO 6000 Blackwell (96 GB) (Runpod's price for the card) or × $0.213/h per NVIDIA DGX Spark (GB10) (its purchase over three years plus electricity; see Method), each run charged in full even when runs shared the machine, so at most; sorting and the cost chart use the estimate; list = maker's list price × our token counts; unmarked = provider's price × usage the API reported.

More views

Every measure of each part, for all models or one group (the keys on each page):

How to read the test pages

What the charts and tables of each test page show:

Data

Every aggregate number is in results.json (per test, Logic, Math and Logic and Math, and per setup: accuracy with its Wilson interval, right and N, Decision Score and chosen-answer Brier, AUROC, non-hallucination, cost with its basis and median latency; no question text, question ids or replies) and on these pages: the leaderboard, the full table of each test page (Every setup, every measure), and each run's record (prompt, parameters, cost basis) on Method.