Vision
Logic & Math questions, as text and as a page image
Logic & Math bench · its 358 figure questions (306 Logic + 52 Math) · original page images · five options · Info
LFM3-VL-2B is withdrawn for a check; see Experiments →
The picture alone (B) · 306 Logic questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 14 queued, 2 blocked, 1 partial · Table
Alibaba Google PelaAI Fastino Nace.AI vLLM Semantic Router Subquadratic Liquid AI Chance, 20% (five options)
The picture alone (B) · 306 Logic questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| Qwen3.5-4B on · chattie group a Qwen3.5-4B (on · chat): 62.1%, 190 right of 306; Wilson 95% interval 56.5% to 67.3% | 62.1% | 190 / 306 | a | – |
| Gemma 4 12B Unified chattie group a Gemma 4 12B Unified (chat): 56.5%, 173 right of 306; Wilson 95% interval 50.9% to 62.0% | 56.5% | 173 / 306 | a | – |
| Gemma 4 E4B chattie group b Gemma 4 E4B (chat): 38.2%, 117 right of 306; Wilson 95% interval 33.0% to 43.8% | 38.2% | 117 / 306 | b | – |
| Gemini Decision 1 20261111tie group b · +2.3 points vs A Gemini Decision 1 (20261111): 33.7%, 103 right of 306; Wilson 95% interval 28.6% to 39.1% | 33.7% | 103 / 306 | b | +2.3 points |
| KnowLine-31B-Gen1.5 Run 2Run 2 of 12 Nov; first run 99/306 kept on Experiments · tie group b · +0.3 points vs A KnowLine-31B-Gen1.5 (Run 2): 32.0%, 98 right of 306; Wilson 95% interval 27.0% to 37.4%; Run 2 of 12 Nov; first run 99/306 kept on Experiments | 32.0% | 98 / 306 | b | +0.3 points |
| GLiDE 27Btie group b · +2.0 points vs A GLiDE 27B: 30.7%, 94 right of 306; Wilson 95% interval 25.8% to 36.1% | 30.7% | 94 / 306 | b | +2.0 points |
| Drex Max v1.1tie group b · −1.6 points vs A Drex Max v1.1: 29.7%, 91 right of 306; Wilson 95% interval 24.9% to 35.1% | 29.7% | 91 / 306 | b | −1.6 points |
| d3-ultratie group b · −2.3 points vs A d3-ultra: 29.1%, 89 right of 306; Wilson 95% interval 24.3% to 34.4% | 29.1% | 89 / 306 | b | −2.3 points |
| SubQ Snap 1.1tie group b · +4.2 points vs A SubQ Snap 1.1: 28.4%, 87 right of 306; Wilson 95% interval 23.7% to 33.7% | 28.4% | 87 / 306 | b | +4.2 points |
| d1-ultratie group b · −1.3 points vs A d1-ultra: 27.5%, 84 right of 306; Wilson 95% interval 22.8% to 32.7% | 27.5% | 84 / 306 | b | −1.3 points |
| Jev 1.14 | Partial | answered 304 / 306 | – | – |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | – | – | – |
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – | – |
| Gemma 4 31B chatneeds a card | Queued | – | – | – |
| Gemma 4 E2B chatneeds a card | Queued | – | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – | – |
| Muse Glimmer 30B chatJob 2 phase | Queued | – | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – | – |
| Qwen3.5-9B chatwaits for its server window | Queued | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – | – |
| Muse Ember chatno qualified Logic & Math chat adapter for its hosted route, and no L&M budget for it | Blocked | – | – | – |
The picture alone (B) · 52 Math questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 14 queued, 2 blocked · Table
Google Alibaba Liquid AI Subquadratic Nace.AI TypeSafe Fastino vLLM Semantic Router PelaAI Chance, 20% (five options)
The picture alone (B) · 52 Math questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 12B Unified chattie group a Gemma 4 12B Unified (chat): 86.5%, 45 right of 52; Wilson 95% interval 74.7% to 93.3% | 86.5% | 45 / 52 | a | – |
| Qwen3.5-4B on · chattie group a Qwen3.5-4B (on · chat): 86.5%, 45 right of 52; Wilson 95% interval 74.7% to 93.3% | 86.5% | 45 / 52 | a | – |
| Gemma 4 E4B chattie group b Gemma 4 E4B (chat): 53.8%, 28 right of 52; Wilson 95% interval 40.5% to 66.7% | 53.8% | 28 / 52 | b | – |
| d1-ultratie group b · +13.5 points vs A d1-ultra: 51.9%, 27 right of 52; Wilson 95% interval 38.7% to 64.9% | 51.9% | 27 / 52 | b | +13.5 points |
| Gemini Decision 1 20261111tie group b · +1.9 points vs A Gemini Decision 1 (20261111): 46.2%, 24 right of 52; Wilson 95% interval 33.3% to 59.5% | 46.2% | 24 / 52 | b | +1.9 points |
| SubQ Snap 1.1tie group b · −1.9 points vs A SubQ Snap 1.1: 42.3%, 22 right of 52; Wilson 95% interval 29.9% to 55.8% | 42.3% | 22 / 52 | b | −1.9 points |
| Drex Max v1.1tie group b · +3.8 points vs A Drex Max v1.1: 40.4%, 21 right of 52; Wilson 95% interval 28.2% to 53.9% | 40.4% | 21 / 52 | b | +3.8 points |
| Jev 1.14tie group b · −3.8 points vs A Jev 1.14: 40.4%, 21 right of 52; Wilson 95% interval 28.2% to 53.9% | 40.4% | 21 / 52 | b | −3.8 points |
| GLiDE 27Btie group b · −1.9 points vs A GLiDE 27B: 38.5%, 20 right of 52; Wilson 95% interval 26.5% to 52.0% | 38.5% | 20 / 52 | b | −1.9 points |
| d3-ultratie group b · −1.9 points vs A d3-ultra: 36.5%, 19 right of 52; Wilson 95% interval 24.8% to 50.1% | 36.5% | 19 / 52 | b | −1.9 points |
| KnowLine-31B-Gen1.5tie group b · −1.9 points vs A KnowLine-31B-Gen1.5: 36.5%, 19 right of 52; Wilson 95% interval 24.8% to 50.1% | 36.5% | 19 / 52 | b | −1.9 points |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | – | – | – |
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – | – |
| Gemma 4 31B chatneeds a card | Queued | – | – | – |
| Gemma 4 E2B chatneeds a card | Queued | – | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – | – |
| Muse Glimmer 30B chatJob 2 phase | Queued | – | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – | – |
| Qwen3.5-9B chatwaits for its server window | Queued | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – | – |
| Muse Ember chatno qualified Logic & Math chat adapter for its hosted route, and no L&M budget for it | Blocked | – | – | – |
The picture alone (B) · 358 Logic & Math questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 14 queued, 2 blocked, 1 partial · Table
Alibaba Google PelaAI Fastino Nace.AI Liquid AI Subquadratic vLLM Semantic Router Chance, 20% (five options)
The picture alone (B) · 358 Logic & Math questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| Qwen3.5-4B on · chattie group a Qwen3.5-4B (on · chat): 65.6%, 235 right of 358; Wilson 95% interval 60.6% to 70.4% | 65.6% | 235 / 358 | a | – |
| Gemma 4 12B Unified chattie group a Gemma 4 12B Unified (chat): 60.9%, 218 right of 358; Wilson 95% interval 55.7% to 65.8% | 60.9% | 218 / 358 | a | – |
| Gemma 4 E4B chattie group b Gemma 4 E4B (chat): 40.5%, 145 right of 358; Wilson 95% interval 35.5% to 45.7% | 40.5% | 145 / 358 | b | – |
| Gemini Decision 1 20261111tie group b · +2.2 points vs A Gemini Decision 1 (20261111): 35.5%, 127 right of 358; Wilson 95% interval 30.7% to 40.6% | 35.5% | 127 / 358 | b | +2.2 points |
| KnowLine-31B-Gen1.5 first runsfirst runs (Logic 99 + Math 19); no complete Run 2 of B on Math yet, so run numbers are not mixed · tie group b · +0.3 points vs A KnowLine-31B-Gen1.5 (first runs): 33.0%, 118 right of 358; Wilson 95% interval 28.3% to 38.0%; first runs (Logic 99 + Math 19); no complete Run 2 of B on Math yet, so run numbers are not mixed | 33.0% | 118 / 358 | b | +0.3 points |
| GLiDE 27Btie group b · +1.4 points vs A GLiDE 27B: 31.8%, 114 right of 358; Wilson 95% interval 27.2% to 36.8% | 31.8% | 114 / 358 | b | +1.4 points |
| Drex Max v1.1tie group b · −0.8 points vs A Drex Max v1.1: 31.3%, 112 right of 358; Wilson 95% interval 26.7% to 36.3% | 31.3% | 112 / 358 | b | −0.8 points |
| d1-ultratie group b · +0.8 points vs A d1-ultra: 31.0%, 111 right of 358; Wilson 95% interval 26.4% to 36.0% | 31.0% | 111 / 358 | b | +0.8 points |
| SubQ Snap 1.1tie group b · +3.4 points vs A SubQ Snap 1.1: 30.4%, 109 right of 358; Wilson 95% interval 25.9% to 35.4% | 30.4% | 109 / 358 | b | +3.4 points |
| d3-ultratie group b · −2.2 points vs A d3-ultra: 30.2%, 108 right of 358; Wilson 95% interval 25.6% to 35.1% | 30.2% | 108 / 358 | b | −2.2 points |
| Jev 1.14 | Partial | answered 356 / 358 | – | – |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | – | – | – |
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – | – |
| Gemma 4 31B chatneeds a card | Queued | – | – | – |
| Gemma 4 E2B chatneeds a card | Queued | – | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – | – |
| Muse Glimmer 30B chatJob 2 phase | Queued | – | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – | – |
| Qwen3.5-9B chatwaits for its server window | Queued | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – | – |
| Muse Ember chatno qualified Logic & Math chat adapter for its hosted route, and no L&M budget for it | Blocked | – | – | – |
Text only, the control (A) · 306 Logic questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 10 queued, 1 blocked, 8 not applicable · Table
PelaAI vLLM Semantic Router Nace.AI Google Liquid AI Fastino TypeSafe Subquadratic Chance, 20% (five options)
Text only, the control (A) · 306 Logic questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| KnowLine-31B-Gen1.5tie group a KnowLine-31B-Gen1.5: 31.7%, 97 right of 306; Wilson 95% interval 26.7% to 37.1% | 31.7% | 97 / 306 | a |
| d3-ultratie group a d3-ultra: 31.4%, 96 right of 306; Wilson 95% interval 26.4% to 36.8% | 31.4% | 96 / 306 | a |
| Drex Max v1.1tie group a Drex Max v1.1: 31.4%, 96 right of 306; Wilson 95% interval 26.4% to 36.8% | 31.4% | 96 / 306 | a |
| Gemini Decision 1 20261111tie group a Gemini Decision 1 (20261111): 31.4%, 96 right of 306; Wilson 95% interval 26.4% to 36.8% | 31.4% | 96 / 306 | a |
| d1-ultratie group a d1-ultra: 28.8%, 88 right of 306; Wilson 95% interval 24.0% to 34.1% | 28.8% | 88 / 306 | a |
| GLiDE 27Btie group a GLiDE 27B: 28.8%, 88 right of 306; Wilson 95% interval 24.0% to 34.1% | 28.8% | 88 / 306 | a |
| Jev 1.14tie group a Jev 1.14: 25.2%, 77 right of 306; Wilson 95% interval 20.6% to 30.3% | 25.2% | 77 / 306 | a |
| SubQ Snap 1.1tie group a SubQ Snap 1.1: 24.2%, 74 right of 306; Wilson 95% interval 19.7% to 29.3% | 24.2% | 74 / 306 | a |
| Gemma 4 12B Unified chat | Not applicable | – | – |
| Gemma 4 E4B chat | Not applicable | – | – |
| Qwen3.5-4B on · chat | Not applicable | – | – |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | answered 136 / 306 | – |
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – |
| Gemma 4 31B chat | Not applicable | – | – |
| Gemma 4 E2B chat | Not applicable | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – |
| Qwen3.5-9B chat | Not applicable | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – |
| Muse Ember chat | Not applicable | – | – |
Text only, the control (A) · 52 Math questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 10 queued, 1 blocked, 8 not applicable · Table
Google TypeSafe Subquadratic Fastino Liquid AI vLLM Semantic Router PelaAI Nace.AI Chance, 20% (five options)
Text only, the control (A) · 52 Math questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| Gemini Decision 1 20261111tie group a Gemini Decision 1 (20261111): 44.2%, 23 right of 52; Wilson 95% interval 31.6% to 57.7% | 44.2% | 23 / 52 | a |
| Jev 1.14tie group a Jev 1.14: 44.2%, 23 right of 52; Wilson 95% interval 31.6% to 57.7% | 44.2% | 23 / 52 | a |
| SubQ Snap 1.1tie group a SubQ Snap 1.1: 44.2%, 23 right of 52; Wilson 95% interval 31.6% to 57.7% | 44.2% | 23 / 52 | a |
| GLiDE 27Btie group a GLiDE 27B: 40.4%, 21 right of 52; Wilson 95% interval 28.2% to 53.9% | 40.4% | 21 / 52 | a |
| d1-ultratie group a d1-ultra: 38.5%, 20 right of 52; Wilson 95% interval 26.5% to 52.0% | 38.5% | 20 / 52 | a |
| d3-ultratie group a d3-ultra: 38.5%, 20 right of 52; Wilson 95% interval 26.5% to 52.0% | 38.5% | 20 / 52 | a |
| KnowLine-31B-Gen1.5tie group a KnowLine-31B-Gen1.5: 38.5%, 20 right of 52; Wilson 95% interval 26.5% to 52.0% | 38.5% | 20 / 52 | a |
| Drex Max v1.1tie group a Drex Max v1.1: 36.5%, 19 right of 52; Wilson 95% interval 24.8% to 50.1% | 36.5% | 19 / 52 | a |
| Gemma 4 12B Unified chat | Not applicable | – | – |
| Gemma 4 E4B chat | Not applicable | – | – |
| Qwen3.5-4B on · chat | Not applicable | – | – |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | – | – |
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – |
| Gemma 4 31B chat | Not applicable | – | – |
| Gemma 4 E2B chat | Not applicable | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – |
| Qwen3.5-9B chat | Not applicable | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – |
| Muse Ember chat | Not applicable | – | – |
Text only, the control (A) · 358 Logic & Math questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 10 queued, 1 blocked, 8 not applicable · Table
Google PelaAI vLLM Semantic Router Nace.AI Fastino Liquid AI TypeSafe Subquadratic Chance, 20% (five options)
Text only, the control (A) · 358 Logic & Math questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| Gemini Decision 1 20261111tie group a Gemini Decision 1 (20261111): 33.2%, 119 right of 358; Wilson 95% interval 28.6% to 38.3% | 33.2% | 119 / 358 | a |
| KnowLine-31B-Gen1.5tie group a KnowLine-31B-Gen1.5: 32.7%, 117 right of 358; Wilson 95% interval 28.0% to 37.7% | 32.7% | 117 / 358 | a |
| d3-ultratie group a d3-ultra: 32.4%, 116 right of 358; Wilson 95% interval 27.8% to 37.4% | 32.4% | 116 / 358 | a |
| Drex Max v1.1tie group a Drex Max v1.1: 32.1%, 115 right of 358; Wilson 95% interval 27.5% to 37.1% | 32.1% | 115 / 358 | a |
| GLiDE 27Btie group a GLiDE 27B: 30.4%, 109 right of 358; Wilson 95% interval 25.9% to 35.4% | 30.4% | 109 / 358 | a |
| d1-ultratie group a d1-ultra: 30.2%, 108 right of 358; Wilson 95% interval 25.6% to 35.1% | 30.2% | 108 / 358 | a |
| Jev 1.14tie group a Jev 1.14: 27.9%, 100 right of 358; Wilson 95% interval 23.5% to 32.8% | 27.9% | 100 / 358 | a |
| SubQ Snap 1.1tie group a SubQ Snap 1.1: 27.1%, 97 right of 358; Wilson 95% interval 22.8% to 31.9% | 27.1% | 97 / 358 | a |
| Gemma 4 12B Unified chat | Not applicable | – | – |
| Gemma 4 E4B chat | Not applicable | – | – |
| Qwen3.5-4B on · chat | Not applicable | – | – |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | answered 136 / 358 | – |
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – |
| Gemma 4 31B chat | Not applicable | – | – |
| Gemma 4 E2B chat | Not applicable | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – |
| Qwen3.5-9B chat | Not applicable | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – |
| Muse Ember chat | Not applicable | – | – |
Text plus picture (C) · 306 Logic questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 10 queued, 1 blocked, 1 partial, 8 not applicable · Table
Fastino PelaAI Liquid AI Google Subquadratic Nace.AI vLLM Semantic Router Chance, 20% (five options)
Text plus picture (C) · 306 Logic questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| GLiDE 27Btie group a · +9.8 points vs A GLiDE 27B: 38.6%, 118 right of 306; Wilson 95% interval 33.3% to 44.1% | 38.6% | 118 / 306 | a | +9.8 points |
| KnowLine-31B-Gen1.5tie group a · +5.6 points vs A KnowLine-31B-Gen1.5: 37.3%, 114 right of 306; Wilson 95% interval 32.0% to 42.8% | 37.3% | 114 / 306 | a | +5.6 points |
| d1-ultratie group a · +6.9 points vs A d1-ultra: 35.6%, 109 right of 306; Wilson 95% interval 30.5% to 41.1% | 35.6% | 109 / 306 | a | +6.9 points |
| Gemini Decision 1 20261111tie group a · +3.3 points vs A Gemini Decision 1 (20261111): 34.6%, 106 right of 306; Wilson 95% interval 29.5% to 40.1% | 34.6% | 106 / 306 | a | +3.3 points |
| SubQ Snap 1.1tie group a · +9.5 points vs A SubQ Snap 1.1: 33.7%, 103 right of 306; Wilson 95% interval 28.6% to 39.1% | 33.7% | 103 / 306 | a | +9.5 points |
| Drex Max v1.1tie group a · +1.6 points vs A Drex Max v1.1: 33.0%, 101 right of 306; Wilson 95% interval 28.0% to 38.5% | 33.0% | 101 / 306 | a | +1.6 points |
| d3-ultratie group a · +0.3 points vs A d3-ultra: 31.7%, 97 right of 306; Wilson 95% interval 26.7% to 37.1% | 31.7% | 97 / 306 | a | +0.3 points |
| Gemma 4 12B Unified chat | Not applicable | – | – | – |
| Gemma 4 E4B chat | Not applicable | – | – | – |
| Qwen3.5-4B on · chat | Not applicable | – | – | – |
| Jev 1.14 | Partial | answered 304 / 306 | – | – |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | – | – | – |
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – | – |
| Gemma 4 31B chat | Not applicable | – | – | – |
| Gemma 4 E2B chat | Not applicable | – | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – | – |
| Qwen3.5-9B chat | Not applicable | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – | – |
| Muse Ember chat | Not applicable | – | – | – |
Text plus picture (C) · 52 Math questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 10 queued, 1 blocked, 8 not applicable · Table
TypeSafe vLLM Semantic Router Google Liquid AI Fastino PelaAI Subquadratic Nace.AI Chance, 20% (five options)
Text plus picture (C) · 52 Math questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| Jev 1.14tie group a · +7.7 points vs A Jev 1.14: 51.9%, 27 right of 52; Wilson 95% interval 38.7% to 64.9% | 51.9% | 27 / 52 | a | +7.7 points |
| d3-ultratie group a · +9.6 points vs A d3-ultra: 48.1%, 25 right of 52; Wilson 95% interval 35.1% to 61.3% | 48.1% | 25 / 52 | a | +9.6 points |
| Gemini Decision 1 20261111tie group a · +3.8 points vs A Gemini Decision 1 (20261111): 48.1%, 25 right of 52; Wilson 95% interval 35.1% to 61.3% | 48.1% | 25 / 52 | a | +3.8 points |
| d1-ultratie group a · +7.7 points vs A d1-ultra: 46.2%, 24 right of 52; Wilson 95% interval 33.3% to 59.5% | 46.2% | 24 / 52 | a | +7.7 points |
| GLiDE 27Btie group a · +5.8 points vs A GLiDE 27B: 46.2%, 24 right of 52; Wilson 95% interval 33.3% to 59.5% | 46.2% | 24 / 52 | a | +5.8 points |
| KnowLine-31B-Gen1.5tie group a · +7.7 points vs A KnowLine-31B-Gen1.5: 46.2%, 24 right of 52; Wilson 95% interval 33.3% to 59.5% | 46.2% | 24 / 52 | a | +7.7 points |
| SubQ Snap 1.1tie group a · 0.0 points vs A SubQ Snap 1.1: 44.2%, 23 right of 52; Wilson 95% interval 31.6% to 57.7% | 44.2% | 23 / 52 | a | 0.0 points |
| Drex Max v1.1tie group a · −3.8 points vs A Drex Max v1.1: 32.7%, 17 right of 52; Wilson 95% interval 21.5% to 46.2% | 32.7% | 17 / 52 | a | −3.8 points |
| Gemma 4 12B Unified chat | Not applicable | – | – | – |
| Gemma 4 E4B chat | Not applicable | – | – | – |
| Qwen3.5-4B on · chat | Not applicable | – | – | – |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | – | – | – |
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – | – |
| Gemma 4 31B chat | Not applicable | – | – | – |
| Gemma 4 E2B chat | Not applicable | – | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – | – |
| Qwen3.5-9B chat | Not applicable | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – | – |
| Muse Ember chat | Not applicable | – | – | – |
Text plus picture (C) · 358 Logic & Math questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 10 queued, 1 blocked, 1 partial, 8 not applicable · Table
Fastino PelaAI Liquid AI Google Subquadratic vLLM Semantic Router Nace.AI Chance, 20% (five options)
Text plus picture (C) · 358 Logic & Math questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| GLiDE 27Btie group a · +9.2 points vs A GLiDE 27B: 39.7%, 142 right of 358; Wilson 95% interval 34.7% to 44.8% | 39.7% | 142 / 358 | a | +9.2 points |
| KnowLine-31B-Gen1.5tie group a · +5.9 points vs A KnowLine-31B-Gen1.5: 38.5%, 138 right of 358; Wilson 95% interval 33.7% to 43.7% | 38.5% | 138 / 358 | a | +5.9 points |
| d1-ultratie group a · +7.0 points vs A d1-ultra: 37.2%, 133 right of 358; Wilson 95% interval 32.3% to 42.3% | 37.2% | 133 / 358 | a | +7.0 points |
| Gemini Decision 1 20261111tie group a · +3.4 points vs A Gemini Decision 1 (20261111): 36.6%, 131 right of 358; Wilson 95% interval 31.8% to 41.7% | 36.6% | 131 / 358 | a | +3.4 points |
| SubQ Snap 1.1tie group a · +8.1 points vs A SubQ Snap 1.1: 35.2%, 126 right of 358; Wilson 95% interval 30.4% to 40.3% | 35.2% | 126 / 358 | a | +8.1 points |
| d3-ultratie group a · +1.7 points vs A d3-ultra: 34.1%, 122 right of 358; Wilson 95% interval 29.4% to 39.1% | 34.1% | 122 / 358 | a | +1.7 points |
| Drex Max v1.1tie group a · +0.8 points vs A Drex Max v1.1: 33.0%, 118 right of 358; Wilson 95% interval 28.3% to 38.0% | 33.0% | 118 / 358 | a | +0.8 points |
| Gemma 4 12B Unified chat | Not applicable | – | – | – |
| Gemma 4 E4B chat | Not applicable | – | – | – |
| Qwen3.5-4B on · chat | Not applicable | – | – | – |
| Jev 1.14 | Partial | answered 356 / 358 | – | – |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | – | – | – |
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – | – |
| Gemma 4 31B chat | Not applicable | – | – | – |
| Gemma 4 E2B chat | Not applicable | – | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – | – |
| Qwen3.5-9B chat | Not applicable | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – | – |
| Muse Ember chat | Not applicable | – | – | – |
OCR route (D) · 306 Logic questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 306 Logic questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| d1-ultra | Not run | – | – | – |
| d3-ultra | Not run | – | – | – |
| Drex Max v1.1 | Not run | – | – | – |
| Gemini Decision 1 20261111 | Not run | – | – | – |
| Gemma 4 12B Unified chat | Not applicable | – | – | – |
| Gemma 4 E4B chat | Not applicable | – | – | – |
| GLiDE 27B | Not run | – | – | – |
| KnowLine-31B-Gen1.5 | Not run | – | – | – |
| Qwen3.5-4B on · chat | Not applicable | – | – | – |
| SubQ Snap 1.1 | Not run | – | – | – |
| Jev 1.14 | Not run | – | – | – |
| OneJev-27B | Not run | – | – | – |
| d1-2B | Not run | – | – | – |
| d1-3B | Not run | – | – | – |
| d1-Omni-2B | Not run | – | – | – |
| d3 | Not run | – | – | – |
| Drex Max | Not run | – | – | – |
| Gemma 4 31B chat | Not applicable | – | – | – |
| Gemma 4 E2B chat | Not applicable | – | – | – |
| KnowLine 4B Gen5.5 | Not run | – | – | – |
| KnowLine-0.8B-Gen1 | Not run | – | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – | – |
| OneJev1.6-0.8B | Not run | – | – | – |
| Qwen3.5-9B chat | Not applicable | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instant | Not run | – | – | – |
| StartLux-Decision-v1.1-2B-Instant | Not run | – | – | – |
| GPT-6 Luna | Not run | – | – | – |
| Muse Ember chat | Not applicable | – | – | – |
OCR route (D) · 52 Math questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 52 Math questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| d1-ultra | Not run | – | – | – |
| d3-ultra | Not run | – | – | – |
| Drex Max v1.1 | Not run | – | – | – |
| Gemini Decision 1 20261111 | Not run | – | – | – |
| Gemma 4 12B Unified chat | Not applicable | – | – | – |
| Gemma 4 E4B chat | Not applicable | – | – | – |
| GLiDE 27B | Not run | – | – | – |
| KnowLine-31B-Gen1.5 | Not run | – | – | – |
| Qwen3.5-4B on · chat | Not applicable | – | – | – |
| SubQ Snap 1.1 | Not run | – | – | – |
| Jev 1.14 | Not run | – | – | – |
| OneJev-27B | Not run | – | – | – |
| d1-2B | Not run | – | – | – |
| d1-3B | Not run | – | – | – |
| d1-Omni-2B | Not run | – | – | – |
| d3 | Not run | – | – | – |
| Drex Max | Not run | – | – | – |
| Gemma 4 31B chat | Not applicable | – | – | – |
| Gemma 4 E2B chat | Not applicable | – | – | – |
| KnowLine 4B Gen5.5 | Not run | – | – | – |
| KnowLine-0.8B-Gen1 | Not run | – | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – | – |
| OneJev1.6-0.8B | Not run | – | – | – |
| Qwen3.5-9B chat | Not applicable | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instant | Not run | – | – | – |
| StartLux-Decision-v1.1-2B-Instant | Not run | – | – | – |
| GPT-6 Luna | Not run | – | – | – |
| Muse Ember chat | Not applicable | – | – | – |
OCR route (D) · 358 Logic & Math questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 358 Logic & Math questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| d1-ultra | Not run | – | – | – |
| d3-ultra | Not run | – | – | – |
| Drex Max v1.1 | Not run | – | – | – |
| Gemini Decision 1 20261111 | Not run | – | – | – |
| Gemma 4 12B Unified chat | Not applicable | – | – | – |
| Gemma 4 E4B chat | Not applicable | – | – | – |
| GLiDE 27B | Not run | – | – | – |
| KnowLine-31B-Gen1.5 | Not run | – | – | – |
| Qwen3.5-4B on · chat | Not applicable | – | – | – |
| SubQ Snap 1.1 | Not run | – | – | – |
| Jev 1.14 | Not run | – | – | – |
| OneJev-27B | Not run | – | – | – |
| d1-2B | Not run | – | – | – |
| d1-3B | Not run | – | – | – |
| d1-Omni-2B | Not run | – | – | – |
| d3 | Not run | – | – | – |
| Drex Max | Not run | – | – | – |
| Gemma 4 31B chat | Not applicable | – | – | – |
| Gemma 4 E2B chat | Not applicable | – | – | – |
| KnowLine 4B Gen5.5 | Not run | – | – | – |
| KnowLine-0.8B-Gen1 | Not run | – | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – | – |
| OneJev1.6-0.8B | Not run | – | – | – |
| Qwen3.5-9B chat | Not applicable | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instant | Not run | – | – | – |
| StartLux-Decision-v1.1-2B-Instant | Not run | – | – | – |
| GPT-6 Luna | Not run | – | – | – |
| Muse Ember chat | Not applicable | – | – | – |
Every setup of the track
28 setups of the models chosen; scored only when a cell is complete (306 Logic, 52 Math).
| Modelstate | Answers storedLogic · Math |
|---|---|
| d1-ultraPublished | A text only: 306/306 · 52/52B picture alone: 306/306 · 52/52C text + picture: 306/306 · 52/52D ocr route: 0/306 · 0/52 |
| d3-ultraPublished | A text only: 306/306 · 52/52B picture alone: 306/306 · 52/52C text + picture: 306/306 · 52/52D ocr route: 0/306 · 0/52 |
| Drex Max v1.1Published | A text only: 306/306 · 52/52B picture alone: 306/306 · 52/52C text + picture: 306/306 · 52/52D ocr route: 0/306 · 0/52 |
| Gemini Decision 1 20261111Published | A text only: 306/306 · 52/52B picture alone: 306/306 · 52/52C text + picture: 306/306 · 52/52D ocr route: 0/306 · 0/52 |
| Gemma 4 12B Unified chatPublished | A text only: not applicable (B only)B picture alone: 306/306 · 52/52C text + picture: not applicable (B only)D ocr route: not applicable (no OCR route for chat setups) |
| Gemma 4 E4B chatPublished | A text only: not applicable (B only)B picture alone: 306/306 · 52/52C text + picture: not applicable (B only)D ocr route: not applicable (no OCR route for chat setups) |
| GLiDE 27BPublished | A text only: 306/306 · 52/52B picture alone: 306/306 · 52/52C text + picture: 306/306 · 52/52D ocr route: 0/306 · 0/52 |
| KnowLine-31B-Gen1.5Published | A text only: 306/306 · 52/52B picture alone: 306/306 · 52/52C text + picture: 306/306 · 52/52D ocr route: 0/306 · 0/52 |
| Qwen3.5-4B on · chatPublished | A text only: not applicable (B only)B picture alone: 306/306 · 52/52C text + picture: not applicable (B only)D ocr route: not applicable (no OCR route for chat setups) |
| SubQ Snap 1.1Published | A text only: 306/306 · 52/52B picture alone: 306/306 · 52/52C text + picture: 306/306 · 52/52D ocr route: 0/306 · 0/52 |
| Jev 1.14Partly runLogic B and C not measured (304/306 accepted): 2 crops refused by the provider, 36×280 px, below TypeSafe's 280 px minimum side; A and Math run in full (Zeki #5277) | A text only: 306/306 · 52/52B picture alone: 304/306 · 52/52C text + picture: 304/306 · 52/52D ocr route: 0/306 · 0/52 |
| OneJev-27BPausedstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | A text only: 136/306 · 0/52B picture alone: 0/306 · 0/52C text + picture: 0/306 · 0/52D ocr route: 0/306 · 0/52 |
| d1-2BQueuednew d1 sizes; needs a card (bariton: no d1 environment on spark2) | none stored yet |
| d1-3BQueuednew d1 sizes; needs a card (bariton: no d1 environment on spark2) | none stored yet |
| d1-Omni-2BQueuednew d1 sizes; needs a card (bariton: no d1 environment on spark2) | none stored yet |
| d3Queuedserved setup prepared on spark2; needs a card | none stored yet |
| Drex MaxQueuedserved setup prepared on spark2; needs a card | none stored yet |
| Gemma 4 31B chatQueuedneeds a card | none stored yet |
| Gemma 4 E2B chatQueuedneeds a card | none stored yet |
| KnowLine 4B Gen5.5Queuedlocal setup; needs a card (spark2) | none stored yet |
| KnowLine-0.8B-Gen1Queuedlocal setup; needs a card (spark2) | none stored yet |
| Muse Glimmer 30B chatQueuedJob 2 phase | none stored yet |
| OneJev1.6-0.8BQueuedlocal setup; needs a card (spark2) | none stored yet |
| Qwen3.5-9B chatQueuedwaits for its server window | none stored yet |
| StartLux-Decision-v1.1-0.8B-InstantQueuedlocal setup; needs a card (spark2) | none stored yet |
| StartLux-Decision-v1.1-2B-InstantQueuedlocal setup; needs a card (spark2) | none stored yet |
| GPT-6 LunaBlockedAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | none stored yet |
| Muse Ember chatBlockedno qualified Logic & Math chat adapter for its hosted route, and no L&M budget for it | none stored yet |
The picture alone (B) · 306 Logic questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 9 queued · Table
Alibaba Chance, 20% (five options)
The picture alone (B) · 306 Logic questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| Qwen3.5-4B on · chattie group a Qwen3.5-4B (on · chat): 62.1%, 190 right of 306; Wilson 95% interval 56.5% to 67.3% | 62.1% | 190 / 306 | a | – |
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| Gemma 4 E2B chatneeds a card | Queued | – | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
The picture alone (B) · 52 Math questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 9 queued · Table
Alibaba Chance, 20% (five options)
The picture alone (B) · 52 Math questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| Qwen3.5-4B on · chattie group a Qwen3.5-4B (on · chat): 86.5%, 45 right of 52; Wilson 95% interval 74.7% to 93.3% | 86.5% | 45 / 52 | a | – |
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| Gemma 4 E2B chatneeds a card | Queued | – | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
The picture alone (B) · 358 Logic & Math questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 9 queued · Table
Alibaba Chance, 20% (five options)
The picture alone (B) · 358 Logic & Math questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| Qwen3.5-4B on · chattie group a Qwen3.5-4B (on · chat): 65.6%, 235 right of 358; Wilson 95% interval 60.6% to 70.4% | 65.6% | 235 / 358 | a | – |
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| Gemma 4 E2B chatneeds a card | Queued | – | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
Text only, the control (A) · 306 Logic questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (8 queued, 2 not applicable). Every setup in the table
Text only, the control (A) · 306 Logic questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| Qwen3.5-4B on · chat | Not applicable | – | – |
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| Gemma 4 E2B chat | Not applicable | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – |
Text only, the control (A) · 52 Math questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (8 queued, 2 not applicable). Every setup in the table
Text only, the control (A) · 52 Math questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| Qwen3.5-4B on · chat | Not applicable | – | – |
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| Gemma 4 E2B chat | Not applicable | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – |
Text only, the control (A) · 358 Logic & Math questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (8 queued, 2 not applicable). Every setup in the table
Text only, the control (A) · 358 Logic & Math questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| Qwen3.5-4B on · chat | Not applicable | – | – |
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| Gemma 4 E2B chat | Not applicable | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – |
Text plus picture (C) · 306 Logic questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (8 queued, 2 not applicable). Every setup in the table
Text plus picture (C) · 306 Logic questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| Qwen3.5-4B on · chat | Not applicable | – | – | – |
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| Gemma 4 E2B chat | Not applicable | – | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
Text plus picture (C) · 52 Math questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (8 queued, 2 not applicable). Every setup in the table
Text plus picture (C) · 52 Math questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| Qwen3.5-4B on · chat | Not applicable | – | – | – |
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| Gemma 4 E2B chat | Not applicable | – | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
Text plus picture (C) · 358 Logic & Math questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (8 queued, 2 not applicable). Every setup in the table
Text plus picture (C) · 358 Logic & Math questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| Qwen3.5-4B on · chat | Not applicable | – | – | – |
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| Gemma 4 E2B chat | Not applicable | – | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
OCR route (D) · 306 Logic questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 306 Logic questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| Qwen3.5-4B on · chat | Not applicable | – | – | – |
| d1-2B | Not run | – | – | – |
| d1-3B | Not run | – | – | – |
| d1-Omni-2B | Not run | – | – | – |
| Gemma 4 E2B chat | Not applicable | – | – | – |
| KnowLine 4B Gen5.5 | Not run | – | – | – |
| KnowLine-0.8B-Gen1 | Not run | – | – | – |
| OneJev1.6-0.8B | Not run | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instant | Not run | – | – | – |
| StartLux-Decision-v1.1-2B-Instant | Not run | – | – | – |
OCR route (D) · 52 Math questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 52 Math questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| Qwen3.5-4B on · chat | Not applicable | – | – | – |
| d1-2B | Not run | – | – | – |
| d1-3B | Not run | – | – | – |
| d1-Omni-2B | Not run | – | – | – |
| Gemma 4 E2B chat | Not applicable | – | – | – |
| KnowLine 4B Gen5.5 | Not run | – | – | – |
| KnowLine-0.8B-Gen1 | Not run | – | – | – |
| OneJev1.6-0.8B | Not run | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instant | Not run | – | – | – |
| StartLux-Decision-v1.1-2B-Instant | Not run | – | – | – |
OCR route (D) · 358 Logic & Math questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 358 Logic & Math questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| Qwen3.5-4B on · chat | Not applicable | – | – | – |
| d1-2B | Not run | – | – | – |
| d1-3B | Not run | – | – | – |
| d1-Omni-2B | Not run | – | – | – |
| Gemma 4 E2B chat | Not applicable | – | – | – |
| KnowLine 4B Gen5.5 | Not run | – | – | – |
| KnowLine-0.8B-Gen1 | Not run | – | – | – |
| OneJev1.6-0.8B | Not run | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instant | Not run | – | – | – |
| StartLux-Decision-v1.1-2B-Instant | Not run | – | – | – |
Every setup of the track
10 setups of the models chosen; scored only when a cell is complete (306 Logic, 52 Math).
| Modelstate | Answers storedLogic · Math |
|---|---|
| Qwen3.5-4B on · chatPublished | A text only: not applicable (B only)B picture alone: 306/306 · 52/52C text + picture: not applicable (B only)D ocr route: not applicable (no OCR route for chat setups) |
| d1-2BQueuednew d1 sizes; needs a card (bariton: no d1 environment on spark2) | none stored yet |
| d1-3BQueuednew d1 sizes; needs a card (bariton: no d1 environment on spark2) | none stored yet |
| d1-Omni-2BQueuednew d1 sizes; needs a card (bariton: no d1 environment on spark2) | none stored yet |
| Gemma 4 E2B chatQueuedneeds a card | none stored yet |
| KnowLine 4B Gen5.5Queuedlocal setup; needs a card (spark2) | none stored yet |
| KnowLine-0.8B-Gen1Queuedlocal setup; needs a card (spark2) | none stored yet |
| OneJev1.6-0.8BQueuedlocal setup; needs a card (spark2) | none stored yet |
| StartLux-Decision-v1.1-0.8B-InstantQueuedlocal setup; needs a card (spark2) | none stored yet |
| StartLux-Decision-v1.1-2B-InstantQueuedlocal setup; needs a card (spark2) | none stored yet |
The picture alone (B) · 306 Logic questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 queued · Table
Google Chance, 20% (five options)
The picture alone (B) · 306 Logic questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 12B Unified chattie group a Gemma 4 12B Unified (chat): 56.5%, 173 right of 306; Wilson 95% interval 50.9% to 62.0% | 56.5% | 173 / 306 | a | – |
| Gemma 4 E4B chattie group b Gemma 4 E4B (chat): 38.2%, 117 right of 306; Wilson 95% interval 33.0% to 43.8% | 38.2% | 117 / 306 | b | – |
| Qwen3.5-9B chatwaits for its server window | Queued | – | – | – |
The picture alone (B) · 52 Math questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 queued · Table
Google Chance, 20% (five options)
The picture alone (B) · 52 Math questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 12B Unified chattie group a Gemma 4 12B Unified (chat): 86.5%, 45 right of 52; Wilson 95% interval 74.7% to 93.3% | 86.5% | 45 / 52 | a | – |
| Gemma 4 E4B chattie group b Gemma 4 E4B (chat): 53.8%, 28 right of 52; Wilson 95% interval 40.5% to 66.7% | 53.8% | 28 / 52 | b | – |
| Qwen3.5-9B chatwaits for its server window | Queued | – | – | – |
The picture alone (B) · 358 Logic & Math questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 queued · Table
Google Chance, 20% (five options)
The picture alone (B) · 358 Logic & Math questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 12B Unified chattie group a Gemma 4 12B Unified (chat): 60.9%, 218 right of 358; Wilson 95% interval 55.7% to 65.8% | 60.9% | 218 / 358 | a | – |
| Gemma 4 E4B chattie group b Gemma 4 E4B (chat): 40.5%, 145 right of 358; Wilson 95% interval 35.5% to 45.7% | 40.5% | 145 / 358 | b | – |
| Qwen3.5-9B chatwaits for its server window | Queued | – | – | – |
Text only, the control (A) · 306 Logic questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (3 not applicable). Every setup in the table
Text only, the control (A) · 306 Logic questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| Gemma 4 12B Unified chat | Not applicable | – | – |
| Gemma 4 E4B chat | Not applicable | – | – |
| Qwen3.5-9B chat | Not applicable | – | – |
Text only, the control (A) · 52 Math questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (3 not applicable). Every setup in the table
Text only, the control (A) · 52 Math questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| Gemma 4 12B Unified chat | Not applicable | – | – |
| Gemma 4 E4B chat | Not applicable | – | – |
| Qwen3.5-9B chat | Not applicable | – | – |
Text only, the control (A) · 358 Logic & Math questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (3 not applicable). Every setup in the table
Text only, the control (A) · 358 Logic & Math questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| Gemma 4 12B Unified chat | Not applicable | – | – |
| Gemma 4 E4B chat | Not applicable | – | – |
| Qwen3.5-9B chat | Not applicable | – | – |
Text plus picture (C) · 306 Logic questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (3 not applicable). Every setup in the table
Text plus picture (C) · 306 Logic questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 12B Unified chat | Not applicable | – | – | – |
| Gemma 4 E4B chat | Not applicable | – | – | – |
| Qwen3.5-9B chat | Not applicable | – | – | – |
Text plus picture (C) · 52 Math questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (3 not applicable). Every setup in the table
Text plus picture (C) · 52 Math questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 12B Unified chat | Not applicable | – | – | – |
| Gemma 4 E4B chat | Not applicable | – | – | – |
| Qwen3.5-9B chat | Not applicable | – | – | – |
Text plus picture (C) · 358 Logic & Math questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (3 not applicable). Every setup in the table
Text plus picture (C) · 358 Logic & Math questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 12B Unified chat | Not applicable | – | – | – |
| Gemma 4 E4B chat | Not applicable | – | – | – |
| Qwen3.5-9B chat | Not applicable | – | – | – |
OCR route (D) · 306 Logic questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 306 Logic questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 12B Unified chat | Not applicable | – | – | – |
| Gemma 4 E4B chat | Not applicable | – | – | – |
| Qwen3.5-9B chat | Not applicable | – | – | – |
OCR route (D) · 52 Math questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 52 Math questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 12B Unified chat | Not applicable | – | – | – |
| Gemma 4 E4B chat | Not applicable | – | – | – |
| Qwen3.5-9B chat | Not applicable | – | – | – |
OCR route (D) · 358 Logic & Math questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 358 Logic & Math questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 12B Unified chat | Not applicable | – | – | – |
| Gemma 4 E4B chat | Not applicable | – | – | – |
| Qwen3.5-9B chat | Not applicable | – | – | – |
Every setup of the track
3 setups of the models chosen; scored only when a cell is complete (306 Logic, 52 Math).
| Modelstate | Answers storedLogic · Math |
|---|---|
| Gemma 4 12B Unified chatPublished | A text only: not applicable (B only)B picture alone: 306/306 · 52/52C text + picture: not applicable (B only)D ocr route: not applicable (no OCR route for chat setups) |
| Gemma 4 E4B chatPublished | A text only: not applicable (B only)B picture alone: 306/306 · 52/52C text + picture: not applicable (B only)D ocr route: not applicable (no OCR route for chat setups) |
| Qwen3.5-9B chatQueuedwaits for its server window | none stored yet |
The picture alone (B) · 306 Logic questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 4 queued · Table
PelaAI Fastino vLLM Semantic Router Chance, 20% (five options)
The picture alone (B) · 306 Logic questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| KnowLine-31B-Gen1.5 Run 2Run 2 of 12 Nov; first run 99/306 kept on Experiments · tie group a · +0.3 points vs A KnowLine-31B-Gen1.5 (Run 2): 32.0%, 98 right of 306; Wilson 95% interval 27.0% to 37.4%; Run 2 of 12 Nov; first run 99/306 kept on Experiments | 32.0% | 98 / 306 | a | +0.3 points |
| GLiDE 27Btie group a · +2.0 points vs A GLiDE 27B: 30.7%, 94 right of 306; Wilson 95% interval 25.8% to 36.1% | 30.7% | 94 / 306 | a | +2.0 points |
| d3-ultratie group a · −2.3 points vs A d3-ultra: 29.1%, 89 right of 306; Wilson 95% interval 24.3% to 34.4% | 29.1% | 89 / 306 | a | −2.3 points |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | – | – | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – | – |
| Gemma 4 31B chatneeds a card | Queued | – | – | – |
| Muse Glimmer 30B chatJob 2 phase | Queued | – | – | – |
The picture alone (B) · 52 Math questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 4 queued · Table
Fastino vLLM Semantic Router PelaAI Chance, 20% (five options)
The picture alone (B) · 52 Math questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| GLiDE 27Btie group a · −1.9 points vs A GLiDE 27B: 38.5%, 20 right of 52; Wilson 95% interval 26.5% to 52.0% | 38.5% | 20 / 52 | a | −1.9 points |
| d3-ultratie group a · −1.9 points vs A d3-ultra: 36.5%, 19 right of 52; Wilson 95% interval 24.8% to 50.1% | 36.5% | 19 / 52 | a | −1.9 points |
| KnowLine-31B-Gen1.5tie group a · −1.9 points vs A KnowLine-31B-Gen1.5: 36.5%, 19 right of 52; Wilson 95% interval 24.8% to 50.1% | 36.5% | 19 / 52 | a | −1.9 points |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | – | – | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – | – |
| Gemma 4 31B chatneeds a card | Queued | – | – | – |
| Muse Glimmer 30B chatJob 2 phase | Queued | – | – | – |
The picture alone (B) · 358 Logic & Math questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 4 queued · Table
PelaAI Fastino vLLM Semantic Router Chance, 20% (five options)
The picture alone (B) · 358 Logic & Math questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| KnowLine-31B-Gen1.5 first runsfirst runs (Logic 99 + Math 19); no complete Run 2 of B on Math yet, so run numbers are not mixed · tie group a · +0.3 points vs A KnowLine-31B-Gen1.5 (first runs): 33.0%, 118 right of 358; Wilson 95% interval 28.3% to 38.0%; first runs (Logic 99 + Math 19); no complete Run 2 of B on Math yet, so run numbers are not mixed | 33.0% | 118 / 358 | a | +0.3 points |
| GLiDE 27Btie group a · +1.4 points vs A GLiDE 27B: 31.8%, 114 right of 358; Wilson 95% interval 27.2% to 36.8% | 31.8% | 114 / 358 | a | +1.4 points |
| d3-ultratie group a · −2.2 points vs A d3-ultra: 30.2%, 108 right of 358; Wilson 95% interval 25.6% to 35.1% | 30.2% | 108 / 358 | a | −2.2 points |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | – | – | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – | – |
| Gemma 4 31B chatneeds a card | Queued | – | – | – |
| Muse Glimmer 30B chatJob 2 phase | Queued | – | – | – |
Text only, the control (A) · 306 Logic questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 2 queued, 2 not applicable · Table
PelaAI vLLM Semantic Router Fastino Chance, 20% (five options)
Text only, the control (A) · 306 Logic questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| KnowLine-31B-Gen1.5tie group a KnowLine-31B-Gen1.5: 31.7%, 97 right of 306; Wilson 95% interval 26.7% to 37.1% | 31.7% | 97 / 306 | a |
| d3-ultratie group a d3-ultra: 31.4%, 96 right of 306; Wilson 95% interval 26.4% to 36.8% | 31.4% | 96 / 306 | a |
| GLiDE 27Btie group a GLiDE 27B: 28.8%, 88 right of 306; Wilson 95% interval 24.0% to 34.1% | 28.8% | 88 / 306 | a |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | answered 136 / 306 | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – |
| Gemma 4 31B chat | Not applicable | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – |
Text only, the control (A) · 52 Math questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 2 queued, 2 not applicable · Table
Fastino vLLM Semantic Router PelaAI Chance, 20% (five options)
Text only, the control (A) · 52 Math questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| GLiDE 27Btie group a GLiDE 27B: 40.4%, 21 right of 52; Wilson 95% interval 28.2% to 53.9% | 40.4% | 21 / 52 | a |
| d3-ultratie group a d3-ultra: 38.5%, 20 right of 52; Wilson 95% interval 26.5% to 52.0% | 38.5% | 20 / 52 | a |
| KnowLine-31B-Gen1.5tie group a KnowLine-31B-Gen1.5: 38.5%, 20 right of 52; Wilson 95% interval 26.5% to 52.0% | 38.5% | 20 / 52 | a |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | – | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – |
| Gemma 4 31B chat | Not applicable | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – |
Text only, the control (A) · 358 Logic & Math questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 2 queued, 2 not applicable · Table
PelaAI vLLM Semantic Router Fastino Chance, 20% (five options)
Text only, the control (A) · 358 Logic & Math questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| KnowLine-31B-Gen1.5tie group a KnowLine-31B-Gen1.5: 32.7%, 117 right of 358; Wilson 95% interval 28.0% to 37.7% | 32.7% | 117 / 358 | a |
| d3-ultratie group a d3-ultra: 32.4%, 116 right of 358; Wilson 95% interval 27.8% to 37.4% | 32.4% | 116 / 358 | a |
| GLiDE 27Btie group a GLiDE 27B: 30.4%, 109 right of 358; Wilson 95% interval 25.9% to 35.4% | 30.4% | 109 / 358 | a |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | answered 136 / 358 | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – |
| Gemma 4 31B chat | Not applicable | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – |
Text plus picture (C) · 306 Logic questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 2 queued, 2 not applicable · Table
Fastino PelaAI vLLM Semantic Router Chance, 20% (five options)
Text plus picture (C) · 306 Logic questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| GLiDE 27Btie group a · +9.8 points vs A GLiDE 27B: 38.6%, 118 right of 306; Wilson 95% interval 33.3% to 44.1% | 38.6% | 118 / 306 | a | +9.8 points |
| KnowLine-31B-Gen1.5tie group a · +5.6 points vs A KnowLine-31B-Gen1.5: 37.3%, 114 right of 306; Wilson 95% interval 32.0% to 42.8% | 37.3% | 114 / 306 | a | +5.6 points |
| d3-ultratie group a · +0.3 points vs A d3-ultra: 31.7%, 97 right of 306; Wilson 95% interval 26.7% to 37.1% | 31.7% | 97 / 306 | a | +0.3 points |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | – | – | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – | – |
| Gemma 4 31B chat | Not applicable | – | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – | – |
Text plus picture (C) · 52 Math questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 2 queued, 2 not applicable · Table
vLLM Semantic Router Fastino PelaAI Chance, 20% (five options)
Text plus picture (C) · 52 Math questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| d3-ultratie group a · +9.6 points vs A d3-ultra: 48.1%, 25 right of 52; Wilson 95% interval 35.1% to 61.3% | 48.1% | 25 / 52 | a | +9.6 points |
| GLiDE 27Btie group a · +5.8 points vs A GLiDE 27B: 46.2%, 24 right of 52; Wilson 95% interval 33.3% to 59.5% | 46.2% | 24 / 52 | a | +5.8 points |
| KnowLine-31B-Gen1.5tie group a · +7.7 points vs A KnowLine-31B-Gen1.5: 46.2%, 24 right of 52; Wilson 95% interval 33.3% to 59.5% | 46.2% | 24 / 52 | a | +7.7 points |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | – | – | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – | – |
| Gemma 4 31B chat | Not applicable | – | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – | – |
Text plus picture (C) · 358 Logic & Math questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 2 queued, 2 not applicable · Table
Fastino PelaAI vLLM Semantic Router Chance, 20% (five options)
Text plus picture (C) · 358 Logic & Math questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| GLiDE 27Btie group a · +9.2 points vs A GLiDE 27B: 39.7%, 142 right of 358; Wilson 95% interval 34.7% to 44.8% | 39.7% | 142 / 358 | a | +9.2 points |
| KnowLine-31B-Gen1.5tie group a · +5.9 points vs A KnowLine-31B-Gen1.5: 38.5%, 138 right of 358; Wilson 95% interval 33.7% to 43.7% | 38.5% | 138 / 358 | a | +5.9 points |
| d3-ultratie group a · +1.7 points vs A d3-ultra: 34.1%, 122 right of 358; Wilson 95% interval 29.4% to 39.1% | 34.1% | 122 / 358 | a | +1.7 points |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | – | – | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – | – |
| Gemma 4 31B chat | Not applicable | – | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – | – |
OCR route (D) · 306 Logic questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 306 Logic questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| d3-ultra | Not run | – | – | – |
| GLiDE 27B | Not run | – | – | – |
| KnowLine-31B-Gen1.5 | Not run | – | – | – |
| OneJev-27B | Not run | – | – | – |
| d3 | Not run | – | – | – |
| Drex Max | Not run | – | – | – |
| Gemma 4 31B chat | Not applicable | – | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – | – |
OCR route (D) · 52 Math questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 52 Math questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| d3-ultra | Not run | – | – | – |
| GLiDE 27B | Not run | – | – | – |
| KnowLine-31B-Gen1.5 | Not run | – | – | – |
| OneJev-27B | Not run | – | – | – |
| d3 | Not run | – | – | – |
| Drex Max | Not run | – | – | – |
| Gemma 4 31B chat | Not applicable | – | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – | – |
OCR route (D) · 358 Logic & Math questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 358 Logic & Math questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| d3-ultra | Not run | – | – | – |
| GLiDE 27B | Not run | – | – | – |
| KnowLine-31B-Gen1.5 | Not run | – | – | – |
| OneJev-27B | Not run | – | – | – |
| d3 | Not run | – | – | – |
| Drex Max | Not run | – | – | – |
| Gemma 4 31B chat | Not applicable | – | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – | – |
Every setup of the track
8 setups of the models chosen; scored only when a cell is complete (306 Logic, 52 Math).
| Modelstate | Answers storedLogic · Math |
|---|---|
| d3-ultraPublished | A text only: 306/306 · 52/52B picture alone: 306/306 · 52/52C text + picture: 306/306 · 52/52D ocr route: 0/306 · 0/52 |
| GLiDE 27BPublished | A text only: 306/306 · 52/52B picture alone: 306/306 · 52/52C text + picture: 306/306 · 52/52D ocr route: 0/306 · 0/52 |
| KnowLine-31B-Gen1.5Published | A text only: 306/306 · 52/52B picture alone: 306/306 · 52/52C text + picture: 306/306 · 52/52D ocr route: 0/306 · 0/52 |
| OneJev-27BPausedstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | A text only: 136/306 · 0/52B picture alone: 0/306 · 0/52C text + picture: 0/306 · 0/52D ocr route: 0/306 · 0/52 |
| d3Queuedserved setup prepared on spark2; needs a card | none stored yet |
| Drex MaxQueuedserved setup prepared on spark2; needs a card | none stored yet |
| Gemma 4 31B chatQueuedneeds a card | none stored yet |
| Muse Glimmer 30B chatQueuedJob 2 phase | none stored yet |
No setup of this class on the Vision track.
The picture alone (B) · 306 Logic questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 2 blocked, 1 partial · Table
Google Nace.AI Subquadratic Liquid AI Chance, 20% (five options)
The picture alone (B) · 306 Logic questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| Gemini Decision 1 20261111tie group a · +2.3 points vs A Gemini Decision 1 (20261111): 33.7%, 103 right of 306; Wilson 95% interval 28.6% to 39.1% | 33.7% | 103 / 306 | a | +2.3 points |
| Drex Max v1.1tie group a · −1.6 points vs A Drex Max v1.1: 29.7%, 91 right of 306; Wilson 95% interval 24.9% to 35.1% | 29.7% | 91 / 306 | a | −1.6 points |
| SubQ Snap 1.1tie group a · +4.2 points vs A SubQ Snap 1.1: 28.4%, 87 right of 306; Wilson 95% interval 23.7% to 33.7% | 28.4% | 87 / 306 | a | +4.2 points |
| d1-ultratie group a · −1.3 points vs A d1-ultra: 27.5%, 84 right of 306; Wilson 95% interval 22.8% to 32.7% | 27.5% | 84 / 306 | a | −1.3 points |
| Jev 1.14 | Partial | answered 304 / 306 | – | – |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – | – |
| Muse Ember chatno qualified Logic & Math chat adapter for its hosted route, and no L&M budget for it | Blocked | – | – | – |
The picture alone (B) · 52 Math questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 2 blocked · Table
Liquid AI Google Subquadratic Nace.AI TypeSafe Chance, 20% (five options)
The picture alone (B) · 52 Math questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| d1-ultratie group a · +13.5 points vs A d1-ultra: 51.9%, 27 right of 52; Wilson 95% interval 38.7% to 64.9% | 51.9% | 27 / 52 | a | +13.5 points |
| Gemini Decision 1 20261111tie group a · +1.9 points vs A Gemini Decision 1 (20261111): 46.2%, 24 right of 52; Wilson 95% interval 33.3% to 59.5% | 46.2% | 24 / 52 | a | +1.9 points |
| SubQ Snap 1.1tie group a · −1.9 points vs A SubQ Snap 1.1: 42.3%, 22 right of 52; Wilson 95% interval 29.9% to 55.8% | 42.3% | 22 / 52 | a | −1.9 points |
| Drex Max v1.1tie group a · +3.8 points vs A Drex Max v1.1: 40.4%, 21 right of 52; Wilson 95% interval 28.2% to 53.9% | 40.4% | 21 / 52 | a | +3.8 points |
| Jev 1.14tie group a · −3.8 points vs A Jev 1.14: 40.4%, 21 right of 52; Wilson 95% interval 28.2% to 53.9% | 40.4% | 21 / 52 | a | −3.8 points |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – | – |
| Muse Ember chatno qualified Logic & Math chat adapter for its hosted route, and no L&M budget for it | Blocked | – | – | – |
The picture alone (B) · 358 Logic & Math questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 2 blocked, 1 partial · Table
Google Nace.AI Liquid AI Subquadratic Chance, 20% (five options)
The picture alone (B) · 358 Logic & Math questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| Gemini Decision 1 20261111tie group a · +2.2 points vs A Gemini Decision 1 (20261111): 35.5%, 127 right of 358; Wilson 95% interval 30.7% to 40.6% | 35.5% | 127 / 358 | a | +2.2 points |
| Drex Max v1.1tie group a · −0.8 points vs A Drex Max v1.1: 31.3%, 112 right of 358; Wilson 95% interval 26.7% to 36.3% | 31.3% | 112 / 358 | a | −0.8 points |
| d1-ultratie group a · +0.8 points vs A d1-ultra: 31.0%, 111 right of 358; Wilson 95% interval 26.4% to 36.0% | 31.0% | 111 / 358 | a | +0.8 points |
| SubQ Snap 1.1tie group a · +3.4 points vs A SubQ Snap 1.1: 30.4%, 109 right of 358; Wilson 95% interval 25.9% to 35.4% | 30.4% | 109 / 358 | a | +3.4 points |
| Jev 1.14 | Partial | answered 356 / 358 | – | – |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – | – |
| Muse Ember chatno qualified Logic & Math chat adapter for its hosted route, and no L&M budget for it | Blocked | – | – | – |
Text only, the control (A) · 306 Logic questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 blocked, 1 not applicable · Table
Nace.AI Google Liquid AI TypeSafe Subquadratic Chance, 20% (five options)
Text only, the control (A) · 306 Logic questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| Drex Max v1.1tie group a Drex Max v1.1: 31.4%, 96 right of 306; Wilson 95% interval 26.4% to 36.8% | 31.4% | 96 / 306 | a |
| Gemini Decision 1 20261111tie group a Gemini Decision 1 (20261111): 31.4%, 96 right of 306; Wilson 95% interval 26.4% to 36.8% | 31.4% | 96 / 306 | a |
| d1-ultratie group a d1-ultra: 28.8%, 88 right of 306; Wilson 95% interval 24.0% to 34.1% | 28.8% | 88 / 306 | a |
| Jev 1.14tie group a Jev 1.14: 25.2%, 77 right of 306; Wilson 95% interval 20.6% to 30.3% | 25.2% | 77 / 306 | a |
| SubQ Snap 1.1tie group a SubQ Snap 1.1: 24.2%, 74 right of 306; Wilson 95% interval 19.7% to 29.3% | 24.2% | 74 / 306 | a |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – |
| Muse Ember chat | Not applicable | – | – |
Text only, the control (A) · 52 Math questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 blocked, 1 not applicable · Table
Google TypeSafe Subquadratic Liquid AI Nace.AI Chance, 20% (five options)
Text only, the control (A) · 52 Math questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| Gemini Decision 1 20261111tie group a Gemini Decision 1 (20261111): 44.2%, 23 right of 52; Wilson 95% interval 31.6% to 57.7% | 44.2% | 23 / 52 | a |
| Jev 1.14tie group a Jev 1.14: 44.2%, 23 right of 52; Wilson 95% interval 31.6% to 57.7% | 44.2% | 23 / 52 | a |
| SubQ Snap 1.1tie group a SubQ Snap 1.1: 44.2%, 23 right of 52; Wilson 95% interval 31.6% to 57.7% | 44.2% | 23 / 52 | a |
| d1-ultratie group a d1-ultra: 38.5%, 20 right of 52; Wilson 95% interval 26.5% to 52.0% | 38.5% | 20 / 52 | a |
| Drex Max v1.1tie group a Drex Max v1.1: 36.5%, 19 right of 52; Wilson 95% interval 24.8% to 50.1% | 36.5% | 19 / 52 | a |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – |
| Muse Ember chat | Not applicable | – | – |
Text only, the control (A) · 358 Logic & Math questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 blocked, 1 not applicable · Table
Google Nace.AI Liquid AI TypeSafe Subquadratic Chance, 20% (five options)
Text only, the control (A) · 358 Logic & Math questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| Gemini Decision 1 20261111tie group a Gemini Decision 1 (20261111): 33.2%, 119 right of 358; Wilson 95% interval 28.6% to 38.3% | 33.2% | 119 / 358 | a |
| Drex Max v1.1tie group a Drex Max v1.1: 32.1%, 115 right of 358; Wilson 95% interval 27.5% to 37.1% | 32.1% | 115 / 358 | a |
| d1-ultratie group a d1-ultra: 30.2%, 108 right of 358; Wilson 95% interval 25.6% to 35.1% | 30.2% | 108 / 358 | a |
| Jev 1.14tie group a Jev 1.14: 27.9%, 100 right of 358; Wilson 95% interval 23.5% to 32.8% | 27.9% | 100 / 358 | a |
| SubQ Snap 1.1tie group a SubQ Snap 1.1: 27.1%, 97 right of 358; Wilson 95% interval 22.8% to 31.9% | 27.1% | 97 / 358 | a |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – |
| Muse Ember chat | Not applicable | – | – |
Text plus picture (C) · 306 Logic questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 blocked, 1 partial, 1 not applicable · Table
Liquid AI Google Subquadratic Nace.AI Chance, 20% (five options)
Text plus picture (C) · 306 Logic questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| d1-ultratie group a · +6.9 points vs A d1-ultra: 35.6%, 109 right of 306; Wilson 95% interval 30.5% to 41.1% | 35.6% | 109 / 306 | a | +6.9 points |
| Gemini Decision 1 20261111tie group a · +3.3 points vs A Gemini Decision 1 (20261111): 34.6%, 106 right of 306; Wilson 95% interval 29.5% to 40.1% | 34.6% | 106 / 306 | a | +3.3 points |
| SubQ Snap 1.1tie group a · +9.5 points vs A SubQ Snap 1.1: 33.7%, 103 right of 306; Wilson 95% interval 28.6% to 39.1% | 33.7% | 103 / 306 | a | +9.5 points |
| Drex Max v1.1tie group a · +1.6 points vs A Drex Max v1.1: 33.0%, 101 right of 306; Wilson 95% interval 28.0% to 38.5% | 33.0% | 101 / 306 | a | +1.6 points |
| Jev 1.14 | Partial | answered 304 / 306 | – | – |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – | – |
| Muse Ember chat | Not applicable | – | – | – |
Text plus picture (C) · 52 Math questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 blocked, 1 not applicable · Table
TypeSafe Google Liquid AI Subquadratic Nace.AI Chance, 20% (five options)
Text plus picture (C) · 52 Math questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| Jev 1.14tie group a · +7.7 points vs A Jev 1.14: 51.9%, 27 right of 52; Wilson 95% interval 38.7% to 64.9% | 51.9% | 27 / 52 | a | +7.7 points |
| Gemini Decision 1 20261111tie group a · +3.8 points vs A Gemini Decision 1 (20261111): 48.1%, 25 right of 52; Wilson 95% interval 35.1% to 61.3% | 48.1% | 25 / 52 | a | +3.8 points |
| d1-ultratie group a · +7.7 points vs A d1-ultra: 46.2%, 24 right of 52; Wilson 95% interval 33.3% to 59.5% | 46.2% | 24 / 52 | a | +7.7 points |
| SubQ Snap 1.1tie group a · 0.0 points vs A SubQ Snap 1.1: 44.2%, 23 right of 52; Wilson 95% interval 31.6% to 57.7% | 44.2% | 23 / 52 | a | 0.0 points |
| Drex Max v1.1tie group a · −3.8 points vs A Drex Max v1.1: 32.7%, 17 right of 52; Wilson 95% interval 21.5% to 46.2% | 32.7% | 17 / 52 | a | −3.8 points |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – | – |
| Muse Ember chat | Not applicable | – | – | – |
Text plus picture (C) · 358 Logic & Math questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 blocked, 1 partial, 1 not applicable · Table
Liquid AI Google Subquadratic Nace.AI Chance, 20% (five options)
Text plus picture (C) · 358 Logic & Math questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| d1-ultratie group a · +7.0 points vs A d1-ultra: 37.2%, 133 right of 358; Wilson 95% interval 32.3% to 42.3% | 37.2% | 133 / 358 | a | +7.0 points |
| Gemini Decision 1 20261111tie group a · +3.4 points vs A Gemini Decision 1 (20261111): 36.6%, 131 right of 358; Wilson 95% interval 31.8% to 41.7% | 36.6% | 131 / 358 | a | +3.4 points |
| SubQ Snap 1.1tie group a · +8.1 points vs A SubQ Snap 1.1: 35.2%, 126 right of 358; Wilson 95% interval 30.4% to 40.3% | 35.2% | 126 / 358 | a | +8.1 points |
| Drex Max v1.1tie group a · +0.8 points vs A Drex Max v1.1: 33.0%, 118 right of 358; Wilson 95% interval 28.3% to 38.0% | 33.0% | 118 / 358 | a | +0.8 points |
| Jev 1.14 | Partial | answered 356 / 358 | – | – |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – | – |
| Muse Ember chat | Not applicable | – | – | – |
OCR route (D) · 306 Logic questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 306 Logic questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| d1-ultra | Not run | – | – | – |
| Drex Max v1.1 | Not run | – | – | – |
| Gemini Decision 1 20261111 | Not run | – | – | – |
| SubQ Snap 1.1 | Not run | – | – | – |
| Jev 1.14 | Not run | – | – | – |
| GPT-6 Luna | Not run | – | – | – |
| Muse Ember chat | Not applicable | – | – | – |
OCR route (D) · 52 Math questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 52 Math questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| d1-ultra | Not run | – | – | – |
| Drex Max v1.1 | Not run | – | – | – |
| Gemini Decision 1 20261111 | Not run | – | – | – |
| SubQ Snap 1.1 | Not run | – | – | – |
| Jev 1.14 | Not run | – | – | – |
| GPT-6 Luna | Not run | – | – | – |
| Muse Ember chat | Not applicable | – | – | – |
OCR route (D) · 358 Logic & Math questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 358 Logic & Math questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| d1-ultra | Not run | – | – | – |
| Drex Max v1.1 | Not run | – | – | – |
| Gemini Decision 1 20261111 | Not run | – | – | – |
| SubQ Snap 1.1 | Not run | – | – | – |
| Jev 1.14 | Not run | – | – | – |
| GPT-6 Luna | Not run | – | – | – |
| Muse Ember chat | Not applicable | – | – | – |
Every setup of the track
7 setups of the models chosen; scored only when a cell is complete (306 Logic, 52 Math).
| Modelstate | Answers storedLogic · Math |
|---|---|
| d1-ultraPublished | A text only: 306/306 · 52/52B picture alone: 306/306 · 52/52C text + picture: 306/306 · 52/52D ocr route: 0/306 · 0/52 |
| Drex Max v1.1Published | A text only: 306/306 · 52/52B picture alone: 306/306 · 52/52C text + picture: 306/306 · 52/52D ocr route: 0/306 · 0/52 |
| Gemini Decision 1 20261111Published | A text only: 306/306 · 52/52B picture alone: 306/306 · 52/52C text + picture: 306/306 · 52/52D ocr route: 0/306 · 0/52 |
| SubQ Snap 1.1Published | A text only: 306/306 · 52/52B picture alone: 306/306 · 52/52C text + picture: 306/306 · 52/52D ocr route: 0/306 · 0/52 |
| Jev 1.14Partly runLogic B and C not measured (304/306 accepted): 2 crops refused by the provider, 36×280 px, below TypeSafe's 280 px minimum side; A and Math run in full (Zeki #5277) | A text only: 306/306 · 52/52B picture alone: 304/306 · 52/52C text + picture: 304/306 · 52/52D ocr route: 0/306 · 0/52 |
| GPT-6 LunaBlockedAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | none stored yet |
| Muse Ember chatBlockedno qualified Logic & Math chat adapter for its hosted route, and no L&M budget for it | none stored yet |
The picture alone (B) · 306 Logic questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 10 queued, 1 blocked, 1 partial · Table
Google PelaAI Fastino Nace.AI vLLM Semantic Router Subquadratic Liquid AI Chance, 20% (five options)
The picture alone (B) · 306 Logic questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| Gemini Decision 1 20261111tie group a · +2.3 points vs A Gemini Decision 1 (20261111): 33.7%, 103 right of 306; Wilson 95% interval 28.6% to 39.1% | 33.7% | 103 / 306 | a | +2.3 points |
| KnowLine-31B-Gen1.5 Run 2Run 2 of 12 Nov; first run 99/306 kept on Experiments · tie group a · +0.3 points vs A KnowLine-31B-Gen1.5 (Run 2): 32.0%, 98 right of 306; Wilson 95% interval 27.0% to 37.4%; Run 2 of 12 Nov; first run 99/306 kept on Experiments | 32.0% | 98 / 306 | a | +0.3 points |
| GLiDE 27Btie group a · +2.0 points vs A GLiDE 27B: 30.7%, 94 right of 306; Wilson 95% interval 25.8% to 36.1% | 30.7% | 94 / 306 | a | +2.0 points |
| Drex Max v1.1tie group a · −1.6 points vs A Drex Max v1.1: 29.7%, 91 right of 306; Wilson 95% interval 24.9% to 35.1% | 29.7% | 91 / 306 | a | −1.6 points |
| d3-ultratie group a · −2.3 points vs A d3-ultra: 29.1%, 89 right of 306; Wilson 95% interval 24.3% to 34.4% | 29.1% | 89 / 306 | a | −2.3 points |
| SubQ Snap 1.1tie group a · +4.2 points vs A SubQ Snap 1.1: 28.4%, 87 right of 306; Wilson 95% interval 23.7% to 33.7% | 28.4% | 87 / 306 | a | +4.2 points |
| d1-ultratie group a · −1.3 points vs A d1-ultra: 27.5%, 84 right of 306; Wilson 95% interval 22.8% to 32.7% | 27.5% | 84 / 306 | a | −1.3 points |
| Jev 1.14 | Partial | answered 304 / 306 | – | – |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | – | – | – |
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – | – |
The picture alone (B) · 52 Math questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 10 queued, 1 blocked · Table
Liquid AI Google Subquadratic Nace.AI TypeSafe Fastino vLLM Semantic Router PelaAI Chance, 20% (five options)
The picture alone (B) · 52 Math questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| d1-ultratie group a · +13.5 points vs A d1-ultra: 51.9%, 27 right of 52; Wilson 95% interval 38.7% to 64.9% | 51.9% | 27 / 52 | a | +13.5 points |
| Gemini Decision 1 20261111tie group a · +1.9 points vs A Gemini Decision 1 (20261111): 46.2%, 24 right of 52; Wilson 95% interval 33.3% to 59.5% | 46.2% | 24 / 52 | a | +1.9 points |
| SubQ Snap 1.1tie group a · −1.9 points vs A SubQ Snap 1.1: 42.3%, 22 right of 52; Wilson 95% interval 29.9% to 55.8% | 42.3% | 22 / 52 | a | −1.9 points |
| Drex Max v1.1tie group a · +3.8 points vs A Drex Max v1.1: 40.4%, 21 right of 52; Wilson 95% interval 28.2% to 53.9% | 40.4% | 21 / 52 | a | +3.8 points |
| Jev 1.14tie group a · −3.8 points vs A Jev 1.14: 40.4%, 21 right of 52; Wilson 95% interval 28.2% to 53.9% | 40.4% | 21 / 52 | a | −3.8 points |
| GLiDE 27Btie group a · −1.9 points vs A GLiDE 27B: 38.5%, 20 right of 52; Wilson 95% interval 26.5% to 52.0% | 38.5% | 20 / 52 | a | −1.9 points |
| d3-ultratie group a · −1.9 points vs A d3-ultra: 36.5%, 19 right of 52; Wilson 95% interval 24.8% to 50.1% | 36.5% | 19 / 52 | a | −1.9 points |
| KnowLine-31B-Gen1.5tie group a · −1.9 points vs A KnowLine-31B-Gen1.5: 36.5%, 19 right of 52; Wilson 95% interval 24.8% to 50.1% | 36.5% | 19 / 52 | a | −1.9 points |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | – | – | – |
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – | – |
The picture alone (B) · 358 Logic & Math questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 10 queued, 1 blocked, 1 partial · Table
Google PelaAI Fastino Nace.AI Liquid AI Subquadratic vLLM Semantic Router Chance, 20% (five options)
The picture alone (B) · 358 Logic & Math questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| Gemini Decision 1 20261111tie group a · +2.2 points vs A Gemini Decision 1 (20261111): 35.5%, 127 right of 358; Wilson 95% interval 30.7% to 40.6% | 35.5% | 127 / 358 | a | +2.2 points |
| KnowLine-31B-Gen1.5 first runsfirst runs (Logic 99 + Math 19); no complete Run 2 of B on Math yet, so run numbers are not mixed · tie group a · +0.3 points vs A KnowLine-31B-Gen1.5 (first runs): 33.0%, 118 right of 358; Wilson 95% interval 28.3% to 38.0%; first runs (Logic 99 + Math 19); no complete Run 2 of B on Math yet, so run numbers are not mixed | 33.0% | 118 / 358 | a | +0.3 points |
| GLiDE 27Btie group a · +1.4 points vs A GLiDE 27B: 31.8%, 114 right of 358; Wilson 95% interval 27.2% to 36.8% | 31.8% | 114 / 358 | a | +1.4 points |
| Drex Max v1.1tie group a · −0.8 points vs A Drex Max v1.1: 31.3%, 112 right of 358; Wilson 95% interval 26.7% to 36.3% | 31.3% | 112 / 358 | a | −0.8 points |
| d1-ultratie group a · +0.8 points vs A d1-ultra: 31.0%, 111 right of 358; Wilson 95% interval 26.4% to 36.0% | 31.0% | 111 / 358 | a | +0.8 points |
| SubQ Snap 1.1tie group a · +3.4 points vs A SubQ Snap 1.1: 30.4%, 109 right of 358; Wilson 95% interval 25.9% to 35.4% | 30.4% | 109 / 358 | a | +3.4 points |
| d3-ultratie group a · −2.2 points vs A d3-ultra: 30.2%, 108 right of 358; Wilson 95% interval 25.6% to 35.1% | 30.2% | 108 / 358 | a | −2.2 points |
| Jev 1.14 | Partial | answered 356 / 358 | – | – |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | – | – | – |
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – | – |
Text only, the control (A) · 306 Logic questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 10 queued, 1 blocked · Table
PelaAI vLLM Semantic Router Nace.AI Google Liquid AI Fastino TypeSafe Subquadratic Chance, 20% (five options)
Text only, the control (A) · 306 Logic questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| KnowLine-31B-Gen1.5tie group a KnowLine-31B-Gen1.5: 31.7%, 97 right of 306; Wilson 95% interval 26.7% to 37.1% | 31.7% | 97 / 306 | a |
| d3-ultratie group a d3-ultra: 31.4%, 96 right of 306; Wilson 95% interval 26.4% to 36.8% | 31.4% | 96 / 306 | a |
| Drex Max v1.1tie group a Drex Max v1.1: 31.4%, 96 right of 306; Wilson 95% interval 26.4% to 36.8% | 31.4% | 96 / 306 | a |
| Gemini Decision 1 20261111tie group a Gemini Decision 1 (20261111): 31.4%, 96 right of 306; Wilson 95% interval 26.4% to 36.8% | 31.4% | 96 / 306 | a |
| d1-ultratie group a d1-ultra: 28.8%, 88 right of 306; Wilson 95% interval 24.0% to 34.1% | 28.8% | 88 / 306 | a |
| GLiDE 27Btie group a GLiDE 27B: 28.8%, 88 right of 306; Wilson 95% interval 24.0% to 34.1% | 28.8% | 88 / 306 | a |
| Jev 1.14tie group a Jev 1.14: 25.2%, 77 right of 306; Wilson 95% interval 20.6% to 30.3% | 25.2% | 77 / 306 | a |
| SubQ Snap 1.1tie group a SubQ Snap 1.1: 24.2%, 74 right of 306; Wilson 95% interval 19.7% to 29.3% | 24.2% | 74 / 306 | a |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | answered 136 / 306 | – |
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – |
Text only, the control (A) · 52 Math questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 10 queued, 1 blocked · Table
Google TypeSafe Subquadratic Fastino Liquid AI vLLM Semantic Router PelaAI Nace.AI Chance, 20% (five options)
Text only, the control (A) · 52 Math questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| Gemini Decision 1 20261111tie group a Gemini Decision 1 (20261111): 44.2%, 23 right of 52; Wilson 95% interval 31.6% to 57.7% | 44.2% | 23 / 52 | a |
| Jev 1.14tie group a Jev 1.14: 44.2%, 23 right of 52; Wilson 95% interval 31.6% to 57.7% | 44.2% | 23 / 52 | a |
| SubQ Snap 1.1tie group a SubQ Snap 1.1: 44.2%, 23 right of 52; Wilson 95% interval 31.6% to 57.7% | 44.2% | 23 / 52 | a |
| GLiDE 27Btie group a GLiDE 27B: 40.4%, 21 right of 52; Wilson 95% interval 28.2% to 53.9% | 40.4% | 21 / 52 | a |
| d1-ultratie group a d1-ultra: 38.5%, 20 right of 52; Wilson 95% interval 26.5% to 52.0% | 38.5% | 20 / 52 | a |
| d3-ultratie group a d3-ultra: 38.5%, 20 right of 52; Wilson 95% interval 26.5% to 52.0% | 38.5% | 20 / 52 | a |
| KnowLine-31B-Gen1.5tie group a KnowLine-31B-Gen1.5: 38.5%, 20 right of 52; Wilson 95% interval 26.5% to 52.0% | 38.5% | 20 / 52 | a |
| Drex Max v1.1tie group a Drex Max v1.1: 36.5%, 19 right of 52; Wilson 95% interval 24.8% to 50.1% | 36.5% | 19 / 52 | a |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | – | – |
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – |
Text only, the control (A) · 358 Logic & Math questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 10 queued, 1 blocked · Table
Google PelaAI vLLM Semantic Router Nace.AI Fastino Liquid AI TypeSafe Subquadratic Chance, 20% (five options)
Text only, the control (A) · 358 Logic & Math questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| Gemini Decision 1 20261111tie group a Gemini Decision 1 (20261111): 33.2%, 119 right of 358; Wilson 95% interval 28.6% to 38.3% | 33.2% | 119 / 358 | a |
| KnowLine-31B-Gen1.5tie group a KnowLine-31B-Gen1.5: 32.7%, 117 right of 358; Wilson 95% interval 28.0% to 37.7% | 32.7% | 117 / 358 | a |
| d3-ultratie group a d3-ultra: 32.4%, 116 right of 358; Wilson 95% interval 27.8% to 37.4% | 32.4% | 116 / 358 | a |
| Drex Max v1.1tie group a Drex Max v1.1: 32.1%, 115 right of 358; Wilson 95% interval 27.5% to 37.1% | 32.1% | 115 / 358 | a |
| GLiDE 27Btie group a GLiDE 27B: 30.4%, 109 right of 358; Wilson 95% interval 25.9% to 35.4% | 30.4% | 109 / 358 | a |
| d1-ultratie group a d1-ultra: 30.2%, 108 right of 358; Wilson 95% interval 25.6% to 35.1% | 30.2% | 108 / 358 | a |
| Jev 1.14tie group a Jev 1.14: 27.9%, 100 right of 358; Wilson 95% interval 23.5% to 32.8% | 27.9% | 100 / 358 | a |
| SubQ Snap 1.1tie group a SubQ Snap 1.1: 27.1%, 97 right of 358; Wilson 95% interval 22.8% to 31.9% | 27.1% | 97 / 358 | a |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | answered 136 / 358 | – |
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – |
Text plus picture (C) · 306 Logic questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 10 queued, 1 blocked, 1 partial · Table
Fastino PelaAI Liquid AI Google Subquadratic Nace.AI vLLM Semantic Router Chance, 20% (five options)
Text plus picture (C) · 306 Logic questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| GLiDE 27Btie group a · +9.8 points vs A GLiDE 27B: 38.6%, 118 right of 306; Wilson 95% interval 33.3% to 44.1% | 38.6% | 118 / 306 | a | +9.8 points |
| KnowLine-31B-Gen1.5tie group a · +5.6 points vs A KnowLine-31B-Gen1.5: 37.3%, 114 right of 306; Wilson 95% interval 32.0% to 42.8% | 37.3% | 114 / 306 | a | +5.6 points |
| d1-ultratie group a · +6.9 points vs A d1-ultra: 35.6%, 109 right of 306; Wilson 95% interval 30.5% to 41.1% | 35.6% | 109 / 306 | a | +6.9 points |
| Gemini Decision 1 20261111tie group a · +3.3 points vs A Gemini Decision 1 (20261111): 34.6%, 106 right of 306; Wilson 95% interval 29.5% to 40.1% | 34.6% | 106 / 306 | a | +3.3 points |
| SubQ Snap 1.1tie group a · +9.5 points vs A SubQ Snap 1.1: 33.7%, 103 right of 306; Wilson 95% interval 28.6% to 39.1% | 33.7% | 103 / 306 | a | +9.5 points |
| Drex Max v1.1tie group a · +1.6 points vs A Drex Max v1.1: 33.0%, 101 right of 306; Wilson 95% interval 28.0% to 38.5% | 33.0% | 101 / 306 | a | +1.6 points |
| d3-ultratie group a · +0.3 points vs A d3-ultra: 31.7%, 97 right of 306; Wilson 95% interval 26.7% to 37.1% | 31.7% | 97 / 306 | a | +0.3 points |
| Jev 1.14 | Partial | answered 304 / 306 | – | – |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | – | – | – |
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – | – |
Text plus picture (C) · 52 Math questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 10 queued, 1 blocked · Table
TypeSafe vLLM Semantic Router Google Liquid AI Fastino PelaAI Subquadratic Nace.AI Chance, 20% (five options)
Text plus picture (C) · 52 Math questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| Jev 1.14tie group a · +7.7 points vs A Jev 1.14: 51.9%, 27 right of 52; Wilson 95% interval 38.7% to 64.9% | 51.9% | 27 / 52 | a | +7.7 points |
| d3-ultratie group a · +9.6 points vs A d3-ultra: 48.1%, 25 right of 52; Wilson 95% interval 35.1% to 61.3% | 48.1% | 25 / 52 | a | +9.6 points |
| Gemini Decision 1 20261111tie group a · +3.8 points vs A Gemini Decision 1 (20261111): 48.1%, 25 right of 52; Wilson 95% interval 35.1% to 61.3% | 48.1% | 25 / 52 | a | +3.8 points |
| d1-ultratie group a · +7.7 points vs A d1-ultra: 46.2%, 24 right of 52; Wilson 95% interval 33.3% to 59.5% | 46.2% | 24 / 52 | a | +7.7 points |
| GLiDE 27Btie group a · +5.8 points vs A GLiDE 27B: 46.2%, 24 right of 52; Wilson 95% interval 33.3% to 59.5% | 46.2% | 24 / 52 | a | +5.8 points |
| KnowLine-31B-Gen1.5tie group a · +7.7 points vs A KnowLine-31B-Gen1.5: 46.2%, 24 right of 52; Wilson 95% interval 33.3% to 59.5% | 46.2% | 24 / 52 | a | +7.7 points |
| SubQ Snap 1.1tie group a · 0.0 points vs A SubQ Snap 1.1: 44.2%, 23 right of 52; Wilson 95% interval 31.6% to 57.7% | 44.2% | 23 / 52 | a | 0.0 points |
| Drex Max v1.1tie group a · −3.8 points vs A Drex Max v1.1: 32.7%, 17 right of 52; Wilson 95% interval 21.5% to 46.2% | 32.7% | 17 / 52 | a | −3.8 points |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | – | – | – |
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – | – |
Text plus picture (C) · 358 Logic & Math questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 10 queued, 1 blocked, 1 partial · Table
Fastino PelaAI Liquid AI Google Subquadratic vLLM Semantic Router Nace.AI Chance, 20% (five options)
Text plus picture (C) · 358 Logic & Math questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| GLiDE 27Btie group a · +9.2 points vs A GLiDE 27B: 39.7%, 142 right of 358; Wilson 95% interval 34.7% to 44.8% | 39.7% | 142 / 358 | a | +9.2 points |
| KnowLine-31B-Gen1.5tie group a · +5.9 points vs A KnowLine-31B-Gen1.5: 38.5%, 138 right of 358; Wilson 95% interval 33.7% to 43.7% | 38.5% | 138 / 358 | a | +5.9 points |
| d1-ultratie group a · +7.0 points vs A d1-ultra: 37.2%, 133 right of 358; Wilson 95% interval 32.3% to 42.3% | 37.2% | 133 / 358 | a | +7.0 points |
| Gemini Decision 1 20261111tie group a · +3.4 points vs A Gemini Decision 1 (20261111): 36.6%, 131 right of 358; Wilson 95% interval 31.8% to 41.7% | 36.6% | 131 / 358 | a | +3.4 points |
| SubQ Snap 1.1tie group a · +8.1 points vs A SubQ Snap 1.1: 35.2%, 126 right of 358; Wilson 95% interval 30.4% to 40.3% | 35.2% | 126 / 358 | a | +8.1 points |
| d3-ultratie group a · +1.7 points vs A d3-ultra: 34.1%, 122 right of 358; Wilson 95% interval 29.4% to 39.1% | 34.1% | 122 / 358 | a | +1.7 points |
| Drex Max v1.1tie group a · +0.8 points vs A Drex Max v1.1: 33.0%, 118 right of 358; Wilson 95% interval 28.3% to 38.0% | 33.0% | 118 / 358 | a | +0.8 points |
| Jev 1.14 | Partial | answered 356 / 358 | – | – |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | – | – | – |
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – | – |
OCR route (D) · 306 Logic questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 306 Logic questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| d1-ultra | Not run | – | – | – |
| d3-ultra | Not run | – | – | – |
| Drex Max v1.1 | Not run | – | – | – |
| Gemini Decision 1 20261111 | Not run | – | – | – |
| GLiDE 27B | Not run | – | – | – |
| KnowLine-31B-Gen1.5 | Not run | – | – | – |
| SubQ Snap 1.1 | Not run | – | – | – |
| Jev 1.14 | Not run | – | – | – |
| OneJev-27B | Not run | – | – | – |
| d1-2B | Not run | – | – | – |
| d1-3B | Not run | – | – | – |
| d1-Omni-2B | Not run | – | – | – |
| d3 | Not run | – | – | – |
| Drex Max | Not run | – | – | – |
| KnowLine 4B Gen5.5 | Not run | – | – | – |
| KnowLine-0.8B-Gen1 | Not run | – | – | – |
| OneJev1.6-0.8B | Not run | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instant | Not run | – | – | – |
| StartLux-Decision-v1.1-2B-Instant | Not run | – | – | – |
| GPT-6 Luna | Not run | – | – | – |
OCR route (D) · 52 Math questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 52 Math questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| d1-ultra | Not run | – | – | – |
| d3-ultra | Not run | – | – | – |
| Drex Max v1.1 | Not run | – | – | – |
| Gemini Decision 1 20261111 | Not run | – | – | – |
| GLiDE 27B | Not run | – | – | – |
| KnowLine-31B-Gen1.5 | Not run | – | – | – |
| SubQ Snap 1.1 | Not run | – | – | – |
| Jev 1.14 | Not run | – | – | – |
| OneJev-27B | Not run | – | – | – |
| d1-2B | Not run | – | – | – |
| d1-3B | Not run | – | – | – |
| d1-Omni-2B | Not run | – | – | – |
| d3 | Not run | – | – | – |
| Drex Max | Not run | – | – | – |
| KnowLine 4B Gen5.5 | Not run | – | – | – |
| KnowLine-0.8B-Gen1 | Not run | – | – | – |
| OneJev1.6-0.8B | Not run | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instant | Not run | – | – | – |
| StartLux-Decision-v1.1-2B-Instant | Not run | – | – | – |
| GPT-6 Luna | Not run | – | – | – |
OCR route (D) · 358 Logic & Math questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 358 Logic & Math questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| d1-ultra | Not run | – | – | – |
| d3-ultra | Not run | – | – | – |
| Drex Max v1.1 | Not run | – | – | – |
| Gemini Decision 1 20261111 | Not run | – | – | – |
| GLiDE 27B | Not run | – | – | – |
| KnowLine-31B-Gen1.5 | Not run | – | – | – |
| SubQ Snap 1.1 | Not run | – | – | – |
| Jev 1.14 | Not run | – | – | – |
| OneJev-27B | Not run | – | – | – |
| d1-2B | Not run | – | – | – |
| d1-3B | Not run | – | – | – |
| d1-Omni-2B | Not run | – | – | – |
| d3 | Not run | – | – | – |
| Drex Max | Not run | – | – | – |
| KnowLine 4B Gen5.5 | Not run | – | – | – |
| KnowLine-0.8B-Gen1 | Not run | – | – | – |
| OneJev1.6-0.8B | Not run | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instant | Not run | – | – | – |
| StartLux-Decision-v1.1-2B-Instant | Not run | – | – | – |
| GPT-6 Luna | Not run | – | – | – |
Every setup of the track
20 setups of the models chosen; scored only when a cell is complete (306 Logic, 52 Math).
| Modelstate | Answers storedLogic · Math |
|---|---|
| d1-ultraPublished | A text only: 306/306 · 52/52B picture alone: 306/306 · 52/52C text + picture: 306/306 · 52/52D ocr route: 0/306 · 0/52 |
| d3-ultraPublished | A text only: 306/306 · 52/52B picture alone: 306/306 · 52/52C text + picture: 306/306 · 52/52D ocr route: 0/306 · 0/52 |
| Drex Max v1.1Published | A text only: 306/306 · 52/52B picture alone: 306/306 · 52/52C text + picture: 306/306 · 52/52D ocr route: 0/306 · 0/52 |
| Gemini Decision 1 20261111Published | A text only: 306/306 · 52/52B picture alone: 306/306 · 52/52C text + picture: 306/306 · 52/52D ocr route: 0/306 · 0/52 |
| GLiDE 27BPublished | A text only: 306/306 · 52/52B picture alone: 306/306 · 52/52C text + picture: 306/306 · 52/52D ocr route: 0/306 · 0/52 |
| KnowLine-31B-Gen1.5Published | A text only: 306/306 · 52/52B picture alone: 306/306 · 52/52C text + picture: 306/306 · 52/52D ocr route: 0/306 · 0/52 |
| SubQ Snap 1.1Published | A text only: 306/306 · 52/52B picture alone: 306/306 · 52/52C text + picture: 306/306 · 52/52D ocr route: 0/306 · 0/52 |
| Jev 1.14Partly runLogic B and C not measured (304/306 accepted): 2 crops refused by the provider, 36×280 px, below TypeSafe's 280 px minimum side; A and Math run in full (Zeki #5277) | A text only: 306/306 · 52/52B picture alone: 304/306 · 52/52C text + picture: 304/306 · 52/52D ocr route: 0/306 · 0/52 |
| OneJev-27BPausedstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | A text only: 136/306 · 0/52B picture alone: 0/306 · 0/52C text + picture: 0/306 · 0/52D ocr route: 0/306 · 0/52 |
| d1-2BQueuednew d1 sizes; needs a card (bariton: no d1 environment on spark2) | none stored yet |
| d1-3BQueuednew d1 sizes; needs a card (bariton: no d1 environment on spark2) | none stored yet |
| d1-Omni-2BQueuednew d1 sizes; needs a card (bariton: no d1 environment on spark2) | none stored yet |
| d3Queuedserved setup prepared on spark2; needs a card | none stored yet |
| Drex MaxQueuedserved setup prepared on spark2; needs a card | none stored yet |
| KnowLine 4B Gen5.5Queuedlocal setup; needs a card (spark2) | none stored yet |
| KnowLine-0.8B-Gen1Queuedlocal setup; needs a card (spark2) | none stored yet |
| OneJev1.6-0.8BQueuedlocal setup; needs a card (spark2) | none stored yet |
| StartLux-Decision-v1.1-0.8B-InstantQueuedlocal setup; needs a card (spark2) | none stored yet |
| StartLux-Decision-v1.1-2B-InstantQueuedlocal setup; needs a card (spark2) | none stored yet |
| GPT-6 LunaBlockedAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | none stored yet |
The picture alone (B) · 306 Logic questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (8 queued). Every setup in the table
The picture alone (B) · 306 Logic questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
The picture alone (B) · 52 Math questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (8 queued). Every setup in the table
The picture alone (B) · 52 Math questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
The picture alone (B) · 358 Logic & Math questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (8 queued). Every setup in the table
The picture alone (B) · 358 Logic & Math questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
Text only, the control (A) · 306 Logic questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (8 queued). Every setup in the table
Text only, the control (A) · 306 Logic questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – |
Text only, the control (A) · 52 Math questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (8 queued). Every setup in the table
Text only, the control (A) · 52 Math questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – |
Text only, the control (A) · 358 Logic & Math questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (8 queued). Every setup in the table
Text only, the control (A) · 358 Logic & Math questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – |
Text plus picture (C) · 306 Logic questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (8 queued). Every setup in the table
Text plus picture (C) · 306 Logic questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
Text plus picture (C) · 52 Math questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (8 queued). Every setup in the table
Text plus picture (C) · 52 Math questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
Text plus picture (C) · 358 Logic & Math questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (8 queued). Every setup in the table
Text plus picture (C) · 358 Logic & Math questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| d1-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-3Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| d1-Omni-2Bnew d1 sizes; needs a card (bariton: no d1 environment on spark2) | Queued | – | – | – |
| KnowLine 4B Gen5.5local setup; needs a card (spark2) | Queued | – | – | – |
| KnowLine-0.8B-Gen1local setup; needs a card (spark2) | Queued | – | – | – |
| OneJev1.6-0.8Blocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
| StartLux-Decision-v1.1-2B-Instantlocal setup; needs a card (spark2) | Queued | – | – | – |
OCR route (D) · 306 Logic questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 306 Logic questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| d1-2B | Not run | – | – | – |
| d1-3B | Not run | – | – | – |
| d1-Omni-2B | Not run | – | – | – |
| KnowLine 4B Gen5.5 | Not run | – | – | – |
| KnowLine-0.8B-Gen1 | Not run | – | – | – |
| OneJev1.6-0.8B | Not run | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instant | Not run | – | – | – |
| StartLux-Decision-v1.1-2B-Instant | Not run | – | – | – |
OCR route (D) · 52 Math questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 52 Math questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| d1-2B | Not run | – | – | – |
| d1-3B | Not run | – | – | – |
| d1-Omni-2B | Not run | – | – | – |
| KnowLine 4B Gen5.5 | Not run | – | – | – |
| KnowLine-0.8B-Gen1 | Not run | – | – | – |
| OneJev1.6-0.8B | Not run | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instant | Not run | – | – | – |
| StartLux-Decision-v1.1-2B-Instant | Not run | – | – | – |
OCR route (D) · 358 Logic & Math questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 358 Logic & Math questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| d1-2B | Not run | – | – | – |
| d1-3B | Not run | – | – | – |
| d1-Omni-2B | Not run | – | – | – |
| KnowLine 4B Gen5.5 | Not run | – | – | – |
| KnowLine-0.8B-Gen1 | Not run | – | – | – |
| OneJev1.6-0.8B | Not run | – | – | – |
| StartLux-Decision-v1.1-0.8B-Instant | Not run | – | – | – |
| StartLux-Decision-v1.1-2B-Instant | Not run | – | – | – |
Every setup of the track
8 setups of the models chosen; scored only when a cell is complete (306 Logic, 52 Math).
| Modelstate | Answers storedLogic · Math |
|---|---|
| d1-2BQueuednew d1 sizes; needs a card (bariton: no d1 environment on spark2) | none stored yet |
| d1-3BQueuednew d1 sizes; needs a card (bariton: no d1 environment on spark2) | none stored yet |
| d1-Omni-2BQueuednew d1 sizes; needs a card (bariton: no d1 environment on spark2) | none stored yet |
| KnowLine 4B Gen5.5Queuedlocal setup; needs a card (spark2) | none stored yet |
| KnowLine-0.8B-Gen1Queuedlocal setup; needs a card (spark2) | none stored yet |
| OneJev1.6-0.8BQueuedlocal setup; needs a card (spark2) | none stored yet |
| StartLux-Decision-v1.1-0.8B-InstantQueuedlocal setup; needs a card (spark2) | none stored yet |
| StartLux-Decision-v1.1-2B-InstantQueuedlocal setup; needs a card (spark2) | none stored yet |
The picture alone (B) · 306 Logic questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 2 queued · Table
PelaAI Fastino vLLM Semantic Router Chance, 20% (five options)
The picture alone (B) · 306 Logic questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| KnowLine-31B-Gen1.5 Run 2Run 2 of 12 Nov; first run 99/306 kept on Experiments · tie group a · +0.3 points vs A KnowLine-31B-Gen1.5 (Run 2): 32.0%, 98 right of 306; Wilson 95% interval 27.0% to 37.4%; Run 2 of 12 Nov; first run 99/306 kept on Experiments | 32.0% | 98 / 306 | a | +0.3 points |
| GLiDE 27Btie group a · +2.0 points vs A GLiDE 27B: 30.7%, 94 right of 306; Wilson 95% interval 25.8% to 36.1% | 30.7% | 94 / 306 | a | +2.0 points |
| d3-ultratie group a · −2.3 points vs A d3-ultra: 29.1%, 89 right of 306; Wilson 95% interval 24.3% to 34.4% | 29.1% | 89 / 306 | a | −2.3 points |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | – | – | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – | – |
The picture alone (B) · 52 Math questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 2 queued · Table
Fastino vLLM Semantic Router PelaAI Chance, 20% (five options)
The picture alone (B) · 52 Math questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| GLiDE 27Btie group a · −1.9 points vs A GLiDE 27B: 38.5%, 20 right of 52; Wilson 95% interval 26.5% to 52.0% | 38.5% | 20 / 52 | a | −1.9 points |
| d3-ultratie group a · −1.9 points vs A d3-ultra: 36.5%, 19 right of 52; Wilson 95% interval 24.8% to 50.1% | 36.5% | 19 / 52 | a | −1.9 points |
| KnowLine-31B-Gen1.5tie group a · −1.9 points vs A KnowLine-31B-Gen1.5: 36.5%, 19 right of 52; Wilson 95% interval 24.8% to 50.1% | 36.5% | 19 / 52 | a | −1.9 points |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | – | – | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – | – |
The picture alone (B) · 358 Logic & Math questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 2 queued · Table
PelaAI Fastino vLLM Semantic Router Chance, 20% (five options)
The picture alone (B) · 358 Logic & Math questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| KnowLine-31B-Gen1.5 first runsfirst runs (Logic 99 + Math 19); no complete Run 2 of B on Math yet, so run numbers are not mixed · tie group a · +0.3 points vs A KnowLine-31B-Gen1.5 (first runs): 33.0%, 118 right of 358; Wilson 95% interval 28.3% to 38.0%; first runs (Logic 99 + Math 19); no complete Run 2 of B on Math yet, so run numbers are not mixed | 33.0% | 118 / 358 | a | +0.3 points |
| GLiDE 27Btie group a · +1.4 points vs A GLiDE 27B: 31.8%, 114 right of 358; Wilson 95% interval 27.2% to 36.8% | 31.8% | 114 / 358 | a | +1.4 points |
| d3-ultratie group a · −2.2 points vs A d3-ultra: 30.2%, 108 right of 358; Wilson 95% interval 25.6% to 35.1% | 30.2% | 108 / 358 | a | −2.2 points |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | – | – | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – | – |
Text only, the control (A) · 306 Logic questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 2 queued · Table
PelaAI vLLM Semantic Router Fastino Chance, 20% (five options)
Text only, the control (A) · 306 Logic questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| KnowLine-31B-Gen1.5tie group a KnowLine-31B-Gen1.5: 31.7%, 97 right of 306; Wilson 95% interval 26.7% to 37.1% | 31.7% | 97 / 306 | a |
| d3-ultratie group a d3-ultra: 31.4%, 96 right of 306; Wilson 95% interval 26.4% to 36.8% | 31.4% | 96 / 306 | a |
| GLiDE 27Btie group a GLiDE 27B: 28.8%, 88 right of 306; Wilson 95% interval 24.0% to 34.1% | 28.8% | 88 / 306 | a |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | answered 136 / 306 | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – |
Text only, the control (A) · 52 Math questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 2 queued · Table
Fastino vLLM Semantic Router PelaAI Chance, 20% (five options)
Text only, the control (A) · 52 Math questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| GLiDE 27Btie group a GLiDE 27B: 40.4%, 21 right of 52; Wilson 95% interval 28.2% to 53.9% | 40.4% | 21 / 52 | a |
| d3-ultratie group a d3-ultra: 38.5%, 20 right of 52; Wilson 95% interval 26.5% to 52.0% | 38.5% | 20 / 52 | a |
| KnowLine-31B-Gen1.5tie group a KnowLine-31B-Gen1.5: 38.5%, 20 right of 52; Wilson 95% interval 26.5% to 52.0% | 38.5% | 20 / 52 | a |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | – | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – |
Text only, the control (A) · 358 Logic & Math questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 2 queued · Table
PelaAI vLLM Semantic Router Fastino Chance, 20% (five options)
Text only, the control (A) · 358 Logic & Math questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| KnowLine-31B-Gen1.5tie group a KnowLine-31B-Gen1.5: 32.7%, 117 right of 358; Wilson 95% interval 28.0% to 37.7% | 32.7% | 117 / 358 | a |
| d3-ultratie group a d3-ultra: 32.4%, 116 right of 358; Wilson 95% interval 27.8% to 37.4% | 32.4% | 116 / 358 | a |
| GLiDE 27Btie group a GLiDE 27B: 30.4%, 109 right of 358; Wilson 95% interval 25.9% to 35.4% | 30.4% | 109 / 358 | a |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | answered 136 / 358 | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – |
Text plus picture (C) · 306 Logic questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 2 queued · Table
Fastino PelaAI vLLM Semantic Router Chance, 20% (five options)
Text plus picture (C) · 306 Logic questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| GLiDE 27Btie group a · +9.8 points vs A GLiDE 27B: 38.6%, 118 right of 306; Wilson 95% interval 33.3% to 44.1% | 38.6% | 118 / 306 | a | +9.8 points |
| KnowLine-31B-Gen1.5tie group a · +5.6 points vs A KnowLine-31B-Gen1.5: 37.3%, 114 right of 306; Wilson 95% interval 32.0% to 42.8% | 37.3% | 114 / 306 | a | +5.6 points |
| d3-ultratie group a · +0.3 points vs A d3-ultra: 31.7%, 97 right of 306; Wilson 95% interval 26.7% to 37.1% | 31.7% | 97 / 306 | a | +0.3 points |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | – | – | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – | – |
Text plus picture (C) · 52 Math questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 2 queued · Table
vLLM Semantic Router Fastino PelaAI Chance, 20% (five options)
Text plus picture (C) · 52 Math questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| d3-ultratie group a · +9.6 points vs A d3-ultra: 48.1%, 25 right of 52; Wilson 95% interval 35.1% to 61.3% | 48.1% | 25 / 52 | a | +9.6 points |
| GLiDE 27Btie group a · +5.8 points vs A GLiDE 27B: 46.2%, 24 right of 52; Wilson 95% interval 33.3% to 59.5% | 46.2% | 24 / 52 | a | +5.8 points |
| KnowLine-31B-Gen1.5tie group a · +7.7 points vs A KnowLine-31B-Gen1.5: 46.2%, 24 right of 52; Wilson 95% interval 33.3% to 59.5% | 46.2% | 24 / 52 | a | +7.7 points |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | – | – | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – | – |
Text plus picture (C) · 358 Logic & Math questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 paused, 2 queued · Table
Fastino PelaAI vLLM Semantic Router Chance, 20% (five options)
Text plus picture (C) · 358 Logic & Math questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| GLiDE 27Btie group a · +9.2 points vs A GLiDE 27B: 39.7%, 142 right of 358; Wilson 95% interval 34.7% to 44.8% | 39.7% | 142 / 358 | a | +9.2 points |
| KnowLine-31B-Gen1.5tie group a · +5.9 points vs A KnowLine-31B-Gen1.5: 38.5%, 138 right of 358; Wilson 95% interval 33.7% to 43.7% | 38.5% | 138 / 358 | a | +5.9 points |
| d3-ultratie group a · +1.7 points vs A d3-ultra: 34.1%, 122 right of 358; Wilson 95% interval 29.4% to 39.1% | 34.1% | 122 / 358 | a | +1.7 points |
| OneJev-27Bstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | Paused | – | – | – |
| d3served setup prepared on spark2; needs a card | Queued | – | – | – |
| Drex Maxserved setup prepared on spark2; needs a card | Queued | – | – | – |
OCR route (D) · 306 Logic questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 306 Logic questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| d3-ultra | Not run | – | – | – |
| GLiDE 27B | Not run | – | – | – |
| KnowLine-31B-Gen1.5 | Not run | – | – | – |
| OneJev-27B | Not run | – | – | – |
| d3 | Not run | – | – | – |
| Drex Max | Not run | – | – | – |
OCR route (D) · 52 Math questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 52 Math questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| d3-ultra | Not run | – | – | – |
| GLiDE 27B | Not run | – | – | – |
| KnowLine-31B-Gen1.5 | Not run | – | – | – |
| OneJev-27B | Not run | – | – | – |
| d3 | Not run | – | – | – |
| Drex Max | Not run | – | – | – |
OCR route (D) · 358 Logic & Math questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 358 Logic & Math questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| d3-ultra | Not run | – | – | – |
| GLiDE 27B | Not run | – | – | – |
| KnowLine-31B-Gen1.5 | Not run | – | – | – |
| OneJev-27B | Not run | – | – | – |
| d3 | Not run | – | – | – |
| Drex Max | Not run | – | – | – |
Every setup of the track
6 setups of the models chosen; scored only when a cell is complete (306 Logic, 52 Math).
| Modelstate | Answers storedLogic · Math |
|---|---|
| d3-ultraPublished | A text only: 306/306 · 52/52B picture alone: 306/306 · 52/52C text + picture: 306/306 · 52/52D ocr route: 0/306 · 0/52 |
| GLiDE 27BPublished | A text only: 306/306 · 52/52B picture alone: 306/306 · 52/52C text + picture: 306/306 · 52/52D ocr route: 0/306 · 0/52 |
| KnowLine-31B-Gen1.5Published | A text only: 306/306 · 52/52B picture alone: 306/306 · 52/52C text + picture: 306/306 · 52/52D ocr route: 0/306 · 0/52 |
| OneJev-27BPausedstopped during its first full run when spark2 went to other models (Ümit #4187); needs a card | A text only: 136/306 · 0/52B picture alone: 0/306 · 0/52C text + picture: 0/306 · 0/52D ocr route: 0/306 · 0/52 |
| d3Queuedserved setup prepared on spark2; needs a card | none stored yet |
| Drex MaxQueuedserved setup prepared on spark2; needs a card | none stored yet |
The picture alone (B) · 306 Logic questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 blocked, 1 partial · Table
Google Nace.AI Subquadratic Liquid AI Chance, 20% (five options)
The picture alone (B) · 306 Logic questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| Gemini Decision 1 20261111tie group a · +2.3 points vs A Gemini Decision 1 (20261111): 33.7%, 103 right of 306; Wilson 95% interval 28.6% to 39.1% | 33.7% | 103 / 306 | a | +2.3 points |
| Drex Max v1.1tie group a · −1.6 points vs A Drex Max v1.1: 29.7%, 91 right of 306; Wilson 95% interval 24.9% to 35.1% | 29.7% | 91 / 306 | a | −1.6 points |
| SubQ Snap 1.1tie group a · +4.2 points vs A SubQ Snap 1.1: 28.4%, 87 right of 306; Wilson 95% interval 23.7% to 33.7% | 28.4% | 87 / 306 | a | +4.2 points |
| d1-ultratie group a · −1.3 points vs A d1-ultra: 27.5%, 84 right of 306; Wilson 95% interval 22.8% to 32.7% | 27.5% | 84 / 306 | a | −1.3 points |
| Jev 1.14 | Partial | answered 304 / 306 | – | – |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – | – |
The picture alone (B) · 52 Math questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 blocked · Table
Liquid AI Google Subquadratic Nace.AI TypeSafe Chance, 20% (five options)
The picture alone (B) · 52 Math questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| d1-ultratie group a · +13.5 points vs A d1-ultra: 51.9%, 27 right of 52; Wilson 95% interval 38.7% to 64.9% | 51.9% | 27 / 52 | a | +13.5 points |
| Gemini Decision 1 20261111tie group a · +1.9 points vs A Gemini Decision 1 (20261111): 46.2%, 24 right of 52; Wilson 95% interval 33.3% to 59.5% | 46.2% | 24 / 52 | a | +1.9 points |
| SubQ Snap 1.1tie group a · −1.9 points vs A SubQ Snap 1.1: 42.3%, 22 right of 52; Wilson 95% interval 29.9% to 55.8% | 42.3% | 22 / 52 | a | −1.9 points |
| Drex Max v1.1tie group a · +3.8 points vs A Drex Max v1.1: 40.4%, 21 right of 52; Wilson 95% interval 28.2% to 53.9% | 40.4% | 21 / 52 | a | +3.8 points |
| Jev 1.14tie group a · −3.8 points vs A Jev 1.14: 40.4%, 21 right of 52; Wilson 95% interval 28.2% to 53.9% | 40.4% | 21 / 52 | a | −3.8 points |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – | – |
The picture alone (B) · 358 Logic & Math questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 blocked, 1 partial · Table
Google Nace.AI Liquid AI Subquadratic Chance, 20% (five options)
The picture alone (B) · 358 Logic & Math questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| Gemini Decision 1 20261111tie group a · +2.2 points vs A Gemini Decision 1 (20261111): 35.5%, 127 right of 358; Wilson 95% interval 30.7% to 40.6% | 35.5% | 127 / 358 | a | +2.2 points |
| Drex Max v1.1tie group a · −0.8 points vs A Drex Max v1.1: 31.3%, 112 right of 358; Wilson 95% interval 26.7% to 36.3% | 31.3% | 112 / 358 | a | −0.8 points |
| d1-ultratie group a · +0.8 points vs A d1-ultra: 31.0%, 111 right of 358; Wilson 95% interval 26.4% to 36.0% | 31.0% | 111 / 358 | a | +0.8 points |
| SubQ Snap 1.1tie group a · +3.4 points vs A SubQ Snap 1.1: 30.4%, 109 right of 358; Wilson 95% interval 25.9% to 35.4% | 30.4% | 109 / 358 | a | +3.4 points |
| Jev 1.14 | Partial | answered 356 / 358 | – | – |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – | – |
Text only, the control (A) · 306 Logic questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 blocked · Table
Nace.AI Google Liquid AI TypeSafe Subquadratic Chance, 20% (five options)
Text only, the control (A) · 306 Logic questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| Drex Max v1.1tie group a Drex Max v1.1: 31.4%, 96 right of 306; Wilson 95% interval 26.4% to 36.8% | 31.4% | 96 / 306 | a |
| Gemini Decision 1 20261111tie group a Gemini Decision 1 (20261111): 31.4%, 96 right of 306; Wilson 95% interval 26.4% to 36.8% | 31.4% | 96 / 306 | a |
| d1-ultratie group a d1-ultra: 28.8%, 88 right of 306; Wilson 95% interval 24.0% to 34.1% | 28.8% | 88 / 306 | a |
| Jev 1.14tie group a Jev 1.14: 25.2%, 77 right of 306; Wilson 95% interval 20.6% to 30.3% | 25.2% | 77 / 306 | a |
| SubQ Snap 1.1tie group a SubQ Snap 1.1: 24.2%, 74 right of 306; Wilson 95% interval 19.7% to 29.3% | 24.2% | 74 / 306 | a |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – |
Text only, the control (A) · 52 Math questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 blocked · Table
Google TypeSafe Subquadratic Liquid AI Nace.AI Chance, 20% (five options)
Text only, the control (A) · 52 Math questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| Gemini Decision 1 20261111tie group a Gemini Decision 1 (20261111): 44.2%, 23 right of 52; Wilson 95% interval 31.6% to 57.7% | 44.2% | 23 / 52 | a |
| Jev 1.14tie group a Jev 1.14: 44.2%, 23 right of 52; Wilson 95% interval 31.6% to 57.7% | 44.2% | 23 / 52 | a |
| SubQ Snap 1.1tie group a SubQ Snap 1.1: 44.2%, 23 right of 52; Wilson 95% interval 31.6% to 57.7% | 44.2% | 23 / 52 | a |
| d1-ultratie group a d1-ultra: 38.5%, 20 right of 52; Wilson 95% interval 26.5% to 52.0% | 38.5% | 20 / 52 | a |
| Drex Max v1.1tie group a Drex Max v1.1: 36.5%, 19 right of 52; Wilson 95% interval 24.8% to 50.1% | 36.5% | 19 / 52 | a |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – |
Text only, the control (A) · 358 Logic & Math questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 blocked · Table
Google Nace.AI Liquid AI TypeSafe Subquadratic Chance, 20% (five options)
Text only, the control (A) · 358 Logic & Math questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| Gemini Decision 1 20261111tie group a Gemini Decision 1 (20261111): 33.2%, 119 right of 358; Wilson 95% interval 28.6% to 38.3% | 33.2% | 119 / 358 | a |
| Drex Max v1.1tie group a Drex Max v1.1: 32.1%, 115 right of 358; Wilson 95% interval 27.5% to 37.1% | 32.1% | 115 / 358 | a |
| d1-ultratie group a d1-ultra: 30.2%, 108 right of 358; Wilson 95% interval 25.6% to 35.1% | 30.2% | 108 / 358 | a |
| Jev 1.14tie group a Jev 1.14: 27.9%, 100 right of 358; Wilson 95% interval 23.5% to 32.8% | 27.9% | 100 / 358 | a |
| SubQ Snap 1.1tie group a SubQ Snap 1.1: 27.1%, 97 right of 358; Wilson 95% interval 22.8% to 31.9% | 27.1% | 97 / 358 | a |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – |
Text plus picture (C) · 306 Logic questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 blocked, 1 partial · Table
Liquid AI Google Subquadratic Nace.AI Chance, 20% (five options)
Text plus picture (C) · 306 Logic questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| d1-ultratie group a · +6.9 points vs A d1-ultra: 35.6%, 109 right of 306; Wilson 95% interval 30.5% to 41.1% | 35.6% | 109 / 306 | a | +6.9 points |
| Gemini Decision 1 20261111tie group a · +3.3 points vs A Gemini Decision 1 (20261111): 34.6%, 106 right of 306; Wilson 95% interval 29.5% to 40.1% | 34.6% | 106 / 306 | a | +3.3 points |
| SubQ Snap 1.1tie group a · +9.5 points vs A SubQ Snap 1.1: 33.7%, 103 right of 306; Wilson 95% interval 28.6% to 39.1% | 33.7% | 103 / 306 | a | +9.5 points |
| Drex Max v1.1tie group a · +1.6 points vs A Drex Max v1.1: 33.0%, 101 right of 306; Wilson 95% interval 28.0% to 38.5% | 33.0% | 101 / 306 | a | +1.6 points |
| Jev 1.14 | Partial | answered 304 / 306 | – | – |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – | – |
Text plus picture (C) · 52 Math questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 blocked · Table
TypeSafe Google Liquid AI Subquadratic Nace.AI Chance, 20% (five options)
Text plus picture (C) · 52 Math questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| Jev 1.14tie group a · +7.7 points vs A Jev 1.14: 51.9%, 27 right of 52; Wilson 95% interval 38.7% to 64.9% | 51.9% | 27 / 52 | a | +7.7 points |
| Gemini Decision 1 20261111tie group a · +3.8 points vs A Gemini Decision 1 (20261111): 48.1%, 25 right of 52; Wilson 95% interval 35.1% to 61.3% | 48.1% | 25 / 52 | a | +3.8 points |
| d1-ultratie group a · +7.7 points vs A d1-ultra: 46.2%, 24 right of 52; Wilson 95% interval 33.3% to 59.5% | 46.2% | 24 / 52 | a | +7.7 points |
| SubQ Snap 1.1tie group a · 0.0 points vs A SubQ Snap 1.1: 44.2%, 23 right of 52; Wilson 95% interval 31.6% to 57.7% | 44.2% | 23 / 52 | a | 0.0 points |
| Drex Max v1.1tie group a · −3.8 points vs A Drex Max v1.1: 32.7%, 17 right of 52; Wilson 95% interval 21.5% to 46.2% | 32.7% | 17 / 52 | a | −3.8 points |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – | – |
Text plus picture (C) · 358 Logic & Math questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 blocked, 1 partial · Table
Liquid AI Google Subquadratic Nace.AI Chance, 20% (five options)
Text plus picture (C) · 358 Logic & Math questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| d1-ultratie group a · +7.0 points vs A d1-ultra: 37.2%, 133 right of 358; Wilson 95% interval 32.3% to 42.3% | 37.2% | 133 / 358 | a | +7.0 points |
| Gemini Decision 1 20261111tie group a · +3.4 points vs A Gemini Decision 1 (20261111): 36.6%, 131 right of 358; Wilson 95% interval 31.8% to 41.7% | 36.6% | 131 / 358 | a | +3.4 points |
| SubQ Snap 1.1tie group a · +8.1 points vs A SubQ Snap 1.1: 35.2%, 126 right of 358; Wilson 95% interval 30.4% to 40.3% | 35.2% | 126 / 358 | a | +8.1 points |
| Drex Max v1.1tie group a · +0.8 points vs A Drex Max v1.1: 33.0%, 118 right of 358; Wilson 95% interval 28.3% to 38.0% | 33.0% | 118 / 358 | a | +0.8 points |
| Jev 1.14 | Partial | answered 356 / 358 | – | – |
| GPT-6 LunaAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | Blocked | – | – | – |
OCR route (D) · 306 Logic questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 306 Logic questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| d1-ultra | Not run | – | – | – |
| Drex Max v1.1 | Not run | – | – | – |
| Gemini Decision 1 20261111 | Not run | – | – | – |
| SubQ Snap 1.1 | Not run | – | – | – |
| Jev 1.14 | Not run | – | – | – |
| GPT-6 Luna | Not run | – | – | – |
OCR route (D) · 52 Math questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 52 Math questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| d1-ultra | Not run | – | – | – |
| Drex Max v1.1 | Not run | – | – | – |
| Gemini Decision 1 20261111 | Not run | – | – | – |
| SubQ Snap 1.1 | Not run | – | – | – |
| Jev 1.14 | Not run | – | – | – |
| GPT-6 Luna | Not run | – | – | – |
OCR route (D) · 358 Logic & Math questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 358 Logic & Math questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| d1-ultra | Not run | – | – | – |
| Drex Max v1.1 | Not run | – | – | – |
| Gemini Decision 1 20261111 | Not run | – | – | – |
| SubQ Snap 1.1 | Not run | – | – | – |
| Jev 1.14 | Not run | – | – | – |
| GPT-6 Luna | Not run | – | – | – |
Every setup of the track
6 setups of the models chosen; scored only when a cell is complete (306 Logic, 52 Math).
| Modelstate | Answers storedLogic · Math |
|---|---|
| d1-ultraPublished | A text only: 306/306 · 52/52B picture alone: 306/306 · 52/52C text + picture: 306/306 · 52/52D ocr route: 0/306 · 0/52 |
| Drex Max v1.1Published | A text only: 306/306 · 52/52B picture alone: 306/306 · 52/52C text + picture: 306/306 · 52/52D ocr route: 0/306 · 0/52 |
| Gemini Decision 1 20261111Published | A text only: 306/306 · 52/52B picture alone: 306/306 · 52/52C text + picture: 306/306 · 52/52D ocr route: 0/306 · 0/52 |
| SubQ Snap 1.1Published | A text only: 306/306 · 52/52B picture alone: 306/306 · 52/52C text + picture: 306/306 · 52/52D ocr route: 0/306 · 0/52 |
| Jev 1.14Partly runLogic B and C not measured (304/306 accepted): 2 crops refused by the provider, 36×280 px, below TypeSafe's 280 px minimum side; A and Math run in full (Zeki #5277) | A text only: 306/306 · 52/52B picture alone: 304/306 · 52/52C text + picture: 304/306 · 52/52D ocr route: 0/306 · 0/52 |
| GPT-6 LunaBlockedAPI setup; its OpenAI account has no credit (HTTP 429 credit_balance_exhausted, 12 Nov 2026) | none stored yet |
The picture alone (B) · 306 Logic questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 4 queued, 1 blocked · Table
Alibaba Google Chance, 20% (five options)
The picture alone (B) · 306 Logic questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| Qwen3.5-4B on · chattie group a Qwen3.5-4B (on · chat): 62.1%, 190 right of 306; Wilson 95% interval 56.5% to 67.3% | 62.1% | 190 / 306 | a | – |
| Gemma 4 12B Unified chattie group a Gemma 4 12B Unified (chat): 56.5%, 173 right of 306; Wilson 95% interval 50.9% to 62.0% | 56.5% | 173 / 306 | a | – |
| Gemma 4 E4B chattie group b Gemma 4 E4B (chat): 38.2%, 117 right of 306; Wilson 95% interval 33.0% to 43.8% | 38.2% | 117 / 306 | b | – |
| Gemma 4 31B chatneeds a card | Queued | – | – | – |
| Gemma 4 E2B chatneeds a card | Queued | – | – | – |
| Muse Glimmer 30B chatJob 2 phase | Queued | – | – | – |
| Qwen3.5-9B chatwaits for its server window | Queued | – | – | – |
| Muse Ember chatno qualified Logic & Math chat adapter for its hosted route, and no L&M budget for it | Blocked | – | – | – |
The picture alone (B) · 52 Math questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 4 queued, 1 blocked · Table
Google Alibaba Chance, 20% (five options)
The picture alone (B) · 52 Math questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 12B Unified chattie group a Gemma 4 12B Unified (chat): 86.5%, 45 right of 52; Wilson 95% interval 74.7% to 93.3% | 86.5% | 45 / 52 | a | – |
| Qwen3.5-4B on · chattie group a Qwen3.5-4B (on · chat): 86.5%, 45 right of 52; Wilson 95% interval 74.7% to 93.3% | 86.5% | 45 / 52 | a | – |
| Gemma 4 E4B chattie group b Gemma 4 E4B (chat): 53.8%, 28 right of 52; Wilson 95% interval 40.5% to 66.7% | 53.8% | 28 / 52 | b | – |
| Gemma 4 31B chatneeds a card | Queued | – | – | – |
| Gemma 4 E2B chatneeds a card | Queued | – | – | – |
| Muse Glimmer 30B chatJob 2 phase | Queued | – | – | – |
| Qwen3.5-9B chatwaits for its server window | Queued | – | – | – |
| Muse Ember chatno qualified Logic & Math chat adapter for its hosted route, and no L&M budget for it | Blocked | – | – | – |
The picture alone (B) · 358 Logic & Math questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 4 queued, 1 blocked · Table
Alibaba Google Chance, 20% (five options)
The picture alone (B) · 358 Logic & Math questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| Qwen3.5-4B on · chattie group a Qwen3.5-4B (on · chat): 65.6%, 235 right of 358; Wilson 95% interval 60.6% to 70.4% | 65.6% | 235 / 358 | a | – |
| Gemma 4 12B Unified chattie group a Gemma 4 12B Unified (chat): 60.9%, 218 right of 358; Wilson 95% interval 55.7% to 65.8% | 60.9% | 218 / 358 | a | – |
| Gemma 4 E4B chattie group b Gemma 4 E4B (chat): 40.5%, 145 right of 358; Wilson 95% interval 35.5% to 45.7% | 40.5% | 145 / 358 | b | – |
| Gemma 4 31B chatneeds a card | Queued | – | – | – |
| Gemma 4 E2B chatneeds a card | Queued | – | – | – |
| Muse Glimmer 30B chatJob 2 phase | Queued | – | – | – |
| Qwen3.5-9B chatwaits for its server window | Queued | – | – | – |
| Muse Ember chatno qualified Logic & Math chat adapter for its hosted route, and no L&M budget for it | Blocked | – | – | – |
Text only, the control (A) · 306 Logic questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (8 not applicable). Every setup in the table
Text only, the control (A) · 306 Logic questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| Gemma 4 12B Unified chat | Not applicable | – | – |
| Gemma 4 E4B chat | Not applicable | – | – |
| Qwen3.5-4B on · chat | Not applicable | – | – |
| Gemma 4 31B chat | Not applicable | – | – |
| Gemma 4 E2B chat | Not applicable | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – |
| Qwen3.5-9B chat | Not applicable | – | – |
| Muse Ember chat | Not applicable | – | – |
Text only, the control (A) · 52 Math questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (8 not applicable). Every setup in the table
Text only, the control (A) · 52 Math questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| Gemma 4 12B Unified chat | Not applicable | – | – |
| Gemma 4 E4B chat | Not applicable | – | – |
| Qwen3.5-4B on · chat | Not applicable | – | – |
| Gemma 4 31B chat | Not applicable | – | – |
| Gemma 4 E2B chat | Not applicable | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – |
| Qwen3.5-9B chat | Not applicable | – | – |
| Muse Ember chat | Not applicable | – | – |
Text only, the control (A) · 358 Logic & Math questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (8 not applicable). Every setup in the table
Text only, the control (A) · 358 Logic & Math questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| Gemma 4 12B Unified chat | Not applicable | – | – |
| Gemma 4 E4B chat | Not applicable | – | – |
| Qwen3.5-4B on · chat | Not applicable | – | – |
| Gemma 4 31B chat | Not applicable | – | – |
| Gemma 4 E2B chat | Not applicable | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – |
| Qwen3.5-9B chat | Not applicable | – | – |
| Muse Ember chat | Not applicable | – | – |
Text plus picture (C) · 306 Logic questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (8 not applicable). Every setup in the table
Text plus picture (C) · 306 Logic questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 12B Unified chat | Not applicable | – | – | – |
| Gemma 4 E4B chat | Not applicable | – | – | – |
| Qwen3.5-4B on · chat | Not applicable | – | – | – |
| Gemma 4 31B chat | Not applicable | – | – | – |
| Gemma 4 E2B chat | Not applicable | – | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – | – |
| Qwen3.5-9B chat | Not applicable | – | – | – |
| Muse Ember chat | Not applicable | – | – | – |
Text plus picture (C) · 52 Math questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (8 not applicable). Every setup in the table
Text plus picture (C) · 52 Math questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 12B Unified chat | Not applicable | – | – | – |
| Gemma 4 E4B chat | Not applicable | – | – | – |
| Qwen3.5-4B on · chat | Not applicable | – | – | – |
| Gemma 4 31B chat | Not applicable | – | – | – |
| Gemma 4 E2B chat | Not applicable | – | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – | – |
| Qwen3.5-9B chat | Not applicable | – | – | – |
| Muse Ember chat | Not applicable | – | – | – |
Text plus picture (C) · 358 Logic & Math questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (8 not applicable). Every setup in the table
Text plus picture (C) · 358 Logic & Math questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 12B Unified chat | Not applicable | – | – | – |
| Gemma 4 E4B chat | Not applicable | – | – | – |
| Qwen3.5-4B on · chat | Not applicable | – | – | – |
| Gemma 4 31B chat | Not applicable | – | – | – |
| Gemma 4 E2B chat | Not applicable | – | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – | – |
| Qwen3.5-9B chat | Not applicable | – | – | – |
| Muse Ember chat | Not applicable | – | – | – |
OCR route (D) · 306 Logic questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 306 Logic questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 12B Unified chat | Not applicable | – | – | – |
| Gemma 4 E4B chat | Not applicable | – | – | – |
| Qwen3.5-4B on · chat | Not applicable | – | – | – |
| Gemma 4 31B chat | Not applicable | – | – | – |
| Gemma 4 E2B chat | Not applicable | – | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – | – |
| Qwen3.5-9B chat | Not applicable | – | – | – |
| Muse Ember chat | Not applicable | – | – | – |
OCR route (D) · 52 Math questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 52 Math questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 12B Unified chat | Not applicable | – | – | – |
| Gemma 4 E4B chat | Not applicable | – | – | – |
| Qwen3.5-4B on · chat | Not applicable | – | – | – |
| Gemma 4 31B chat | Not applicable | – | – | – |
| Gemma 4 E2B chat | Not applicable | – | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – | – |
| Qwen3.5-9B chat | Not applicable | – | – | – |
| Muse Ember chat | Not applicable | – | – | – |
OCR route (D) · 358 Logic & Math questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 358 Logic & Math questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 12B Unified chat | Not applicable | – | – | – |
| Gemma 4 E4B chat | Not applicable | – | – | – |
| Qwen3.5-4B on · chat | Not applicable | – | – | – |
| Gemma 4 31B chat | Not applicable | – | – | – |
| Gemma 4 E2B chat | Not applicable | – | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – | – |
| Qwen3.5-9B chat | Not applicable | – | – | – |
| Muse Ember chat | Not applicable | – | – | – |
Every setup of the track
8 setups of the models chosen; scored only when a cell is complete (306 Logic, 52 Math).
| Modelstate | Answers storedLogic · Math |
|---|---|
| Gemma 4 12B Unified chatPublished | A text only: not applicable (B only)B picture alone: 306/306 · 52/52C text + picture: not applicable (B only)D ocr route: not applicable (no OCR route for chat setups) |
| Gemma 4 E4B chatPublished | A text only: not applicable (B only)B picture alone: 306/306 · 52/52C text + picture: not applicable (B only)D ocr route: not applicable (no OCR route for chat setups) |
| Qwen3.5-4B on · chatPublished | A text only: not applicable (B only)B picture alone: 306/306 · 52/52C text + picture: not applicable (B only)D ocr route: not applicable (no OCR route for chat setups) |
| Gemma 4 31B chatQueuedneeds a card | none stored yet |
| Gemma 4 E2B chatQueuedneeds a card | none stored yet |
| Muse Glimmer 30B chatQueuedJob 2 phase | none stored yet |
| Qwen3.5-9B chatQueuedwaits for its server window | none stored yet |
| Muse Ember chatBlockedno qualified Logic & Math chat adapter for its hosted route, and no L&M budget for it | none stored yet |
Text-only chat models: not applicable, they take no images. The chat models that take images are listed above with their state.
The picture alone (B) · 306 Logic questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 queued · Table
Alibaba Chance, 20% (five options)
The picture alone (B) · 306 Logic questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| Qwen3.5-4B on · chattie group a Qwen3.5-4B (on · chat): 62.1%, 190 right of 306; Wilson 95% interval 56.5% to 67.3% | 62.1% | 190 / 306 | a | – |
| Gemma 4 E2B chatneeds a card | Queued | – | – | – |
The picture alone (B) · 52 Math questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 queued · Table
Alibaba Chance, 20% (five options)
The picture alone (B) · 52 Math questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| Qwen3.5-4B on · chattie group a Qwen3.5-4B (on · chat): 86.5%, 45 right of 52; Wilson 95% interval 74.7% to 93.3% | 86.5% | 45 / 52 | a | – |
| Gemma 4 E2B chatneeds a card | Queued | – | – | – |
The picture alone (B) · 358 Logic & Math questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 queued · Table
Alibaba Chance, 20% (five options)
The picture alone (B) · 358 Logic & Math questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| Qwen3.5-4B on · chattie group a Qwen3.5-4B (on · chat): 65.6%, 235 right of 358; Wilson 95% interval 60.6% to 70.4% | 65.6% | 235 / 358 | a | – |
| Gemma 4 E2B chatneeds a card | Queued | – | – | – |
Text only, the control (A) · 306 Logic questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (2 not applicable). Every setup in the table
Text only, the control (A) · 306 Logic questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| Qwen3.5-4B on · chat | Not applicable | – | – |
| Gemma 4 E2B chat | Not applicable | – | – |
Text only, the control (A) · 52 Math questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (2 not applicable). Every setup in the table
Text only, the control (A) · 52 Math questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| Qwen3.5-4B on · chat | Not applicable | – | – |
| Gemma 4 E2B chat | Not applicable | – | – |
Text only, the control (A) · 358 Logic & Math questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (2 not applicable). Every setup in the table
Text only, the control (A) · 358 Logic & Math questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| Qwen3.5-4B on · chat | Not applicable | – | – |
| Gemma 4 E2B chat | Not applicable | – | – |
Text plus picture (C) · 306 Logic questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (2 not applicable). Every setup in the table
Text plus picture (C) · 306 Logic questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| Qwen3.5-4B on · chat | Not applicable | – | – | – |
| Gemma 4 E2B chat | Not applicable | – | – | – |
Text plus picture (C) · 52 Math questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (2 not applicable). Every setup in the table
Text plus picture (C) · 52 Math questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| Qwen3.5-4B on · chat | Not applicable | – | – | – |
| Gemma 4 E2B chat | Not applicable | – | – | – |
Text plus picture (C) · 358 Logic & Math questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (2 not applicable). Every setup in the table
Text plus picture (C) · 358 Logic & Math questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| Qwen3.5-4B on · chat | Not applicable | – | – | – |
| Gemma 4 E2B chat | Not applicable | – | – | – |
OCR route (D) · 306 Logic questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 306 Logic questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| Qwen3.5-4B on · chat | Not applicable | – | – | – |
| Gemma 4 E2B chat | Not applicable | – | – | – |
OCR route (D) · 52 Math questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 52 Math questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| Qwen3.5-4B on · chat | Not applicable | – | – | – |
| Gemma 4 E2B chat | Not applicable | – | – | – |
OCR route (D) · 358 Logic & Math questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 358 Logic & Math questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| Qwen3.5-4B on · chat | Not applicable | – | – | – |
| Gemma 4 E2B chat | Not applicable | – | – | – |
Every setup of the track
2 setups of the models chosen; scored only when a cell is complete (306 Logic, 52 Math).
| Modelstate | Answers storedLogic · Math |
|---|---|
| Qwen3.5-4B on · chatPublished | A text only: not applicable (B only)B picture alone: 306/306 · 52/52C text + picture: not applicable (B only)D ocr route: not applicable (no OCR route for chat setups) |
| Gemma 4 E2B chatQueuedneeds a card | none stored yet |
Text-only chat models: not applicable, they take no images. The chat models that take images are listed above with their state.
The picture alone (B) · 306 Logic questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 queued · Table
Google Chance, 20% (five options)
The picture alone (B) · 306 Logic questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 12B Unified chattie group a Gemma 4 12B Unified (chat): 56.5%, 173 right of 306; Wilson 95% interval 50.9% to 62.0% | 56.5% | 173 / 306 | a | – |
| Gemma 4 E4B chattie group b Gemma 4 E4B (chat): 38.2%, 117 right of 306; Wilson 95% interval 33.0% to 43.8% | 38.2% | 117 / 306 | b | – |
| Qwen3.5-9B chatwaits for its server window | Queued | – | – | – |
The picture alone (B) · 52 Math questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 queued · Table
Google Chance, 20% (five options)
The picture alone (B) · 52 Math questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 12B Unified chattie group a Gemma 4 12B Unified (chat): 86.5%, 45 right of 52; Wilson 95% interval 74.7% to 93.3% | 86.5% | 45 / 52 | a | – |
| Gemma 4 E4B chattie group b Gemma 4 E4B (chat): 53.8%, 28 right of 52; Wilson 95% interval 40.5% to 66.7% | 53.8% | 28 / 52 | b | – |
| Qwen3.5-9B chatwaits for its server window | Queued | – | – | – |
The picture alone (B) · 358 Logic & Math questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
Not in the chart, no complete run here: 1 queued · Table
Google Chance, 20% (five options)
The picture alone (B) · 358 Logic & Math questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 12B Unified chattie group a Gemma 4 12B Unified (chat): 60.9%, 218 right of 358; Wilson 95% interval 55.7% to 65.8% | 60.9% | 218 / 358 | a | – |
| Gemma 4 E4B chattie group b Gemma 4 E4B (chat): 40.5%, 145 right of 358; Wilson 95% interval 35.5% to 45.7% | 40.5% | 145 / 358 | b | – |
| Qwen3.5-9B chatwaits for its server window | Queued | – | – | – |
Text only, the control (A) · 306 Logic questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (3 not applicable). Every setup in the table
Text only, the control (A) · 306 Logic questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| Gemma 4 12B Unified chat | Not applicable | – | – |
| Gemma 4 E4B chat | Not applicable | – | – |
| Qwen3.5-9B chat | Not applicable | – | – |
Text only, the control (A) · 52 Math questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (3 not applicable). Every setup in the table
Text only, the control (A) · 52 Math questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| Gemma 4 12B Unified chat | Not applicable | – | – |
| Gemma 4 E4B chat | Not applicable | – | – |
| Qwen3.5-9B chat | Not applicable | – | – |
Text only, the control (A) · 358 Logic & Math questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (3 not applicable). Every setup in the table
Text only, the control (A) · 358 Logic & Math questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| Gemma 4 12B Unified chat | Not applicable | – | – |
| Gemma 4 E4B chat | Not applicable | – | – |
| Qwen3.5-9B chat | Not applicable | – | – |
Text plus picture (C) · 306 Logic questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (3 not applicable). Every setup in the table
Text plus picture (C) · 306 Logic questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 12B Unified chat | Not applicable | – | – | – |
| Gemma 4 E4B chat | Not applicable | – | – | – |
| Qwen3.5-9B chat | Not applicable | – | – | – |
Text plus picture (C) · 52 Math questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (3 not applicable). Every setup in the table
Text plus picture (C) · 52 Math questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 12B Unified chat | Not applicable | – | – | – |
| Gemma 4 E4B chat | Not applicable | – | – | – |
| Qwen3.5-9B chat | Not applicable | – | – | – |
Text plus picture (C) · 358 Logic & Math questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (3 not applicable). Every setup in the table
Text plus picture (C) · 358 Logic & Math questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 12B Unified chat | Not applicable | – | – | – |
| Gemma 4 E4B chat | Not applicable | – | – | – |
| Qwen3.5-9B chat | Not applicable | – | – | – |
OCR route (D) · 306 Logic questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 306 Logic questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 12B Unified chat | Not applicable | – | – | – |
| Gemma 4 E4B chat | Not applicable | – | – | – |
| Qwen3.5-9B chat | Not applicable | – | – | – |
OCR route (D) · 52 Math questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 52 Math questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 12B Unified chat | Not applicable | – | – | – |
| Gemma 4 E4B chat | Not applicable | – | – | – |
| Qwen3.5-9B chat | Not applicable | – | – | – |
OCR route (D) · 358 Logic & Math questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 358 Logic & Math questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 12B Unified chat | Not applicable | – | – | – |
| Gemma 4 E4B chat | Not applicable | – | – | – |
| Qwen3.5-9B chat | Not applicable | – | – | – |
Every setup of the track
3 setups of the models chosen; scored only when a cell is complete (306 Logic, 52 Math).
| Modelstate | Answers storedLogic · Math |
|---|---|
| Gemma 4 12B Unified chatPublished | A text only: not applicable (B only)B picture alone: 306/306 · 52/52C text + picture: not applicable (B only)D ocr route: not applicable (no OCR route for chat setups) |
| Gemma 4 E4B chatPublished | A text only: not applicable (B only)B picture alone: 306/306 · 52/52C text + picture: not applicable (B only)D ocr route: not applicable (no OCR route for chat setups) |
| Qwen3.5-9B chatQueuedwaits for its server window | none stored yet |
Text-only chat models: not applicable, they take no images. The chat models that take images are listed above with their state.
The picture alone (B) · 306 Logic questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (2 queued). Every setup in the table
The picture alone (B) · 306 Logic questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 31B chatneeds a card | Queued | – | – | – |
| Muse Glimmer 30B chatJob 2 phase | Queued | – | – | – |
The picture alone (B) · 52 Math questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (2 queued). Every setup in the table
The picture alone (B) · 52 Math questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 31B chatneeds a card | Queued | – | – | – |
| Muse Glimmer 30B chatJob 2 phase | Queued | – | – | – |
The picture alone (B) · 358 Logic & Math questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (2 queued). Every setup in the table
The picture alone (B) · 358 Logic & Math questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 31B chatneeds a card | Queued | – | – | – |
| Muse Glimmer 30B chatJob 2 phase | Queued | – | – | – |
Text only, the control (A) · 306 Logic questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (2 not applicable). Every setup in the table
Text only, the control (A) · 306 Logic questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| Gemma 4 31B chat | Not applicable | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – |
Text only, the control (A) · 52 Math questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (2 not applicable). Every setup in the table
Text only, the control (A) · 52 Math questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| Gemma 4 31B chat | Not applicable | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – |
Text only, the control (A) · 358 Logic & Math questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (2 not applicable). Every setup in the table
Text only, the control (A) · 358 Logic & Math questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| Gemma 4 31B chat | Not applicable | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – |
Text plus picture (C) · 306 Logic questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (2 not applicable). Every setup in the table
Text plus picture (C) · 306 Logic questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 31B chat | Not applicable | – | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – | – |
Text plus picture (C) · 52 Math questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (2 not applicable). Every setup in the table
Text plus picture (C) · 52 Math questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 31B chat | Not applicable | – | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – | – |
Text plus picture (C) · 358 Logic & Math questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (2 not applicable). Every setup in the table
Text plus picture (C) · 358 Logic & Math questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 31B chat | Not applicable | – | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – | – |
OCR route (D) · 306 Logic questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 306 Logic questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 31B chat | Not applicable | – | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – | – |
OCR route (D) · 52 Math questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 52 Math questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 31B chat | Not applicable | – | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – | – |
OCR route (D) · 358 Logic & Math questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 358 Logic & Math questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| Gemma 4 31B chat | Not applicable | – | – | – |
| Muse Glimmer 30B chat | Not applicable | – | – | – |
Every setup of the track
2 setups of the models chosen; scored only when a cell is complete (306 Logic, 52 Math).
| Modelstate | Answers storedLogic · Math |
|---|---|
| Gemma 4 31B chatQueuedneeds a card | none stored yet |
| Muse Glimmer 30B chatQueuedJob 2 phase | none stored yet |
Text-only chat models: not applicable, they take no images. The chat models that take images are listed above with their state.
No setup of this class on the Vision track.
Text-only chat models: not applicable, they take no images. The chat models that take images are listed above with their state.
The picture alone (B) · 306 Logic questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (1 blocked). Every setup in the table
The picture alone (B) · 306 Logic questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| Muse Ember chatno qualified Logic & Math chat adapter for its hosted route, and no L&M budget for it | Blocked | – | – | – |
The picture alone (B) · 52 Math questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (1 blocked). Every setup in the table
The picture alone (B) · 52 Math questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| Muse Ember chatno qualified Logic & Math chat adapter for its hosted route, and no L&M budget for it | Blocked | – | – | – |
The picture alone (B) · 358 Logic & Math questions
Share answered right from a picture of the question alone (crops of the original page) · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (1 blocked). Every setup in the table
The picture alone (B) · 358 Logic & Math questions
Share answered right from a picture of the question alone (crops of the original page); scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | B − Apoints vs control |
|---|---|---|---|---|
| Muse Ember chatno qualified Logic & Math chat adapter for its hosted route, and no L&M budget for it | Blocked | – | – | – |
Text only, the control (A) · 306 Logic questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (1 not applicable). Every setup in the table
Text only, the control (A) · 306 Logic questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| Muse Ember chat | Not applicable | – | – |
Text only, the control (A) · 52 Math questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (1 not applicable). Every setup in the table
Text only, the control (A) · 52 Math questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| Muse Ember chat | Not applicable | – | – |
Text only, the control (A) · 358 Logic & Math questions
Share answered right from the question text with the figure left out (a control, not a score) · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (1 not applicable). Every setup in the table
Text only, the control (A) · 358 Logic & Math questions
Share answered right from the question text with the figure left out (a control, not a score); scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar |
|---|---|---|---|
| Muse Ember chat | Not applicable | – | – |
Text plus picture (C) · 306 Logic questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (1 not applicable). Every setup in the table
Text plus picture (C) · 306 Logic questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| Muse Ember chat | Not applicable | – | – | – |
Text plus picture (C) · 52 Math questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (1 not applicable). Every setup in the table
Text plus picture (C) · 52 Math questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| Muse Ember chat | Not applicable | – | – | – |
Text plus picture (C) · 358 Logic & Math questions
Share answered right from the question text with the picture after it · higher is better · the line: chance (20%) · a shared letter: not told apart
No complete run here yet (1 not applicable). Every setup in the table
Text plus picture (C) · 358 Logic & Math questions
Share answered right from the question text with the picture after it; scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | C − Apoints vs control |
|---|---|---|---|---|
| Muse Ember chat | Not applicable | – | – | – |
OCR route (D) · 306 Logic questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 306 Logic questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 306)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| Muse Ember chat | Not applicable | – | – | – |
OCR route (D) · 52 Math questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 52 Math questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 52)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| Muse Ember chat | Not applicable | – | – | – |
OCR route (D) · 358 Logic & Math questions
Share answered right from an OCR model's text of the picture · higher is better · the line: chance (20%) · a shared letter: not told apart
Not run: no OCR model has read the crops yet, so no setup has an OCR route (and no OCR error rate is measured). Every setup in the table
OCR route (D) · 358 Logic & Math questions
Share answered right from an OCR model's text of the picture; scored only when complete. Other setups: their state and the answers stored so far (n / 358)
| Model | Accuracyinterval on hover | Right / Nanswers stored | Tie groupMcNemar | D − Apoints vs control |
|---|---|---|---|---|
| Muse Ember chat | Not applicable | – | – | – |
Every setup of the track
1 setup of the models chosen; scored only when a cell is complete (306 Logic, 52 Math).
| Modelstate | Answers storedLogic · Math |
|---|---|
| Muse Ember chatBlockedno qualified Logic & Math chat adapter for its hosted route, and no L&M budget for it | none stored yet |
Text-only chat models: not applicable, they take no images. The chat models that take images are listed above with their state.
Info: conditions, averages, classes
The questions are asked in Turkish, as original page images, unlike the Text tab, which uses English translations. A, text only, is the control, not a score; B and C are for setups that take images.
- B · Picture alone. The crops of the page as the whole question, context first; instructions “Which option is correct?”; the options are the question's letters only. The crops are original Turkish pages, so B is a Turkish page-reading task.
- A · Text only. The Turkish question text and options as the text test sends them, with the figure left out. Where a stem or an option exists only as a picture, the text holds a placeholder (“[Görsel seçenek X]”: 90 Logic and 6 Math texts; “[Görsel soru]” for one Math stem). A control: how much of each question the text alone answers, not a score.
- C · Text + picture. Condition A's Turkish text, options and instructions, with the crops after the text.
- D · OCR route. One OCR model's text of the crops (context first, joined by a blank line, layout tags removed) as the question, with B's instructions and letters. Each OCR model is a separate route; a setup's best route is shown.
Against the control, on all 358 questions (Logic and Math pooled):
- B − A (picture alone against the control): +0.7 points · mean over 7 image models
- C − A (text + picture against the control): +5.1 points · mean over 7 image models
Each line is the mean, over the models with both cells complete, of a model's accuracy in that condition minus its accuracy in A; one point is one percentage point.
Class: Size by total parameters, as disclosed by the maker (for MoE models the label also shows active parameters): Tiny under 6B · Small 6B–15B · Medium over 15B up to 40B · Large over 40B. Models of undisclosed size, by standard list price per 1M input tokens: Efficient up to $0.50 · Workhorse up to $3 · Flagship over $3. Undisclosed: neither a disclosed size nor a list price.
9,662 graded answers in 27 runs, read at build time.
Where the self-hosted models ran
- d3-ultra · DGX Spark (GB10, aarch64, 121 GB unified memory) on spark2 · BF16 · server: d3_server.py (CPython 3.12.13; torch 2.14.1+cu130, transformers 5.17.0, accelerate 1.15.0, flash-linear-attention 0.5.2)
- GLiDE 27B · DGX Spark (GB10, aarch64, 121 GB unified memory) on spark2 · BF16 · server: glide.server (fastino/GLiDE-27B@4d32743 (its glide package))
- KnowLine-31B-Gen1.5 · DGX Spark (GB10, aarch64, 121 GB unified memory) on spark2 · BF16 · server: CPython 3.12.13; sglang 0.5.21, torch 2.13.0+cu130, transformers 5.12.1, flashinfer 0.6.18
API setups ran on their providers' services.
Method: questions, crops, conditions, OCR, runs and cost
Questions
- 358 questions: Logic 306 (numerical reasoning 269, verbal reasoning 37); Math 52 (basic concepts 6, digits and place value 7, division and divisibility 33, prime factorisation and gcd/lcm 1, rational numbers 1, ordering 2, factorisation 2), from 5 books of the published question bank (not named here). They are the answered questions whose figure the text test could not carry: every Logic question with a figure crop, and the Math questions the text parser dropped for their images (each with a number and an answer-key entry; one has its options only as pictures).
- One data issue: a Logic question printed its number inside its figure, so the text parser merged it into the shared context of the two questions after it. The vision track cuts that part out of both their crops and their text; that question itself is still missing from the benchmark.
- All four conditions are in Turkish, with the original text and no translation; the Text tab's results come from English translations of the other questions.
Crops
- 670 crops for 358 questions, on average 0.493 Mpx in 1.87 crops per question; 54 questions have 1 crop, 296 questions have 2 crops, 8 questions have 3 crops.
- Boxes: the layout blocks of the book scans (MinerU content_list, a 0–1000 grid of the page). Pages rendered with PyMuPDF at zoom 2.5; 6 px padding (the blocks are tight on the glyphs). Crops under 280 px high are padded with white below (136), crops over 1568 px a side scaled down (0).
- Context first, then the question; one crop per page column, so a question that runs on into the next column or page gets one crop per column.
- Answer keys are separate lines or strips on the page; no block of a key line lies inside any crop (0 found).
- Each question's blocks were found again on the page from its figure, its own number and its shared context, growing over neighbouring blocks of its text and stopping at another question's number, a group header or an answer key; questions with weak text coverage were listed and looked at. Every box was checked to sit on its text in all five books.
- Checked by eye: 30 questions from the five books (six at random per book plus every question the locator flagged). The review led to three fixes (the merged question above; a question's own figure cut as a “context” crop; options printed side by side split by the first column rule). After them every reviewed crop holds the whole question and nothing of its neighbours; the only foreign block is a book ornament inside one context box.
Conditions
- A · Control: text without the figure. The Turkish question text and options as the text test sends them, with the figure left out. Where a stem or an option exists only as a picture, the text holds a placeholder (“[Görsel seçenek X]”: 90 Logic and 6 Math texts; “[Görsel soru]” for one Math stem). A control: how much of each question the text alone answers, not a score.
- B · Crop only. The crops of the page as the whole question, context first; instructions “Which option is correct?”; the options are the question's letters only. The crops are original Turkish pages, so B is a Turkish page-reading task.
- C · Text + crop. Condition A's Turkish text, options and instructions, with the crops after the text.
- D · OCR of the crop. One OCR model's text of the crops (context first, joined by a blank line, layout tags removed) as the question, with B's instructions and letters. Each OCR model is a separate route; a setup's best route is shown.
The method follows the Turkish decision-model benchmark's page-image track. One question per request; the request is built by the benchmark runner's own code, with the images in each provider's format.
OCR (condition D)
No OCR model has read the crops yet, so condition D has no results. Planned, as in the Turkish decision-model benchmark's page-image track: Chandra OCR 2, dots.mocr, Surya OCR 2 and Baidu Unlimited-OCR-2, each its own D route, with each OCR model's character error rate measured against the clean text where it exists (normalised Levenshtein, micro-averaged over crops, the median also given).
Measures
- Accuracy over the questions of the part; a refusal or a letter outside the options counts as wrong. Its 95% Wilson interval shows on hover or tap.
- Tie group: setups in accuracy order, every pair compared on the same questions with the exact McNemar test, Holm's correction over the pairs; two setups share a letter unless the corrected p < 0.05 (as on the Text tab).
- Points: a model's accuracy in a condition minus its accuracy in A, in percentage points; the averages under the results (behind '*') are over the models that have both cells on all 358 questions.
- Logic and Math are reported separately and pooled. The vision results are ranked apart from the Text tab.
Runs
Decision setups ran each condition once on both benchmarks: a 20-question smoke run per benchmark, the runner's probability check, then the full run (306 Logic, 52 Math). Chat setups (setup lm-chat-vision-1, condition B only) ran once per benchmark with no smoke gate and no probabilities: one question per request, the letter of the reply's last “CEVAP: X” line scored, and a reply cut off or without that line counted wrong; their measure is accuracy only. Only complete cells are scored; every other setup of the track is listed with its state and the answers stored so far.
| Model | Cell | StartedUTC | Runnercommit | AnswersLogic / Math | Input tokensas reported | Request timesummed | Costwhole run |
|---|---|---|---|---|---|---|---|
| d1-ultra | A | 2026-11-12 11:44:51 UTC | df9ae27 | 306 / 52 | 109,235 | 91 s | $0.005 (provider price × reported usage) |
| d1-ultra | B | 2026-11-12 11:28:02 UTC | df9ae27 | 306 / 52 | 287,577 | 333 s | $0.012 (provider price × reported usage) |
| d1-ultra | C | 2026-11-12 11:59:56 UTC | df9ae27 | 306 / 52 | 381,060 | 274 s | $0.016 (provider price × reported usage) |
| d3-ultra | A | 2026-11-10 15:01:24 UTC | 9ba1b40 | 306 / 52 | 157,406 | 241 s | $0.015estimate · DGX Spark |
| d3-ultra | B | 2026-11-10 15:06:00 UTC | 9ba1b40 | 306 / 52 | 214,661 | 315 s | $0.020estimate · DGX Spark |
| d3-ultra | C | 2026-11-10 15:11:59 UTC | 9ba1b40 | 306 / 52 | 308,813 | 392 s | $0.024estimate · DGX Spark |
| Drex Max v1.1 | A | 2026-11-12 11:47:02 UTC | df9ae27 | 306 / 52 | 130,310 | 183 s | $0.008 (provider price × reported usage) |
| Drex Max v1.1 | B | 2026-11-12 11:28:11 UTC | df9ae27 | 306 / 52 | 210,723 | 406 s | $0.013 (provider price × reported usage) |
| Drex Max v1.1 | C | 2026-11-12 12:20:38 UTC | df9ae27 | 306 / 52 | 305,949 | 433 s | $0.018 (provider price × reported usage) |
| Gemini Decision 1 20261111 | A | 2026-11-12 11:35:54 UTC | df9ae27 | 306 / 52 | 124,834 | 95 s | $0.012 (provider price × reported usage) |
| Gemini Decision 1 20261111 | B | 2026-11-12 11:28:02 UTC | df9ae27 | 306 / 52 | 765,797 | 261 s | $0.077 (provider price × reported usage) |
| Gemini Decision 1 20261111 | C | 2026-11-12 11:49:37 UTC | df9ae27 | 306 / 52 | 853,041 | 256 s | $0.085 (provider price × reported usage) |
| Gemma 4 12B Unified chat | B | 2026-11-12 15:39:02 UTC | df9ae27 | 306 / 52 | 200,637 | 1,137 s | — |
| Gemma 4 E4B chat | B | 2026-11-12 16:02:00 UTC | df9ae27 | 306 / 52 | 199,205 | 524 s | — |
| GLiDE 27B | A | 2026-11-10 13:07:22 UTC | 9ba1b40 | 306 / 52 | 120,966 | 195 s | $0.012estimate · DGX Spark |
| GLiDE 27B | B | 2026-11-10 13:11:13 UTC | 9ba1b40 | 306 / 52 | 198,193 | 297 s | $0.018estimate · DGX Spark |
| GLiDE 27B | C | 2026-11-10 13:16:53 UTC | 9ba1b40 | 306 / 52 | 296,963 | 422 s | $0.026estimate · DGX Spark |
| Jev 1.14 | A | 2026-11-12 11:40:58 UTC | df9ae27 | 306 / 52 | 230,080 | 60 s | $0.010 (provider price × reported usage) |
| Jev 1.14 | B | 2026-11-12 11:30:58 UTC | df9ae27 | 304 / 52 | 600,252 | 103 s | $0.025 (provider price × reported usage) |
| Jev 1.14 | C | 2026-11-12 11:51:30 UTC | df9ae27 | 304 / 52 | 708,017 | 99 s | $0.030 (provider price × reported usage) |
| KnowLine-31B-Gen1.5 | A | 2026-11-11 00:45:13 UTC | 5fb5178 | 306 / 52 | 249,920 | 348 s | $0.022estimate · DGX Spark |
| KnowLine-31B-Gen1.5 | B | 2026-11-11 00:51:52 UTC | 5fb5178 | 306 / 52 | 212,863 | 402 s | $0.025estimate · DGX Spark |
| KnowLine-31B-Gen1.5 | C | 2026-11-11 00:59:38 UTC | 5fb5178 | 306 / 52 | 302,560 | 540 s | $0.033estimate · DGX Spark |
| Qwen3.5-4B on · chat | B | 2026-11-13 07:31:38 UTC | df9ae27 | 306 / 52 | 202,964 | 4,593 s | — |
| SubQ Snap 1.1 | A | 2026-11-12 11:42:58 UTC | df9ae27 | 306 / 52 | 130,310 | 139 s | $0.005 (provider price × reported usage) |
| SubQ Snap 1.1 | B | 2026-11-12 11:28:07 UTC | df9ae27 | 306 / 52 | 210,723 | 320 s | $0.008 (provider price × reported usage) |
| SubQ Snap 1.1 | C | 2026-11-12 11:53:53 UTC | df9ae27 | 306 / 52 | 305,949 | 322 s | $0.012 (provider price × reported usage) |
Cost
API setups: the provider's price × the usage it reported. Models served on our NVIDIA DGX Spark (GB10) show “DGX Spark”; hover, focus or tap it for the estimate: the run's GPU time (first to last answer, pauses over 10 minutes left out) × $0.213/h, as on the Text tab (see its Method). Drex Max keeps the maker's token list price, as in its text runs: that price does not depend on the hardware.