Experiments
Experiments
Repeat checks
Logic · 1,125 questions
| Rankby the repeat | Model | First runright / N | Repeatright / N | Changed answersvs the first run | Status |
|---|---|---|---|---|---|
| 1 | REAX 1.3667 → 667 of 1,125DetailsFirst run 667 / 1,125 · Repeat 667 / 1,125 · Changed answers 0 First run: reax13_en, started 2026-11-07 17:02:40 UTC; hardware NVIDIA RTX PRO 6000 Blackwell (96 GB); cost basis estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h. Repeat 2: reax13__logic_en_standard__repeat2.sqlite, runner run 1; reax13 via reax, thinking off, seed 0, batch size 1, harness b55c226; 2026-11-11 14:36:21 UTC to 2026-11-11 14:42:00 UTC; on the same machine as the first run (NVIDIA RTX PRO 6000 Blackwell (96 GB)); its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 667 / 1,125 | 667 / 1,125 | 0 | same answers |
| 2 | KnowLine-31B-Gen1.5664 → 663 of 1,125DetailsFirst run 664 / 1,125 · Repeat 663 / 1,125 · Changed answers 2 First run: knowline-31b_en, started 2026-11-10 15:26:45 UTC; hardware NVIDIA DGX Spark (GB10); cost basis estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h. Repeat 2: knowline-31b__logic_en_standard__repeat2.sqlite, runner run 1; PelaAI/KnowLine-31B-Gen1.5 via knowline, thinking off, seed 0, batch size 1, harness 3b14502; 2026-11-10 23:55:08 UTC to 2026-11-11 00:14:53 UTC; on the same machine as the first run (NVIDIA DGX Spark (GB10)); its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 664 / 1,125 | 663 / 1,125 | 2 | same answers except 2 |
| 3 | GLiDE 27B657 → 657 of 1,125DetailsFirst run 657 / 1,125 · Repeat 657 / 1,125 · Changed answers 0 First run: glide-27b_en, started 2026-11-10 08:52:51 UTC; hardware NVIDIA DGX Spark (GB10); cost basis estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h. Repeat 2: glide-27b__logic_en_standard__repeat2.sqlite, runner run 1; glide-27b via glide, thinking off, seed 0, batch size 1, harness b55c226; 2026-11-11 16:16:28 UTC to 2026-11-11 16:27:00 UTC; on the same machine as the first run (NVIDIA DGX Spark (GB10)); its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 657 / 1,125 | 657 / 1,125 | 0 | same answers |
| 4 | GLiDE (v1.2)621 → 621 of 1,125DetailsFirst run 621 / 1,125 · Repeat 621 / 1,125 · Changed answers 0 First run: glide-v1.2_en, started 2026-11-07 13:43:06 UTC; hardware NVIDIA RTX PRO 6000 Blackwell (96 GB); cost basis maker's list price. Repeat 2: glide-v1.2__logic_en_standard__repeat2.sqlite, runner run 1; glide-v1.2 via glide, thinking off, seed 0, batch size 1, harness b55c226; 2026-11-11 16:16:57 UTC to 2026-11-11 16:19:17 UTC; on the same machine as the first run (NVIDIA RTX PRO 6000 Blackwell (96 GB)); its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 621 / 1,125 | 621 / 1,125 | 0 | same answers |
| 5 | d1-ultra618 → 616 of 1,125DetailsFirst run 618 / 1,125 · Repeat 616 / 1,125 · Changed answers 209 First run: d1-ultra_en, started 2026-11-07 14:56:53 UTC; hardware the provider’s API; cost basis provider price × reported usage. Repeat 2: d1-ultra__logic_en_standard__repeat2.sqlite, runner run 1; d1-ultra via liquid, thinking off, seed 0, batch size 1, harness b55c226; 2026-11-11 11:35:21 UTC to 2026-11-11 11:40:17 UTC; on the provider’s API; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 618 / 1,125 | 616 / 1,125 | 209 | same answers except 209 |
| 6 | Drex Max v1.1616 → 616 of 1,125DetailsFirst run 616 / 1,125 · Repeat 616 / 1,125 · Changed answers 0 First run: drex-max-1.1_en, started 2026-11-07 15:07:00 UTC; hardware the provider’s API; cost basis provider price × reported usage. Repeat 2: drex-max-1.1__logic_en_standard__repeat2.sqlite, runner run 1; drex-max-v1.1 via nace, thinking off, seed 0, batch size 1, harness b55c226; 2026-11-11 16:02:44 UTC to 2026-11-11 16:12:18 UTC; on the provider’s API; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 616 / 1,125 | 616 / 1,125 | 0 | same answers |
| 7 | GLiDE 2B363 → 363 of 1,125DetailsFirst run 363 / 1,125 · Repeat 363 / 1,125 · Changed answers 0 First run: glide-2b_en, started 2026-11-10 12:41:00 UTC; hardware NVIDIA DGX Spark (GB10); cost basis estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h. Repeat 2: glide-2b__logic_en_standard__repeat2.sqlite, runner run 1; glide-2b via glide, thinking off, seed 0, batch size 1, harness 6ad0eac; 2026-11-13 23:48:47 UTC to 2026-11-13 23:49:30 UTC; on the same machine as the first run (NVIDIA DGX Spark (GB10)); its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 363 / 1,125 | 363 / 1,125 | 0 | same answers |
| 8 | d1-2B345 → 345 of 1,125DetailsFirst run 345 / 1,125 · Repeat 345 / 1,125 · Changed answers 0 First run: d1-2b_en, started 2026-11-07 13:49:27 UTC; hardware NVIDIA RTX PRO 6000 Blackwell (96 GB); cost basis estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h. Repeat 2: d1-2b__logic_en_standard__repeat2.sqlite, runner run 1; LiquidAI/d1-2B via d1-server, thinking off, seed 0, batch size 1, harness 6ad0eac; 2026-11-14 16:07:59 UTC to 2026-11-14 16:08:29 UTC; on the same machine as the first run (NVIDIA RTX PRO 6000 Blackwell (96 GB)); its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 345 / 1,125 | 345 / 1,125 | 0 | same answers |
Math · 2,163 questions
| Rankby the repeat | Model | First runright / N | Repeatright / N | Changed answersvs the first run | Status |
|---|---|---|---|---|---|
| 1 | REAX 1.31,687 → 1,687 of 2,163DetailsFirst run 1,687 / 2,163 · Repeat 1,687 / 2,163 · Changed answers 0 First run: reax13_en, started 2026-11-07 17:12:29 UTC; hardware NVIDIA RTX PRO 6000 Blackwell (96 GB); cost basis estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h. Repeat 2: reax13__math_en_standard__repeat2.sqlite, runner run 1; reax13 via reax, thinking off, seed 0, batch size 1, harness b55c226; 2026-11-11 14:42:00 UTC to 2026-11-11 14:49:11 UTC; on the same machine as the first run (NVIDIA RTX PRO 6000 Blackwell (96 GB)); its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 1,687 / 2,163 | 1,687 / 2,163 | 0 | same answers |
| 2 | KnowLine-31B-Gen1.51,674 → 1,674 of 2,163DetailsFirst run 1,674 / 2,163 · Repeat 1,674 / 2,163 · Changed answers 0 First run: knowline-31b_en, started 2026-11-10 15:46:33 UTC; hardware NVIDIA DGX Spark (GB10); cost basis estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h. Repeat 2: knowline-31b__math_en_standard__repeat2.sqlite, runner run 1; PelaAI/KnowLine-31B-Gen1.5 via knowline, thinking off, seed 0, batch size 1, harness 3b14502; 2026-11-11 00:14:54 UTC to 2026-11-11 00:43:33 UTC; on the same machine as the first run (NVIDIA DGX Spark (GB10)); its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 1,674 / 2,163 | 1,674 / 2,163 | 0 | same answers |
| 3 | d1-ultra1,603 → 1,601 of 2,163DetailsFirst run 1,603 / 2,163 · Repeat 1,601 / 2,163 · Changed answers 232 First run: d1-ultra_en, started 2026-11-07 15:19:54 UTC; hardware the provider’s API; cost basis provider price × reported usage. Repeat 2: d1-ultra__math_en_standard__repeat2.sqlite, runner run 1; d1-ultra via liquid, thinking off, seed 0, batch size 1, harness b55c226; 2026-11-11 11:40:19 UTC to 2026-11-11 11:49:39 UTC; on the provider’s API; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 1,603 / 2,163 | 1,601 / 2,163 | 232 | same answers except 232 |
| 4 | GLiDE 27B1,557 → 1,557 of 2,163DetailsFirst run 1,557 / 2,163 · Repeat 1,557 / 2,163 · Changed answers 0 First run: glide-27b_en, started 2026-11-10 09:03:33 UTC; hardware NVIDIA DGX Spark (GB10); cost basis estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h. Repeat 2: glide-27b__math_en_standard__repeat2.sqlite, runner run 1; glide-27b via glide, thinking off, seed 0, batch size 1, harness b55c226; 2026-11-11 16:27:01 UTC to 2026-11-11 16:42:54 UTC; on the same machine as the first run (NVIDIA DGX Spark (GB10)); its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 1,557 / 2,163 | 1,557 / 2,163 | 0 | same answers |
| 5 | Drex Max v1.11,516 → 1,516 of 2,163DetailsFirst run 1,516 / 2,163 · Repeat 1,516 / 2,163 · Changed answers 0 First run: drex-max-1.1_en, started 2026-11-07 15:26:35 UTC; hardware the provider’s API; cost basis provider price × reported usage. Repeat 2: drex-max-1.1__math_en_standard__repeat2.sqlite, runner run 1; drex-max-v1.1 via nace, thinking off, seed 0, batch size 1, harness b55c226; 2026-11-11 16:12:18 UTC to 2026-11-11 16:30:37 UTC; on the provider’s API; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 1,516 / 2,163 | 1,516 / 2,163 | 0 | same answers |
| 6 | GLiDE (v1.2)1,511 → 1,511 of 2,163DetailsFirst run 1,511 / 2,163 · Repeat 1,511 / 2,163 · Changed answers 0 First run: glide-v1.2_en, started 2026-11-07 13:54:15 UTC; hardware NVIDIA RTX PRO 6000 Blackwell (96 GB); cost basis maker's list price. Repeat 2: glide-v1.2__math_en_standard__repeat2.sqlite, runner run 1; glide-v1.2 via glide, thinking off, seed 0, batch size 1, harness b55c226; 2026-11-11 16:19:18 UTC to 2026-11-11 16:22:21 UTC; on the same machine as the first run (NVIDIA RTX PRO 6000 Blackwell (96 GB)); its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 1,511 / 2,163 | 1,511 / 2,163 | 0 | same answers |
| 7 | d1-2B802 → 802 of 2,163DetailsFirst run 802 / 2,163 · Repeat 802 / 2,163 · Changed answers 0 First run: d1-2b_en, started 2026-11-07 13:51:57 UTC; hardware NVIDIA RTX PRO 6000 Blackwell (96 GB); cost basis estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h. Repeat 2: d1-2b__math_en_standard__repeat2.sqlite, runner run 1; LiquidAI/d1-2B via d1-server, thinking off, seed 0, batch size 1, harness 6ad0eac; 2026-11-14 16:08:31 UTC to 2026-11-14 16:09:23 UTC; on the same machine as the first run (NVIDIA RTX PRO 6000 Blackwell (96 GB)); its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 802 / 2,163 | 802 / 2,163 | 0 | same answers |
| 8 | GLiDE 2B795 → 795 of 2,163DetailsFirst run 795 / 2,163 · Repeat 795 / 2,163 · Changed answers 0 First run: glide-2b_en, started 2026-11-10 12:41:45 UTC; hardware NVIDIA DGX Spark (GB10); cost basis estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h. Repeat 2: glide-2b__math_en_standard__repeat2.sqlite, runner run 1; glide-2b via glide, thinking off, seed 0, batch size 1, harness 6ad0eac; 2026-11-13 23:49:31 UTC to 2026-11-13 23:50:37 UTC; on the same machine as the first run (NVIDIA DGX Spark (GB10)); its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 795 / 2,163 | 795 / 2,163 | 0 | same answers |
Vision · Picture alone (B) · 306 Logic questions
| Rankby the repeat | Model | First runright / N | Repeatright / N | Changed answersvs the first run | Status |
|---|---|---|---|---|---|
| 1 | Gemini Decision 1 (20261111)103 → 103 of 306DetailsFirst run 103 / 306 · Repeat 103 / 306 · Changed answers 0 First run: runner run 2 of results/gemini-decision-1__logic_vision_B.sqlite (the Vision tab's run), 2026-11-12 11:28:02 UTC to 2026-11-12 11:31:52 UTC. Repeat 2: gemini-decision-1__logic_vision_B.sqlite, runner run 3; google/gemini-decision-1 via openrouter, harness 6ad0eac; 2026-11-12 11:56:25 UTC to 2026-11-12 11:59:05 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 103 / 306 | 103 / 306 | 0 | same answers |
| 2 | KnowLine-31B-Gen1.599 → 98 of 306DetailsFirst run 99 / 306 · Repeat 98 / 306 · Changed answers 6 First run: runner run 2 of results/knowline-31b__logic_vision_B.sqlite (the Vision tab's run), 2026-11-11 00:51:52 UTC to 2026-11-11 00:57:54 UTC. Repeat 2: knowline-31b__logic_vision_B__repeat2.sqlite, runner run 1; PelaAI/KnowLine-31B-Gen1.5 via knowline, harness 6ad0eac; 2026-11-12 15:32:21 UTC to 2026-11-12 15:38:18 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 99 / 306 | 98 / 306 | 6 | same answers except 6 |
| 3 | GLiDE 27B94 → 94 of 306DetailsFirst run 94 / 306 · Repeat 94 / 306 · Changed answers 0 First run: runner run 2 of results/glide-27b__logic_vision_B.sqlite (the Vision tab's run), 2026-11-10 13:11:13 UTC to 2026-11-10 13:15:39 UTC. Repeat 2: glide-27b__logic_vision_B__repeat2.sqlite, runner run 1; glide-27b via glide, harness 6ad0eac; 2026-11-12 16:04:04 UTC to 2026-11-12 16:08:33 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 94 / 306 | 94 / 306 | 0 | same answers |
| 4 | d1-ultra84 → 92 of 306DetailsFirst run 84 / 306 · Repeat 92 / 306 · Changed answers 99 First run: runner run 2 of results/d1-ultra__logic_vision_B.sqlite (the Vision tab's run), 2026-11-12 11:28:02 UTC to 2026-11-12 11:32:41 UTC. Repeat 2: d1-ultra__logic_vision_B.sqlite, runner run 3; d1-ultra via liquid, harness 6ad0eac; 2026-11-12 14:56:07 UTC to 2026-11-12 14:59:51 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 84 / 306 | 92 / 306 | 99 | same answers except 99 · run 12 Nov under the 32-cell roster (#5424/#5427) |
| 5 | Drex Max v1.191 → 91 of 306DetailsFirst run 91 / 306 · Repeat 91 / 306 · Changed answers 0 First run: runner run 2 of results/drex-max-1.1__logic_vision_B.sqlite (the Vision tab's run), 2026-11-12 11:28:11 UTC to 2026-11-12 11:34:13 UTC. Repeat 2: drex-max-1.1__logic_vision_B.sqlite, runner run 3; drex-max-v1.1 via nace, harness 6ad0eac; 2026-11-12 14:56:07 UTC to 2026-11-12 15:02:15 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 91 / 306 | 91 / 306 | 0 | same answers · run 12 Nov under the 32-cell roster (#5424/#5427) |
| 6 | d3-ultra89 → 89 of 306DetailsFirst run 89 / 306 · Repeat 89 / 306 · Changed answers 0 First run: runner run 2 of results/d3-ultra__logic_vision_B.sqlite (the Vision tab's run), 2026-11-10 15:06:00 UTC to 2026-11-10 15:10:41 UTC. Repeat 2: d3-ultra__logic_vision_B__repeat2.sqlite, runner run 1; vllm-sr/d3-ultra via d3-server, harness 6ad0eac; 2026-11-12 15:49:33 UTC to 2026-11-12 15:54:13 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 89 / 306 | 89 / 306 | 0 | same answers |
| 7 | SubQ Snap 1.187 → 87 of 306DetailsFirst run 87 / 306 · Repeat 87 / 306 · Changed answers 0 First run: runner run 2 of results/subq-1.1__logic_vision_B.sqlite (the Vision tab's run), 2026-11-12 11:28:07 UTC to 2026-11-12 11:32:53 UTC. Repeat 2: subq-1.1__logic_vision_B.sqlite, runner run 3; subq-snap-1.1 via subq, harness 6ad0eac; 2026-11-12 14:56:07 UTC to 2026-11-12 15:01:00 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 87 / 306 | 87 / 306 | 0 | same answers · run 12 Nov under the 32-cell roster (#5424/#5427) |
| 8 | Jev 1.1474 → 79 of 304DetailsFirst run 74 / 304 · Repeat 79 / 304 · Changed answers 44 First run: runner run 3 of results/jev-1.14__logic_vision_B.sqlite (the Vision tab's run), 2026-11-12 11:31:13 UTC to 2026-11-12 11:32:44 UTC. Repeat 2: jev-1.14__logic_vision_B.sqlite, runner run 4; jev-1.14.0 via jev, harness 6ad0eac; 2026-11-12 14:56:07 UTC to 2026-11-12 14:57:38 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 74 / 304 | 79 / 304 | 44 | same answers except 44 · 304 of 306: 2 crops refused by the provider; run 12 Nov under the 32-cell roster (#5424/#5427) |
Vision · Text + picture (C) · 306 Logic questions
| Rankby the repeat | Model | First runright / N | Repeatright / N | Changed answersvs the first run | Status |
|---|---|---|---|---|---|
| 1 | GLiDE 27B118 → 118 of 306DetailsFirst run 118 / 306 · Repeat 118 / 306 · Changed answers 0 First run: runner run 2 of results/glide-27b__logic_vision_C.sqlite (the Vision tab's run), 2026-11-10 13:16:53 UTC to 2026-11-10 13:23:18 UTC. Repeat 2: glide-27b__logic_vision_C__repeat2.sqlite, runner run 1; glide-27b via glide, harness 6ad0eac; 2026-11-12 16:08:34 UTC to 2026-11-12 16:15:00 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 118 / 306 | 118 / 306 | 0 | same answers |
| 2 | KnowLine-31B-Gen1.5114 → 116 of 306DetailsFirst run 114 / 306 · Repeat 116 / 306 · Changed answers 6 First run: runner run 2 of results/knowline-31b__logic_vision_C.sqlite (the Vision tab's run), 2026-11-11 00:59:38 UTC to 2026-11-11 01:07:52 UTC. Repeat 2: knowline-31b__logic_vision_C__repeat2.sqlite, runner run 1; PelaAI/KnowLine-31B-Gen1.5 via knowline, harness 6ad0eac; 2026-11-12 16:21:20 UTC to 2026-11-12 16:29:32 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 114 / 306 | 116 / 306 | 6 | same answers except 6 |
| 3 | Gemini Decision 1 (20261111)106 → 106 of 306DetailsFirst run 106 / 306 · Repeat 106 / 306 · Changed answers 0 First run: runner run 2 of results/gemini-decision-1__logic_vision_C.sqlite (the Vision tab's run), 2026-11-12 11:49:37 UTC to 2026-11-12 11:53:20 UTC. Repeat 2: gemini-decision-1__logic_vision_C.sqlite, runner run 3; google/gemini-decision-1 via openrouter, harness 6ad0eac; 2026-11-12 13:01:43 UTC to 2026-11-12 13:04:07 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 106 / 306 | 106 / 306 | 0 | same answers |
| 4 | SubQ Snap 1.1103 → 103 of 306DetailsFirst run 103 / 306 · Repeat 103 / 306 · Changed answers 0 First run: runner run 2 of results/subq-1.1__logic_vision_C.sqlite (the Vision tab's run), 2026-11-12 11:53:53 UTC to 2026-11-12 11:58:40 UTC. Repeat 2: subq-1.1__logic_vision_C.sqlite, runner run 3; subq-snap-1.1 via subq, harness 6ad0eac; 2026-11-12 13:01:43 UTC to 2026-11-12 13:06:31 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 103 / 306 | 103 / 306 | 0 | same answers |
| 5 | d1-ultra109 → 102 of 306DetailsFirst run 109 / 306 · Repeat 102 / 306 · Changed answers 79 First run: runner run 2 of results/d1-ultra__logic_vision_C.sqlite (the Vision tab's run), 2026-11-12 11:59:56 UTC to 2026-11-12 12:03:59 UTC. Repeat 2: d1-ultra__logic_vision_C.sqlite, runner run 3; d1-ultra via liquid, harness 6ad0eac; 2026-11-12 13:01:43 UTC to 2026-11-12 13:06:05 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 109 / 306 | 102 / 306 | 79 | same answers except 79 |
| 6 | Drex Max v1.1101 → 101 of 306DetailsFirst run 101 / 306 · Repeat 101 / 306 · Changed answers 0 First run: runner run 2 of results/drex-max-1.1__logic_vision_C.sqlite (the Vision tab's run), 2026-11-12 12:20:38 UTC to 2026-11-12 12:27:06 UTC. Repeat 2: drex-max-1.1__logic_vision_C.sqlite, runner run 3; drex-max-v1.1 via nace, harness 6ad0eac; 2026-11-12 13:01:43 UTC to 2026-11-12 13:07:51 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 101 / 306 | 101 / 306 | 0 | same answers |
| 7 | d3-ultra97 → 97 of 306DetailsFirst run 97 / 306 · Repeat 97 / 306 · Changed answers 0 First run: runner run 2 of results/d3-ultra__logic_vision_C.sqlite (the Vision tab's run), 2026-11-10 15:11:59 UTC to 2026-11-10 15:17:51 UTC. Repeat 2: d3-ultra__logic_vision_C__repeat2.sqlite, runner run 1; vllm-sr/d3-ultra via d3-server, harness 6ad0eac; 2026-11-12 15:54:14 UTC to 2026-11-12 16:00:09 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 97 / 306 | 97 / 306 | 0 | same answers |
| 8 | Jev 1.1487 → 88 of 304DetailsFirst run 87 / 304 · Repeat 88 / 304 · Changed answers 39 First run: runner run 2 of results/jev-1.14__logic_vision_C.sqlite (the Vision tab's run), 2026-11-12 11:51:51 UTC to 2026-11-12 11:53:22 UTC. Repeat 2: jev-1.14__logic_vision_C.sqlite, runner run 3; jev-1.14.0 via jev, harness 6ad0eac; 2026-11-12 14:57:48 UTC to 2026-11-12 14:59:19 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 87 / 304 | 88 / 304 | 39 | same answers except 39 · 304 of 306: 2 crops refused by the provider; run 12 Nov under the 32-cell roster (#5424/#5427) |
Vision · Picture alone (B) · 52 Math questions
| Rankby the repeat | Model | First runright / N | Repeatright / N | Changed answersvs the first run | Status |
|---|---|---|---|---|---|
| 1 | d1-ultra27 → 24 of 52DetailsFirst run 27 / 52 · Repeat 24 / 52 · Changed answers 17 First run: runner run 2 of results/d1-ultra__math_vision_B.sqlite (the Vision tab's run), 2026-11-12 11:32:52 UTC to 2026-11-12 11:33:49 UTC. Repeat 2: d1-ultra__math_vision_B.sqlite, runner run 3; d1-ultra via liquid, harness 6ad0eac; 2026-11-12 14:59:58 UTC to 2026-11-12 15:00:25 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 27 / 52 | 24 / 52 | 17 | same answers except 17 · run 12 Nov under the 32-cell roster (#5424/#5427) |
| 2 | Gemini Decision 1 (20261111)24 → 24 of 52DetailsFirst run 24 / 52 · Repeat 24 / 52 · Changed answers 0 First run: runner run 2 of results/gemini-decision-1__math_vision_B.sqlite (the Vision tab's run), 2026-11-12 11:32:06 UTC to 2026-11-12 11:32:39 UTC. Repeat 2: gemini-decision-1__math_vision_B.sqlite, runner run 3; google/gemini-decision-1 via openrouter, harness 6ad0eac; 2026-11-12 13:04:14 UTC to 2026-11-12 13:04:36 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 24 / 52 | 24 / 52 | 0 | same answers |
| 3 | SubQ Snap 1.122 → 22 of 52DetailsFirst run 22 / 52 · Repeat 22 / 52 · Changed answers 0 First run: runner run 2 of results/subq-1.1__math_vision_B.sqlite (the Vision tab's run), 2026-11-12 11:33:08 UTC to 2026-11-12 11:33:44 UTC. Repeat 2: subq-1.1__math_vision_B.sqlite, runner run 3; subq-snap-1.1 via subq, harness 6ad0eac; 2026-11-12 15:01:08 UTC to 2026-11-12 15:01:44 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 22 / 52 | 22 / 52 | 0 | same answers · run 12 Nov under the 32-cell roster (#5424/#5427) |
| 4 | Drex Max v1.121 → 21 of 52DetailsFirst run 21 / 52 · Repeat 21 / 52 · Changed answers 0 First run: runner run 2 of results/drex-max-1.1__math_vision_B.sqlite (the Vision tab's run), 2026-11-12 11:34:31 UTC to 2026-11-12 11:35:18 UTC. Repeat 2: drex-max-1.1__math_vision_B.sqlite, runner run 3; drex-max-v1.1 via nace, harness 6ad0eac; 2026-11-12 15:02:18 UTC to 2026-11-12 15:03:07 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 21 / 52 | 21 / 52 | 0 | same answers · run 12 Nov under the 32-cell roster (#5424/#5427) |
| 5 | GLiDE 27B20 → 20 of 52DetailsFirst run 20 / 52 · Repeat 20 / 52 · Changed answers 0 First run: runner run 2 of results/glide-27b__math_vision_B.sqlite (the Vision tab's run), 2026-11-10 13:15:53 UTC to 2026-11-10 13:16:25 UTC. Repeat 2: glide-27b__math_vision_B__repeat2.sqlite, runner run 1; glide-27b via glide, harness 6ad0eac; 2026-11-12 16:15:01 UTC to 2026-11-12 16:15:33 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 20 / 52 | 20 / 52 | 0 | same answers |
| 6 | d3-ultra19 → 19 of 52DetailsFirst run 19 / 52 · Repeat 19 / 52 · Changed answers 0 First run: runner run 2 of results/d3-ultra__math_vision_B.sqlite (the Vision tab's run), 2026-11-10 15:10:57 UTC to 2026-11-10 15:11:34 UTC. Repeat 2: d3-ultra__math_vision_B__repeat2.sqlite, runner run 1; vllm-sr/d3-ultra via d3-server, harness 6ad0eac; 2026-11-12 16:00:10 UTC to 2026-11-12 16:00:47 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 19 / 52 | 19 / 52 | 0 | same answers |
| 7 | KnowLine-31B-Gen1.519 → 19 of 52DetailsFirst run 19 / 52 · Repeat 19 / 52 · Changed answers 0 First run: runner run 2 of results/knowline-31b__math_vision_B.sqlite (the Vision tab's run), 2026-11-11 00:58:16 UTC to 2026-11-11 00:58:58 UTC. Repeat 2: knowline-31b__math_vision_B__repeat2.sqlite, runner run 1; PelaAI/KnowLine-31B-Gen1.5 via knowline, harness 6ad0eac; 2026-11-12 16:29:33 UTC to 2026-11-12 16:30:25 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 19 / 52 | 19 / 52 | 0 | same answers |
| 8 | Jev 1.1421 → 18 of 52DetailsFirst run 21 / 52 · Repeat 18 / 52 · Changed answers 10 First run: runner run 2 of results/jev-1.14__math_vision_B.sqlite (the Vision tab's run), 2026-11-12 11:30:58 UTC to 2026-11-12 11:31:12 UTC. Repeat 2: jev-1.14__math_vision_B.sqlite, runner run 3; jev-1.14.0 via jev, harness 6ad0eac; 2026-11-12 14:59:29 UTC to 2026-11-12 14:59:42 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 21 / 52 | 18 / 52 | 10 | same answers except 10 · run 12 Nov under the 32-cell roster (#5424/#5427) |
Vision · Text + picture (C) · 52 Math questions
| Rankby the repeat | Model | First runright / N | Repeatright / N | Changed answersvs the first run | Status |
|---|---|---|---|---|---|
| 1 | Jev 1.1427 → 28 of 52DetailsFirst run 27 / 52 · Repeat 28 / 52 · Changed answers 4 First run: runner run 2 of results/jev-1.14__math_vision_C.sqlite (the Vision tab's run), 2026-11-12 11:51:30 UTC to 2026-11-12 11:51:43 UTC. Repeat 2: jev-1.14__math_vision_C.sqlite, runner run 3; jev-1.14.0 via jev, harness 6ad0eac; 2026-11-12 14:59:50 UTC to 2026-11-12 15:00:02 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 27 / 52 | 28 / 52 | 4 | same answers except 4 · run 12 Nov under the 32-cell roster (#5424/#5427) |
| 2 | d3-ultra25 → 25 of 52DetailsFirst run 25 / 52 · Repeat 25 / 52 · Changed answers 0 First run: runner run 2 of results/d3-ultra__math_vision_C.sqlite (the Vision tab's run), 2026-11-10 15:18:08 UTC to 2026-11-10 15:18:50 UTC. Repeat 2: d3-ultra__math_vision_C__repeat2.sqlite, runner run 1; vllm-sr/d3-ultra via d3-server, harness 6ad0eac; 2026-11-12 15:46:32 UTC to 2026-11-12 15:47:14 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 25 / 52 | 25 / 52 | 0 | same answers |
| 3 | Gemini Decision 1 (20261111)25 → 25 of 52DetailsFirst run 25 / 52 · Repeat 25 / 52 · Changed answers 0 First run: runner run 2 of results/gemini-decision-1__math_vision_C.sqlite (the Vision tab's run), 2026-11-12 11:53:31 UTC to 2026-11-12 11:54:07 UTC. Repeat 2: gemini-decision-1__math_vision_C.sqlite, runner run 3; google/gemini-decision-1 via openrouter, harness 6ad0eac; 2026-11-12 13:04:45 UTC to 2026-11-12 13:05:05 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 25 / 52 | 25 / 52 | 0 | same answers |
| 4 | d1-ultra24 → 24 of 52DetailsFirst run 24 / 52 · Repeat 24 / 52 · Changed answers 9 First run: runner run 2 of results/d1-ultra__math_vision_C.sqlite (the Vision tab's run), 2026-11-12 12:04:13 UTC to 2026-11-12 12:04:48 UTC. Repeat 2: d1-ultra__math_vision_C.sqlite, runner run 3; d1-ultra via liquid, harness 6ad0eac; 2026-11-12 13:06:14 UTC to 2026-11-12 13:06:42 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 24 / 52 | 24 / 52 | 9 | same answers except 9 |
| 5 | GLiDE 27B24 → 24 of 52DetailsFirst run 24 / 52 · Repeat 24 / 52 · Changed answers 0 First run: runner run 2 of results/glide-27b__math_vision_C.sqlite (the Vision tab's run), 2026-11-10 13:23:34 UTC to 2026-11-10 13:24:14 UTC. Repeat 2: glide-27b__math_vision_C__repeat2.sqlite, runner run 1; glide-27b via glide, harness 6ad0eac; 2026-11-12 15:40:18 UTC to 2026-11-12 15:40:57 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 24 / 52 | 24 / 52 | 0 | same answers |
| 6 | KnowLine-31B-Gen1.524 → 23 of 52DetailsFirst run 24 / 52 · Repeat 23 / 52 · Changed answers 1 First run: runner run 2 of results/knowline-31b__math_vision_C.sqlite (the Vision tab's run), 2026-11-11 01:08:17 UTC to 2026-11-11 01:09:05 UTC. Repeat 2: knowline-31b__math_vision_C__repeat2.sqlite, runner run 1; PelaAI/KnowLine-31B-Gen1.5 via knowline, harness 6ad0eac; 2026-11-12 15:38:19 UTC to 2026-11-12 15:39:20 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 24 / 52 | 23 / 52 | 1 | same answers except 1 |
| 7 | SubQ Snap 1.123 → 23 of 52DetailsFirst run 23 / 52 · Repeat 23 / 52 · Changed answers 0 First run: runner run 2 of results/subq-1.1__math_vision_C.sqlite (the Vision tab's run), 2026-11-12 11:58:55 UTC to 2026-11-12 11:59:32 UTC. Repeat 2: subq-1.1__math_vision_C.sqlite, runner run 3; subq-snap-1.1 via subq, harness 6ad0eac; 2026-11-12 15:01:49 UTC to 2026-11-12 15:02:25 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 23 / 52 | 23 / 52 | 0 | same answers · run 12 Nov under the 32-cell roster (#5424/#5427) |
| 8 | Drex Max v1.117 → 17 of 52DetailsFirst run 17 / 52 · Repeat 17 / 52 · Changed answers 0 First run: runner run 2 of results/drex-max-1.1__math_vision_C.sqlite (the Vision tab's run), 2026-11-12 12:27:28 UTC to 2026-11-12 12:28:16 UTC. Repeat 2: drex-max-1.1__math_vision_C.sqlite, runner run 3; drex-max-v1.1 via nace, harness 6ad0eac; 2026-11-12 15:03:09 UTC to 2026-11-12 15:03:57 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site. Changed answers: questions where the repeat chose another letter than the first run. | 17 / 52 | 17 / 52 | 0 | same answers · run 12 Nov under the 32-cell roster (#5424/#5427) |
A repeat asks the same questions again with the same settings and compares the letters with the first run. The published scores stay the first runs (the Vision tab's too); repeats are not ranked or summarised anywhere else.
Withdrawn from the Vision page
| Modelright / N per condition: Logic · Math · Both; status |
|---|
| LFM3-VL-2B off · chatA text only: Logic 69/306 · Math 3/52 · Both 72/358B picture alone: Logic 69/306 · Math 13/52 · Both 82/358C text + picture: Logic 66/306 · Math 17/52 · Both 83/358 Status: Withdrawn for a check: Math text-only scored 3/52, below chance, possibly an answer-format problem. Turkish is not listed on its model card. |
| LFM3-VL-2B on · chatA text only: Logic 11/306 · Math 10/52 · Both 21/358B picture alone: Logic 58/306 · Math 20/52 · Both 78/358C text + picture: Logic 99/306 · Math 34/52 · Both 133/358 Status: Smoke test failed (6 of 25 answers cut off at 8,192 tokens with thinking on); full runs made anyway at the same settings (Ümit #8797/#8799, zeki #8830), unreadable answers counted wrong; kept here, off the Vision page, until Ümit says otherwise. |
Taken off the Vision page until they are checked; their runs stay stored. The Vision page ranks the other setups without them.