Experiments

Experiments

Repeat checks

Logic · 1,125 questions

Rankby the repeatModelFirst runright / NRepeatright / NChanged answersvs the first runStatus
1REAX 1.3667 → 667 of 1,125
Details

First run 667 / 1,125 · Repeat 667 / 1,125 · Changed answers 0

First run: reax13_en, started 2026-11-07 17:02:40 UTC; hardware NVIDIA RTX PRO 6000 Blackwell (96 GB); cost basis estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h.

Repeat 2: reax13__logic_en_standard__repeat2.sqlite, runner run 1; reax13 via reax, thinking off, seed 0, batch size 1, harness b55c226; 2026-11-11 14:36:21 UTC to 2026-11-11 14:42:00 UTC; on the same machine as the first run (NVIDIA RTX PRO 6000 Blackwell (96 GB)); its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

667 / 1,125667 / 1,1250same answers
2KnowLine-31B-Gen1.5664 → 663 of 1,125
Details

First run 664 / 1,125 · Repeat 663 / 1,125 · Changed answers 2

First run: knowline-31b_en, started 2026-11-10 15:26:45 UTC; hardware NVIDIA DGX Spark (GB10); cost basis estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h.

Repeat 2: knowline-31b__logic_en_standard__repeat2.sqlite, runner run 1; PelaAI/KnowLine-31B-Gen1.5 via knowline, thinking off, seed 0, batch size 1, harness 3b14502; 2026-11-10 23:55:08 UTC to 2026-11-11 00:14:53 UTC; on the same machine as the first run (NVIDIA DGX Spark (GB10)); its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

664 / 1,125663 / 1,1252same answers except 2
3GLiDE 27B657 → 657 of 1,125
Details

First run 657 / 1,125 · Repeat 657 / 1,125 · Changed answers 0

First run: glide-27b_en, started 2026-11-10 08:52:51 UTC; hardware NVIDIA DGX Spark (GB10); cost basis estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h.

Repeat 2: glide-27b__logic_en_standard__repeat2.sqlite, runner run 1; glide-27b via glide, thinking off, seed 0, batch size 1, harness b55c226; 2026-11-11 16:16:28 UTC to 2026-11-11 16:27:00 UTC; on the same machine as the first run (NVIDIA DGX Spark (GB10)); its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

657 / 1,125657 / 1,1250same answers
4GLiDE (v1.2)621 → 621 of 1,125
Details

First run 621 / 1,125 · Repeat 621 / 1,125 · Changed answers 0

First run: glide-v1.2_en, started 2026-11-07 13:43:06 UTC; hardware NVIDIA RTX PRO 6000 Blackwell (96 GB); cost basis maker's list price.

Repeat 2: glide-v1.2__logic_en_standard__repeat2.sqlite, runner run 1; glide-v1.2 via glide, thinking off, seed 0, batch size 1, harness b55c226; 2026-11-11 16:16:57 UTC to 2026-11-11 16:19:17 UTC; on the same machine as the first run (NVIDIA RTX PRO 6000 Blackwell (96 GB)); its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

621 / 1,125621 / 1,1250same answers
5d1-ultra618 → 616 of 1,125
Details

First run 618 / 1,125 · Repeat 616 / 1,125 · Changed answers 209

First run: d1-ultra_en, started 2026-11-07 14:56:53 UTC; hardware the provider’s API; cost basis provider price × reported usage.

Repeat 2: d1-ultra__logic_en_standard__repeat2.sqlite, runner run 1; d1-ultra via liquid, thinking off, seed 0, batch size 1, harness b55c226; 2026-11-11 11:35:21 UTC to 2026-11-11 11:40:17 UTC; on the provider’s API; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

618 / 1,125616 / 1,125209same answers except 209
6Drex Max v1.1616 → 616 of 1,125
Details

First run 616 / 1,125 · Repeat 616 / 1,125 · Changed answers 0

First run: drex-max-1.1_en, started 2026-11-07 15:07:00 UTC; hardware the provider’s API; cost basis provider price × reported usage.

Repeat 2: drex-max-1.1__logic_en_standard__repeat2.sqlite, runner run 1; drex-max-v1.1 via nace, thinking off, seed 0, batch size 1, harness b55c226; 2026-11-11 16:02:44 UTC to 2026-11-11 16:12:18 UTC; on the provider’s API; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

616 / 1,125616 / 1,1250same answers
7GLiDE 2B363 → 363 of 1,125
Details

First run 363 / 1,125 · Repeat 363 / 1,125 · Changed answers 0

First run: glide-2b_en, started 2026-11-10 12:41:00 UTC; hardware NVIDIA DGX Spark (GB10); cost basis estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h.

Repeat 2: glide-2b__logic_en_standard__repeat2.sqlite, runner run 1; glide-2b via glide, thinking off, seed 0, batch size 1, harness 6ad0eac; 2026-11-13 23:48:47 UTC to 2026-11-13 23:49:30 UTC; on the same machine as the first run (NVIDIA DGX Spark (GB10)); its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

363 / 1,125363 / 1,1250same answers
8d1-2B345 → 345 of 1,125
Details

First run 345 / 1,125 · Repeat 345 / 1,125 · Changed answers 0

First run: d1-2b_en, started 2026-11-07 13:49:27 UTC; hardware NVIDIA RTX PRO 6000 Blackwell (96 GB); cost basis estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h.

Repeat 2: d1-2b__logic_en_standard__repeat2.sqlite, runner run 1; LiquidAI/d1-2B via d1-server, thinking off, seed 0, batch size 1, harness 6ad0eac; 2026-11-14 16:07:59 UTC to 2026-11-14 16:08:29 UTC; on the same machine as the first run (NVIDIA RTX PRO 6000 Blackwell (96 GB)); its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

345 / 1,125345 / 1,1250same answers

Math · 2,163 questions

Rankby the repeatModelFirst runright / NRepeatright / NChanged answersvs the first runStatus
1REAX 1.31,687 → 1,687 of 2,163
Details

First run 1,687 / 2,163 · Repeat 1,687 / 2,163 · Changed answers 0

First run: reax13_en, started 2026-11-07 17:12:29 UTC; hardware NVIDIA RTX PRO 6000 Blackwell (96 GB); cost basis estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h.

Repeat 2: reax13__math_en_standard__repeat2.sqlite, runner run 1; reax13 via reax, thinking off, seed 0, batch size 1, harness b55c226; 2026-11-11 14:42:00 UTC to 2026-11-11 14:49:11 UTC; on the same machine as the first run (NVIDIA RTX PRO 6000 Blackwell (96 GB)); its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

1,687 / 2,1631,687 / 2,1630same answers
2KnowLine-31B-Gen1.51,674 → 1,674 of 2,163
Details

First run 1,674 / 2,163 · Repeat 1,674 / 2,163 · Changed answers 0

First run: knowline-31b_en, started 2026-11-10 15:46:33 UTC; hardware NVIDIA DGX Spark (GB10); cost basis estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h.

Repeat 2: knowline-31b__math_en_standard__repeat2.sqlite, runner run 1; PelaAI/KnowLine-31B-Gen1.5 via knowline, thinking off, seed 0, batch size 1, harness 3b14502; 2026-11-11 00:14:54 UTC to 2026-11-11 00:43:33 UTC; on the same machine as the first run (NVIDIA DGX Spark (GB10)); its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

1,674 / 2,1631,674 / 2,1630same answers
3d1-ultra1,603 → 1,601 of 2,163
Details

First run 1,603 / 2,163 · Repeat 1,601 / 2,163 · Changed answers 232

First run: d1-ultra_en, started 2026-11-07 15:19:54 UTC; hardware the provider’s API; cost basis provider price × reported usage.

Repeat 2: d1-ultra__math_en_standard__repeat2.sqlite, runner run 1; d1-ultra via liquid, thinking off, seed 0, batch size 1, harness b55c226; 2026-11-11 11:40:19 UTC to 2026-11-11 11:49:39 UTC; on the provider’s API; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

1,603 / 2,1631,601 / 2,163232same answers except 232
4GLiDE 27B1,557 → 1,557 of 2,163
Details

First run 1,557 / 2,163 · Repeat 1,557 / 2,163 · Changed answers 0

First run: glide-27b_en, started 2026-11-10 09:03:33 UTC; hardware NVIDIA DGX Spark (GB10); cost basis estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h.

Repeat 2: glide-27b__math_en_standard__repeat2.sqlite, runner run 1; glide-27b via glide, thinking off, seed 0, batch size 1, harness b55c226; 2026-11-11 16:27:01 UTC to 2026-11-11 16:42:54 UTC; on the same machine as the first run (NVIDIA DGX Spark (GB10)); its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

1,557 / 2,1631,557 / 2,1630same answers
5Drex Max v1.11,516 → 1,516 of 2,163
Details

First run 1,516 / 2,163 · Repeat 1,516 / 2,163 · Changed answers 0

First run: drex-max-1.1_en, started 2026-11-07 15:26:35 UTC; hardware the provider’s API; cost basis provider price × reported usage.

Repeat 2: drex-max-1.1__math_en_standard__repeat2.sqlite, runner run 1; drex-max-v1.1 via nace, thinking off, seed 0, batch size 1, harness b55c226; 2026-11-11 16:12:18 UTC to 2026-11-11 16:30:37 UTC; on the provider’s API; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

1,516 / 2,1631,516 / 2,1630same answers
6GLiDE (v1.2)1,511 → 1,511 of 2,163
Details

First run 1,511 / 2,163 · Repeat 1,511 / 2,163 · Changed answers 0

First run: glide-v1.2_en, started 2026-11-07 13:54:15 UTC; hardware NVIDIA RTX PRO 6000 Blackwell (96 GB); cost basis maker's list price.

Repeat 2: glide-v1.2__math_en_standard__repeat2.sqlite, runner run 1; glide-v1.2 via glide, thinking off, seed 0, batch size 1, harness b55c226; 2026-11-11 16:19:18 UTC to 2026-11-11 16:22:21 UTC; on the same machine as the first run (NVIDIA RTX PRO 6000 Blackwell (96 GB)); its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

1,511 / 2,1631,511 / 2,1630same answers
7d1-2B802 → 802 of 2,163
Details

First run 802 / 2,163 · Repeat 802 / 2,163 · Changed answers 0

First run: d1-2b_en, started 2026-11-07 13:51:57 UTC; hardware NVIDIA RTX PRO 6000 Blackwell (96 GB); cost basis estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h.

Repeat 2: d1-2b__math_en_standard__repeat2.sqlite, runner run 1; LiquidAI/d1-2B via d1-server, thinking off, seed 0, batch size 1, harness 6ad0eac; 2026-11-14 16:08:31 UTC to 2026-11-14 16:09:23 UTC; on the same machine as the first run (NVIDIA RTX PRO 6000 Blackwell (96 GB)); its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

802 / 2,163802 / 2,1630same answers
8GLiDE 2B795 → 795 of 2,163
Details

First run 795 / 2,163 · Repeat 795 / 2,163 · Changed answers 0

First run: glide-2b_en, started 2026-11-10 12:41:45 UTC; hardware NVIDIA DGX Spark (GB10); cost basis estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h.

Repeat 2: glide-2b__math_en_standard__repeat2.sqlite, runner run 1; glide-2b via glide, thinking off, seed 0, batch size 1, harness 6ad0eac; 2026-11-13 23:49:31 UTC to 2026-11-13 23:50:37 UTC; on the same machine as the first run (NVIDIA DGX Spark (GB10)); its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

795 / 2,163795 / 2,1630same answers

Vision · Picture alone (B) · 306 Logic questions

Rankby the repeatModelFirst runright / NRepeatright / NChanged answersvs the first runStatus
1Gemini Decision 1 (20261111)103 → 103 of 306
Details

First run 103 / 306 · Repeat 103 / 306 · Changed answers 0

First run: runner run 2 of results/gemini-decision-1__logic_vision_B.sqlite (the Vision tab's run), 2026-11-12 11:28:02 UTC to 2026-11-12 11:31:52 UTC.

Repeat 2: gemini-decision-1__logic_vision_B.sqlite, runner run 3; google/gemini-decision-1 via openrouter, harness 6ad0eac; 2026-11-12 11:56:25 UTC to 2026-11-12 11:59:05 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

103 / 306103 / 3060same answers
2KnowLine-31B-Gen1.599 → 98 of 306
Details

First run 99 / 306 · Repeat 98 / 306 · Changed answers 6

First run: runner run 2 of results/knowline-31b__logic_vision_B.sqlite (the Vision tab's run), 2026-11-11 00:51:52 UTC to 2026-11-11 00:57:54 UTC.

Repeat 2: knowline-31b__logic_vision_B__repeat2.sqlite, runner run 1; PelaAI/KnowLine-31B-Gen1.5 via knowline, harness 6ad0eac; 2026-11-12 15:32:21 UTC to 2026-11-12 15:38:18 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

99 / 30698 / 3066same answers except 6
3GLiDE 27B94 → 94 of 306
Details

First run 94 / 306 · Repeat 94 / 306 · Changed answers 0

First run: runner run 2 of results/glide-27b__logic_vision_B.sqlite (the Vision tab's run), 2026-11-10 13:11:13 UTC to 2026-11-10 13:15:39 UTC.

Repeat 2: glide-27b__logic_vision_B__repeat2.sqlite, runner run 1; glide-27b via glide, harness 6ad0eac; 2026-11-12 16:04:04 UTC to 2026-11-12 16:08:33 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

94 / 30694 / 3060same answers
4d1-ultra84 → 92 of 306
Details

First run 84 / 306 · Repeat 92 / 306 · Changed answers 99

First run: runner run 2 of results/d1-ultra__logic_vision_B.sqlite (the Vision tab's run), 2026-11-12 11:28:02 UTC to 2026-11-12 11:32:41 UTC.

Repeat 2: d1-ultra__logic_vision_B.sqlite, runner run 3; d1-ultra via liquid, harness 6ad0eac; 2026-11-12 14:56:07 UTC to 2026-11-12 14:59:51 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

84 / 30692 / 30699same answers except 99 · run 12 Nov under the 32-cell roster (#5424/#5427)
5Drex Max v1.191 → 91 of 306
Details

First run 91 / 306 · Repeat 91 / 306 · Changed answers 0

First run: runner run 2 of results/drex-max-1.1__logic_vision_B.sqlite (the Vision tab's run), 2026-11-12 11:28:11 UTC to 2026-11-12 11:34:13 UTC.

Repeat 2: drex-max-1.1__logic_vision_B.sqlite, runner run 3; drex-max-v1.1 via nace, harness 6ad0eac; 2026-11-12 14:56:07 UTC to 2026-11-12 15:02:15 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

91 / 30691 / 3060same answers · run 12 Nov under the 32-cell roster (#5424/#5427)
6d3-ultra89 → 89 of 306
Details

First run 89 / 306 · Repeat 89 / 306 · Changed answers 0

First run: runner run 2 of results/d3-ultra__logic_vision_B.sqlite (the Vision tab's run), 2026-11-10 15:06:00 UTC to 2026-11-10 15:10:41 UTC.

Repeat 2: d3-ultra__logic_vision_B__repeat2.sqlite, runner run 1; vllm-sr/d3-ultra via d3-server, harness 6ad0eac; 2026-11-12 15:49:33 UTC to 2026-11-12 15:54:13 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

89 / 30689 / 3060same answers
7SubQ Snap 1.187 → 87 of 306
Details

First run 87 / 306 · Repeat 87 / 306 · Changed answers 0

First run: runner run 2 of results/subq-1.1__logic_vision_B.sqlite (the Vision tab's run), 2026-11-12 11:28:07 UTC to 2026-11-12 11:32:53 UTC.

Repeat 2: subq-1.1__logic_vision_B.sqlite, runner run 3; subq-snap-1.1 via subq, harness 6ad0eac; 2026-11-12 14:56:07 UTC to 2026-11-12 15:01:00 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

87 / 30687 / 3060same answers · run 12 Nov under the 32-cell roster (#5424/#5427)
8Jev 1.1474 → 79 of 304
Details

First run 74 / 304 · Repeat 79 / 304 · Changed answers 44

First run: runner run 3 of results/jev-1.14__logic_vision_B.sqlite (the Vision tab's run), 2026-11-12 11:31:13 UTC to 2026-11-12 11:32:44 UTC.

Repeat 2: jev-1.14__logic_vision_B.sqlite, runner run 4; jev-1.14.0 via jev, harness 6ad0eac; 2026-11-12 14:56:07 UTC to 2026-11-12 14:57:38 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

74 / 30479 / 30444same answers except 44 · 304 of 306: 2 crops refused by the provider; run 12 Nov under the 32-cell roster (#5424/#5427)

Vision · Text + picture (C) · 306 Logic questions

Rankby the repeatModelFirst runright / NRepeatright / NChanged answersvs the first runStatus
1GLiDE 27B118 → 118 of 306
Details

First run 118 / 306 · Repeat 118 / 306 · Changed answers 0

First run: runner run 2 of results/glide-27b__logic_vision_C.sqlite (the Vision tab's run), 2026-11-10 13:16:53 UTC to 2026-11-10 13:23:18 UTC.

Repeat 2: glide-27b__logic_vision_C__repeat2.sqlite, runner run 1; glide-27b via glide, harness 6ad0eac; 2026-11-12 16:08:34 UTC to 2026-11-12 16:15:00 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

118 / 306118 / 3060same answers
2KnowLine-31B-Gen1.5114 → 116 of 306
Details

First run 114 / 306 · Repeat 116 / 306 · Changed answers 6

First run: runner run 2 of results/knowline-31b__logic_vision_C.sqlite (the Vision tab's run), 2026-11-11 00:59:38 UTC to 2026-11-11 01:07:52 UTC.

Repeat 2: knowline-31b__logic_vision_C__repeat2.sqlite, runner run 1; PelaAI/KnowLine-31B-Gen1.5 via knowline, harness 6ad0eac; 2026-11-12 16:21:20 UTC to 2026-11-12 16:29:32 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

114 / 306116 / 3066same answers except 6
3Gemini Decision 1 (20261111)106 → 106 of 306
Details

First run 106 / 306 · Repeat 106 / 306 · Changed answers 0

First run: runner run 2 of results/gemini-decision-1__logic_vision_C.sqlite (the Vision tab's run), 2026-11-12 11:49:37 UTC to 2026-11-12 11:53:20 UTC.

Repeat 2: gemini-decision-1__logic_vision_C.sqlite, runner run 3; google/gemini-decision-1 via openrouter, harness 6ad0eac; 2026-11-12 13:01:43 UTC to 2026-11-12 13:04:07 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

106 / 306106 / 3060same answers
4SubQ Snap 1.1103 → 103 of 306
Details

First run 103 / 306 · Repeat 103 / 306 · Changed answers 0

First run: runner run 2 of results/subq-1.1__logic_vision_C.sqlite (the Vision tab's run), 2026-11-12 11:53:53 UTC to 2026-11-12 11:58:40 UTC.

Repeat 2: subq-1.1__logic_vision_C.sqlite, runner run 3; subq-snap-1.1 via subq, harness 6ad0eac; 2026-11-12 13:01:43 UTC to 2026-11-12 13:06:31 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

103 / 306103 / 3060same answers
5d1-ultra109 → 102 of 306
Details

First run 109 / 306 · Repeat 102 / 306 · Changed answers 79

First run: runner run 2 of results/d1-ultra__logic_vision_C.sqlite (the Vision tab's run), 2026-11-12 11:59:56 UTC to 2026-11-12 12:03:59 UTC.

Repeat 2: d1-ultra__logic_vision_C.sqlite, runner run 3; d1-ultra via liquid, harness 6ad0eac; 2026-11-12 13:01:43 UTC to 2026-11-12 13:06:05 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

109 / 306102 / 30679same answers except 79
6Drex Max v1.1101 → 101 of 306
Details

First run 101 / 306 · Repeat 101 / 306 · Changed answers 0

First run: runner run 2 of results/drex-max-1.1__logic_vision_C.sqlite (the Vision tab's run), 2026-11-12 12:20:38 UTC to 2026-11-12 12:27:06 UTC.

Repeat 2: drex-max-1.1__logic_vision_C.sqlite, runner run 3; drex-max-v1.1 via nace, harness 6ad0eac; 2026-11-12 13:01:43 UTC to 2026-11-12 13:07:51 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

101 / 306101 / 3060same answers
7d3-ultra97 → 97 of 306
Details

First run 97 / 306 · Repeat 97 / 306 · Changed answers 0

First run: runner run 2 of results/d3-ultra__logic_vision_C.sqlite (the Vision tab's run), 2026-11-10 15:11:59 UTC to 2026-11-10 15:17:51 UTC.

Repeat 2: d3-ultra__logic_vision_C__repeat2.sqlite, runner run 1; vllm-sr/d3-ultra via d3-server, harness 6ad0eac; 2026-11-12 15:54:14 UTC to 2026-11-12 16:00:09 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

97 / 30697 / 3060same answers
8Jev 1.1487 → 88 of 304
Details

First run 87 / 304 · Repeat 88 / 304 · Changed answers 39

First run: runner run 2 of results/jev-1.14__logic_vision_C.sqlite (the Vision tab's run), 2026-11-12 11:51:51 UTC to 2026-11-12 11:53:22 UTC.

Repeat 2: jev-1.14__logic_vision_C.sqlite, runner run 3; jev-1.14.0 via jev, harness 6ad0eac; 2026-11-12 14:57:48 UTC to 2026-11-12 14:59:19 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

87 / 30488 / 30439same answers except 39 · 304 of 306: 2 crops refused by the provider; run 12 Nov under the 32-cell roster (#5424/#5427)

Vision · Picture alone (B) · 52 Math questions

Rankby the repeatModelFirst runright / NRepeatright / NChanged answersvs the first runStatus
1d1-ultra27 → 24 of 52
Details

First run 27 / 52 · Repeat 24 / 52 · Changed answers 17

First run: runner run 2 of results/d1-ultra__math_vision_B.sqlite (the Vision tab's run), 2026-11-12 11:32:52 UTC to 2026-11-12 11:33:49 UTC.

Repeat 2: d1-ultra__math_vision_B.sqlite, runner run 3; d1-ultra via liquid, harness 6ad0eac; 2026-11-12 14:59:58 UTC to 2026-11-12 15:00:25 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

27 / 5224 / 5217same answers except 17 · run 12 Nov under the 32-cell roster (#5424/#5427)
2Gemini Decision 1 (20261111)24 → 24 of 52
Details

First run 24 / 52 · Repeat 24 / 52 · Changed answers 0

First run: runner run 2 of results/gemini-decision-1__math_vision_B.sqlite (the Vision tab's run), 2026-11-12 11:32:06 UTC to 2026-11-12 11:32:39 UTC.

Repeat 2: gemini-decision-1__math_vision_B.sqlite, runner run 3; google/gemini-decision-1 via openrouter, harness 6ad0eac; 2026-11-12 13:04:14 UTC to 2026-11-12 13:04:36 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

24 / 5224 / 520same answers
3SubQ Snap 1.122 → 22 of 52
Details

First run 22 / 52 · Repeat 22 / 52 · Changed answers 0

First run: runner run 2 of results/subq-1.1__math_vision_B.sqlite (the Vision tab's run), 2026-11-12 11:33:08 UTC to 2026-11-12 11:33:44 UTC.

Repeat 2: subq-1.1__math_vision_B.sqlite, runner run 3; subq-snap-1.1 via subq, harness 6ad0eac; 2026-11-12 15:01:08 UTC to 2026-11-12 15:01:44 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

22 / 5222 / 520same answers · run 12 Nov under the 32-cell roster (#5424/#5427)
4Drex Max v1.121 → 21 of 52
Details

First run 21 / 52 · Repeat 21 / 52 · Changed answers 0

First run: runner run 2 of results/drex-max-1.1__math_vision_B.sqlite (the Vision tab's run), 2026-11-12 11:34:31 UTC to 2026-11-12 11:35:18 UTC.

Repeat 2: drex-max-1.1__math_vision_B.sqlite, runner run 3; drex-max-v1.1 via nace, harness 6ad0eac; 2026-11-12 15:02:18 UTC to 2026-11-12 15:03:07 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

21 / 5221 / 520same answers · run 12 Nov under the 32-cell roster (#5424/#5427)
5GLiDE 27B20 → 20 of 52
Details

First run 20 / 52 · Repeat 20 / 52 · Changed answers 0

First run: runner run 2 of results/glide-27b__math_vision_B.sqlite (the Vision tab's run), 2026-11-10 13:15:53 UTC to 2026-11-10 13:16:25 UTC.

Repeat 2: glide-27b__math_vision_B__repeat2.sqlite, runner run 1; glide-27b via glide, harness 6ad0eac; 2026-11-12 16:15:01 UTC to 2026-11-12 16:15:33 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

20 / 5220 / 520same answers
6d3-ultra19 → 19 of 52
Details

First run 19 / 52 · Repeat 19 / 52 · Changed answers 0

First run: runner run 2 of results/d3-ultra__math_vision_B.sqlite (the Vision tab's run), 2026-11-10 15:10:57 UTC to 2026-11-10 15:11:34 UTC.

Repeat 2: d3-ultra__math_vision_B__repeat2.sqlite, runner run 1; vllm-sr/d3-ultra via d3-server, harness 6ad0eac; 2026-11-12 16:00:10 UTC to 2026-11-12 16:00:47 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

19 / 5219 / 520same answers
7KnowLine-31B-Gen1.519 → 19 of 52
Details

First run 19 / 52 · Repeat 19 / 52 · Changed answers 0

First run: runner run 2 of results/knowline-31b__math_vision_B.sqlite (the Vision tab's run), 2026-11-11 00:58:16 UTC to 2026-11-11 00:58:58 UTC.

Repeat 2: knowline-31b__math_vision_B__repeat2.sqlite, runner run 1; PelaAI/KnowLine-31B-Gen1.5 via knowline, harness 6ad0eac; 2026-11-12 16:29:33 UTC to 2026-11-12 16:30:25 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

19 / 5219 / 520same answers
8Jev 1.1421 → 18 of 52
Details

First run 21 / 52 · Repeat 18 / 52 · Changed answers 10

First run: runner run 2 of results/jev-1.14__math_vision_B.sqlite (the Vision tab's run), 2026-11-12 11:30:58 UTC to 2026-11-12 11:31:12 UTC.

Repeat 2: jev-1.14__math_vision_B.sqlite, runner run 3; jev-1.14.0 via jev, harness 6ad0eac; 2026-11-12 14:59:29 UTC to 2026-11-12 14:59:42 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

21 / 5218 / 5210same answers except 10 · run 12 Nov under the 32-cell roster (#5424/#5427)

Vision · Text + picture (C) · 52 Math questions

Rankby the repeatModelFirst runright / NRepeatright / NChanged answersvs the first runStatus
1Jev 1.1427 → 28 of 52
Details

First run 27 / 52 · Repeat 28 / 52 · Changed answers 4

First run: runner run 2 of results/jev-1.14__math_vision_C.sqlite (the Vision tab's run), 2026-11-12 11:51:30 UTC to 2026-11-12 11:51:43 UTC.

Repeat 2: jev-1.14__math_vision_C.sqlite, runner run 3; jev-1.14.0 via jev, harness 6ad0eac; 2026-11-12 14:59:50 UTC to 2026-11-12 15:00:02 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

27 / 5228 / 524same answers except 4 · run 12 Nov under the 32-cell roster (#5424/#5427)
2d3-ultra25 → 25 of 52
Details

First run 25 / 52 · Repeat 25 / 52 · Changed answers 0

First run: runner run 2 of results/d3-ultra__math_vision_C.sqlite (the Vision tab's run), 2026-11-10 15:18:08 UTC to 2026-11-10 15:18:50 UTC.

Repeat 2: d3-ultra__math_vision_C__repeat2.sqlite, runner run 1; vllm-sr/d3-ultra via d3-server, harness 6ad0eac; 2026-11-12 15:46:32 UTC to 2026-11-12 15:47:14 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

25 / 5225 / 520same answers
3Gemini Decision 1 (20261111)25 → 25 of 52
Details

First run 25 / 52 · Repeat 25 / 52 · Changed answers 0

First run: runner run 2 of results/gemini-decision-1__math_vision_C.sqlite (the Vision tab's run), 2026-11-12 11:53:31 UTC to 2026-11-12 11:54:07 UTC.

Repeat 2: gemini-decision-1__math_vision_C.sqlite, runner run 3; google/gemini-decision-1 via openrouter, harness 6ad0eac; 2026-11-12 13:04:45 UTC to 2026-11-12 13:05:05 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

25 / 5225 / 520same answers
4d1-ultra24 → 24 of 52
Details

First run 24 / 52 · Repeat 24 / 52 · Changed answers 9

First run: runner run 2 of results/d1-ultra__math_vision_C.sqlite (the Vision tab's run), 2026-11-12 12:04:13 UTC to 2026-11-12 12:04:48 UTC.

Repeat 2: d1-ultra__math_vision_C.sqlite, runner run 3; d1-ultra via liquid, harness 6ad0eac; 2026-11-12 13:06:14 UTC to 2026-11-12 13:06:42 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

24 / 5224 / 529same answers except 9
5GLiDE 27B24 → 24 of 52
Details

First run 24 / 52 · Repeat 24 / 52 · Changed answers 0

First run: runner run 2 of results/glide-27b__math_vision_C.sqlite (the Vision tab's run), 2026-11-10 13:23:34 UTC to 2026-11-10 13:24:14 UTC.

Repeat 2: glide-27b__math_vision_C__repeat2.sqlite, runner run 1; glide-27b via glide, harness 6ad0eac; 2026-11-12 15:40:18 UTC to 2026-11-12 15:40:57 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

24 / 5224 / 520same answers
6KnowLine-31B-Gen1.524 → 23 of 52
Details

First run 24 / 52 · Repeat 23 / 52 · Changed answers 1

First run: runner run 2 of results/knowline-31b__math_vision_C.sqlite (the Vision tab's run), 2026-11-11 01:08:17 UTC to 2026-11-11 01:09:05 UTC.

Repeat 2: knowline-31b__math_vision_C__repeat2.sqlite, runner run 1; PelaAI/KnowLine-31B-Gen1.5 via knowline, harness 6ad0eac; 2026-11-12 15:38:19 UTC to 2026-11-12 15:39:20 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

24 / 5223 / 521same answers except 1
7SubQ Snap 1.123 → 23 of 52
Details

First run 23 / 52 · Repeat 23 / 52 · Changed answers 0

First run: runner run 2 of results/subq-1.1__math_vision_C.sqlite (the Vision tab's run), 2026-11-12 11:58:55 UTC to 2026-11-12 11:59:32 UTC.

Repeat 2: subq-1.1__math_vision_C.sqlite, runner run 3; subq-snap-1.1 via subq, harness 6ad0eac; 2026-11-12 15:01:49 UTC to 2026-11-12 15:02:25 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

23 / 5223 / 520same answers · run 12 Nov under the 32-cell roster (#5424/#5427)
8Drex Max v1.117 → 17 of 52
Details

First run 17 / 52 · Repeat 17 / 52 · Changed answers 0

First run: runner run 2 of results/drex-max-1.1__math_vision_C.sqlite (the Vision tab's run), 2026-11-12 12:27:28 UTC to 2026-11-12 12:28:16 UTC.

Repeat 2: drex-max-1.1__math_vision_C.sqlite, runner run 3; drex-max-v1.1 via nace, harness 6ad0eac; 2026-11-12 15:03:09 UTC to 2026-11-12 15:03:57 UTC; the same endpoint as the first run; its cost is not counted anywhere on this site.

Changed answers: questions where the repeat chose another letter than the first run.

17 / 5217 / 520same answers · run 12 Nov under the 32-cell roster (#5424/#5427)

A repeat asks the same questions again with the same settings and compares the letters with the first run. The published scores stay the first runs (the Vision tab's too); repeats are not ranked or summarised anywhere else.

Withdrawn from the Vision page

Modelright / N per condition: Logic · Math · Both; status
LFM3-VL-2B off · chatA text only: Logic 69/306 · Math 3/52 · Both 72/358B picture alone: Logic 69/306 · Math 13/52 · Both 82/358C text + picture: Logic 66/306 · Math 17/52 · Both 83/358

Status: Withdrawn for a check: Math text-only scored 3/52, below chance, possibly an answer-format problem. Turkish is not listed on its model card.

LFM3-VL-2B on · chatA text only: Logic 11/306 · Math 10/52 · Both 21/358B picture alone: Logic 58/306 · Math 20/52 · Both 78/358C text + picture: Logic 99/306 · Math 34/52 · Both 133/358

Status: Smoke test failed (6 of 25 answers cut off at 8,192 tokens with thinking on); full runs made anyway at the same settings (Ümit #8797/#8799, zeki #8830), unreadable answers counted wrong; kept here, off the Vision page, until Ümit says otherwise.

Taken off the Vision page until they are checked; their runs stay stored. The Vision page ranks the other setups without them.