Decision model
Liquid d1-2B (open weights)
1 question per request
29.4% of the 2,198 questions right (27.5% to 31.3%, 95% interval). Back to the leaderboard.
How we called it
- Kind
- Decision model
- Access
- d1 server (Liquid's model code in Transformers, the model card's call), self-hosted, BF16
- Paid as
- Our own hardware
- Model ID
LiquidAI/d1-2B- Parameters
- 1.9B
- Hardware
- 1 × RTX PRO 6000 Blackwell (96 GB); BF16
- Note
- Liquid AI's released d1-2B weights (LiquidAI/d1-2B at 10e7671), a decision model built on LFM3-VL-2B that also reads images. It ran on our hardware through its model card's own call (model.system_one, Liquid's model code in Transformers) behind our HTTP server (servers/d1_server.py), one request at a time. Liquid lists 16 languages for it; Turkish is not among them. Its request times come from one GPU with no other known work while its runs took place, though the GPU was not monitored throughout. Only this model's server showed on it at 11:55 UTC on 1 November 2026, before the runs, and nothing at 12:35 UTC, after them; the one other project with a claim on that GPU confirmed it ran nothing there in between.
- Confidence from
- Option score
- Run
- 340, 1 November 2026
Settings
- Reasoning
- off
- Request format
- 1 question per request
- Requests at once
- 1
- Temperature
- not recorded
- Harness seed
- 0
- Harness commit
737862ea6d74- Prompt version
254597b8be16
Results
| Questions | Accuracy | 95% interval | Right |
|---|---|---|---|
| All questions | 29.4% | 27.5% to 31.3% | 646 of 2,198 |
| Reading | 40.9% | 37.2% to 44.7% | 268 of 656 |
| Grammar | 24.5% | 22.4% to 26.7% | 378 of 1,542 |
Comparison group: G. Compare with another setup.
Accuracy by topic
Page images (preliminary)
Preliminary: these numbers are not final and may change. This setup is condition A of the Page images tab, which asks the 763 questions with underlined or numbered words as text (A), as a page image (B), as both (C) and through OCR models (D); the share answered right, with its 95% interval.
| Model | A · text onlyaccuracy, higher is better | B · image onlyaccuracy, higher is better | C · text plus imageaccuracy, higher is better | Best OCR routeaccuracy, higher is better |
|---|---|---|---|---|
| Liquid d1-2B (open weights)1 question per request | 29.2%26.1% to 32.6% | 23.3%20.5% to 26.5% | 26.9%23.8% to 30.1% | Not available: not run |
Confidence
- Answers kept at 99% or more
- 0, of which 0 wrong; 2,198 skipped, the model less than 99% sure
- Calibration gap
- 5.8 points
- AUROC, all questions
- 0.64 0.61 to 0.66
- AUROC, reading
- 0.66 0.62 to 0.70
- AUROC, grammar
- 0.58 0.55 to 0.61
- Brier score
- 0.198
- Accuracy of its surest answers
- most confident fifth: 47.8% right; most confident half: 37.6% right; most confident four fifths: 31.8% right
- Typical score of a right answer
- 34.4%
- Typical score of a wrong answer
- 28.9%
Repeats
Run 5 times on all questions.
- Answers changed in 5 runs
- 0 of 2,198
- Same wrong answer in all 5
- 1,552 of 2,198
- Wrong in at least one run
- 1,552 of 2,198
- Accuracy in each run
- 29.4%, 29.4%, 29.4%, 29.4%, 29.4%
Latency
- Latency, median request
- 0.019 s (this run)
- Latency, 90th percentile
- 0.022 s
- Requests at once
- 1
- Throughput
- 3150.6 questions per minute of request time
- Request time
- 0.7 minutes, over the requests whose answers were kept
Latency: the time from sending one request to receiving its answer, measured from our computer, network and any retries within the request included. Request time: the time at least one request was in flight, over the requests whose answers were kept. It is not the run's elapsed time: it leaves out requests that failed and any pause before a run was resumed.
Cost
- $ per 1,000 questions
- Not measured
- Basis
- No dollar estimate; see hardware and time
- Input tokens per question
- 339
- Output tokens per question
- 0
Going first in front of an LLM
Replay of recorded answers, cutoff chosen on half the questions, averaged over the splits. Savings are estimates from API prices; the Method page explains how.
| LLM behind it | Share taken | Accuracy change | Saving |
|---|---|---|---|
| Gemini 3.8 Flashhigh reasoning · 16 questions per request | 0.8% | −0.11 points | 1%† |
| GPT-6 Astrahigh reasoning · 16 questions per request | 0.8% | −0.11 points | 1%† |
| GPT-6.1 Solhigh reasoning · 16 questions per request | 0.8% | −0.11 points | 1%† |
| GPT-6 Astralow reasoning · 16 questions per request | 0.8% | −0.11 points | 1%† |
| GPT-6 Astralow reasoning · 1 question per request | 0.8% | −0.11 points | 1%† |
| Gemini 3.8 Flashlow reasoning · 1 question per request | 0.8% | −0.11 points | 1%† |
| GPT-6.1 Sollow reasoning · 1 question per request | 0.8% | −0.11 points | 1%† |
| GPT-6.1 Sollow reasoning · 16 questions per request | 0.8% | −0.11 points | 1%† |
| Claude Opus 5.5high reasoning · 16 questions per request | 0.8% | −0.11 points | 1%† |
| GPT-6.1 Terrahigh reasoning · 16 questions per request | 0.8% | −0.11 points | 1%† |
| Claude Opus 5.5low reasoning · 1 question per request | 0.8% | −0.11 points | 1%† |
| Gemini 3.8 Flashlow reasoning · 16 questions per request | 0.8% | −0.11 points | 1%† |
| GPT-6.1 Terralow reasoning · 16 questions per request | 0.8% | −0.11 points | 1%† |
| Claude Opus 5.5low reasoning · 16 questions per request | 0.8% | −0.11 points | 1%† |
| Claude Sonnet 5.5high reasoning · 1 question per request | 0.8% | −0.11 points | 1%† |
| Claude Sonnet 5.5high reasoning · 16 questions per request | 0.8% | −0.11 points | 1%† |
† Leaves out the first model’s hardware cost, which we have not estimated.