Decision model
Mammoth-1
1 question per request
67.1% of the 2,198 questions right (65.1% to 69.0%, 95% interval). Back to the leaderboard.
How we called it
- Kind
- Decision model
- Access
- Base Labs server (qev, the model card's loader), self-hosted, BF16
- Paid as
- Our own hardware
- Model ID
baselabs/mammoth-1- Parameters
- 27.4B
- Hardware
- 1 × RTX PRO 6000 Blackwell (96 GB); BF16
- Note
- Base Labs' released mammoth-1 weights (baselabs/mammoth-1 at 61cec44), a decision model post-trained from Qwen3.5-27B that also reads images. It ran on our hardware through its model card's loader (qev's DecisionEngine from OneJev at 81ce62f with the checkpoint's serve.patch, as the card's setup.sh installs it) behind our HTTP server (servers/baselabs_server.py), one request at a time. That runtime reads images only through an engine the patch leaves without the attention the model was trained with, so it is not on the Page images tab. The card mentions a trained "I don't know" option, but its answers have no such output, and our questions offer five options. Probabilities rounded to 6 decimals by the model's own loader. They sum to 1 only within that rounding, and the check before its runs allowed for it. Its card lists English only.
- Confidence from
- Option score
- Run
- 374, 1 November 2026
Settings
- Reasoning
- off
- Request format
- 1 question per request
- Requests at once
- 1
- Temperature
- not recorded
- Harness seed
- 0
- Harness commit
12a2ea11acaf- Prompt version
254597b8be16
Results
| Questions | Accuracy | 95% interval | Right |
|---|---|---|---|
| All questions | 67.1% | 65.1% to 69.0% | 1,475 of 2,198 |
| Reading | 90.4% | 87.9% to 92.4% | 593 of 656 |
| Grammar | 57.2% | 54.7% to 59.6% | 882 of 1,542 |
Comparison group: pqrs. Compare with another setup.
Accuracy by topic
Confidence
- Answers kept at 99% or more
- 285, of which 0 wrong (share right: 98.7% to 100.0%, 95% interval); 1,913 skipped, the model less than 99% sure
- Calibration gap
- 5.6 points
- AUROC, all questions
- 0.83 0.82 to 0.85
- AUROC, reading
- 0.92 0.90 to 0.95
- AUROC, grammar
- 0.76 0.74 to 0.79
- Brier score
- 0.160
- Accuracy of its surest answers
- most confident fifth: 99.3% right; most confident half: 89.5% right; most confident four fifths: 75.8% right
- Typical score of a right answer
- 92.2%
- Typical score of a wrong answer
- 50.6%
Repeats
Run 5 times on all questions.
- Answers changed in 5 runs
- 0 of 2,198
- Same wrong answer in all 5
- 723 of 2,198
- Wrong in at least one run
- 723 of 2,198
- Accuracy in each run
- 67.1%, 67.1%, 67.1%, 67.1%, 67.1%
Latency
- Latency, median request
- 0.169 s (this run)
- Latency, 90th percentile
- 0.191 s
- Requests at once
- 1
- Throughput
- 355.8 questions per minute of request time
- Request time
- 6.2 minutes, over the requests whose answers were kept
Latency: the time from sending one request to receiving its answer, measured from our computer, network and any retries within the request included. Request time: the time at least one request was in flight, over the requests whose answers were kept. It is not the run's elapsed time: it leaves out requests that failed and any pause before a run was resumed.
Cost
- $ per 1,000 questions
- $0.040
- Whole run
- $0.087
- Basis
- Estimated at Baseten's list price for the same model; our hardware cost is not measured.
- Price
- $0.10 input, output free per 1M tokens (source, 2026-11-01)
- Input tokens per question
- 396
- Output tokens per question
- 1
Going first in front of an LLM
Replay of recorded answers, cutoff chosen on half the questions, averaged over the splits. Savings are estimates from API prices; the Method page explains how.
| LLM behind it | Share taken | Accuracy change | Saving |
|---|---|---|---|
| Gemini 3.8 Flashhigh reasoning · 16 questions per request | 22.6% | −0.13 points | 21%§ |
| GPT-6 Astrahigh reasoning · 16 questions per request | 23.9% | −0.15 points | 23%§ |
| GPT-6.1 Solhigh reasoning · 16 questions per request | 25.6% | −0.15 points | 23%§ |
| GPT-6 Astralow reasoning · 16 questions per request | 23.9% | −0.15 points | 23%§ |
| GPT-6 Astralow reasoning · 1 question per request | 25.2% | −0.15 points | 25%§ |
| Gemini 3.8 Flashlow reasoning · 1 question per request | 24.3% | −0.15 points | 20%§ |
| GPT-6.1 Sollow reasoning · 1 question per request | 25.9% | −0.15 points | 25%§ |
| GPT-6.1 Sollow reasoning · 16 questions per request | 23.9% | −0.15 points | 20%§ |
| Claude Opus 5.5high reasoning · 16 questions per request | 24.8% | −0.13 points | 24%§ |
| GPT-6.1 Terrahigh reasoning · 16 questions per request | 25.5% | −0.17 points | 18%§ |
| Claude Opus 5.5low reasoning · 1 question per request | 24.8% | −0.19 points | 24%§ |
| Gemini 3.8 Flashlow reasoning · 16 questions per request | 25.2% | −0.17 points | 10%§ |
| GPT-6.1 Terralow reasoning · 16 questions per request | 29.5% | −0.21 points | 20%§ |
| Claude Opus 5.5low reasoning · 16 questions per request | 29.6% | −0.20 points | 29%§ |
| Claude Sonnet 5.5high reasoning · 1 question per request | 32.3% | −0.17 points | 32%§ |
| Claude Sonnet 5.5high reasoning · 16 questions per request | 47.4% | −0.25 points | 46%§ |
§ Counts the first model at its maker’s list price for the same model; our hardware cost is not measured.