Decision model

Elephant-1

1 question per request

52.4% of the 2,198 questions right (50.3% to 54.4%, 95% interval). Back to the leaderboard.

How we called it

Kind

Decision model

Access

Base Labs server (qev, the model card's loader), self-hosted, BF16

Paid as

Our own hardware

Model ID

baselabs/elephant-1

Parameters

9.4B

Hardware

1 × RTX PRO 6000 Blackwell (96 GB); BF16

Note

Base Labs' released elephant-1 weights (baselabs/elephant-1 at b3f8a93), a decision model post-trained from Qwen3.5-9B that also reads images. It ran on our hardware through its model card's loader (qev's DecisionEngine from OneJev at 81ce62f with the checkpoint's serve.patch, as the card's setup.sh installs it) behind our HTTP server (servers/baselabs_server.py), one request at a time. That runtime reads images only through an engine the patch leaves without the attention the model was trained with, so it is not on the Page images tab. The card mentions a trained "I don't know" option, but its answers have no such output, and our questions offer five options. Probabilities rounded to 6 decimals by the model's own loader. They sum to 1 only within that rounding, and the check before its runs allowed for it. Its card lists English only.

Confidence from

Option score

Run

373, 1 November 2026

Settings

Reasoning

off

Request format

1 question per request

Requests at once

1

Temperature

not recorded

Harness seed

0

Harness commit

12a2ea11acaf

Prompt version

254597b8be16

Results

Comparison group: C. Compare with another setup.

Confidence

Answers kept at 99% or more

155, of which 0 wrong (share right: 97.6% to 100.0%, 95% interval); 2,043 skipped, the model less than 99% sure

Calibration gap

11.8 points

AUROC, all questions

0.79 0.77 to 0.81

AUROC, reading

0.86 0.83 to 0.89

AUROC, grammar

0.69 0.67 to 0.72

Brier score

0.200

Accuracy of its surest answers

most confident fifth: 95.2% right; most confident half: 72.9% right; most confident four fifths: 59.5% right

Typical score of a right answer

83.6%

Typical score of a wrong answer

47.0%

Repeats

Run 5 times on all questions.

Answers changed in 5 runs

0 of 2,198

Same wrong answer in all 5

1,047 of 2,198

Wrong in at least one run

1,047 of 2,198

Accuracy in each run

52.4%, 52.4%, 52.4%, 52.4%, 52.4%

Latency

Latency, median request

0.074 s (this run)

Latency, 90th percentile

0.089 s

Requests at once

1

Throughput

783.6 questions per minute of request time

Request time

2.8 minutes, over the requests whose answers were kept

Latency: the time from sending one request to receiving its answer, measured from our computer, network and any retries within the request included. Request time: the time at least one request was in flight, over the requests whose answers were kept. It is not the run's elapsed time: it leaves out requests that failed and any pause before a run was resumed.

Cost

$ per 1,000 questions

$0.014

Whole run

$0.030

Basis

Estimated at Baseten's list price for the same model; our hardware cost is not measured.

Price

$0.035 input, output free per 1M tokens (source, 2026-11-01)

Input tokens per question

396

Output tokens per question

1

Going first in front of an LLM

Replay of recorded answers, cutoff chosen on half the questions, averaged over the splits. Savings are estimates from API prices; the Method page explains how.

§ Counts the first model at its maker’s list price for the same model; our hardware cost is not measured.