Decision model

Inception Mercury Decide

via OpenRouter · 1 question per request

58.3% of the 2,198 questions right (56.2% to 60.3%, 95% interval). Back to the leaderboard.

How we called it

Kind

Decision model

Access

OpenRouter System One API

Paid as

Provider service

Model ID

inception/mercury-decide

Variant

via OpenRouter

Note

Inception's decision model, called through OpenRouter's System One route (paid listing). It returns a probability for each option.

Confidence from

Option score

Run

204, 17 October 2026

Settings

Reasoning

off

Request format

1 question per request

Requests at once

1

Temperature

not recorded

Harness seed

0

Harness commit

95cbecfbd428

Prompt version

254597b8be16

Results

Comparison group: wxyzA. Compare with another setup.

Confidence

Answers kept at 99% or more

451, of which 13 wrong (share right: 95.1% to 98.3%, 95% interval); 1,747 skipped, the model less than 99% sure

Calibration gap

15.5 points

AUROC, all questions

0.81 0.79 to 0.82

AUROC, reading

0.91 0.88 to 0.93

AUROC, grammar

0.72 0.69 to 0.74

Brier score

0.208

Accuracy of its surest answers

most confident fifth: 97.3% right; most confident half: 82.0% right; most confident four fifths: 65.4% right

Typical score of a right answer

96.6%

Typical score of a wrong answer

55.7%

Repeats

Run 5 times on all questions.

Answers changed in 5 runs

191 of 2,198

Same wrong answer in all 5

778 of 2,198

Wrong in at least one run

969 of 2,198

Accuracy in each run

58.3%, 59.1%, 58.6%, 58.9%, 58.8%

Latency

Latency, median request

0.313 s (this run)

Latency, 90th percentile

0.772 s

Requests at once

1

Throughput

140.9 questions per minute of request time

Request time

15.6 minutes, over the requests whose answers were kept

Latency: the time from sending one request to receiving its answer, measured from our computer, network and any retries within the request included. Request time: the time at least one request was in flight, over the requests whose answers were kept. It is not the run's elapsed time: it leaves out requests that failed and any pause before a run was resumed.

Cost

$ per 1,000 questions

$0.008

Whole run

$0.017

Basis

Estimated from published prices and recorded token usage

Price

$0.02 input, output free per 1M tokens (source, 2026-10-17)

Input tokens per question

386

Output tokens per question

1

Going first in front of an LLM

Replay of recorded answers, cutoff chosen on half the questions, averaged over the splits. Savings are estimates from API prices; the Method page explains how.