Chat model

MiniCPM5-2B reasoning off · chat

reasoning off · 1 question per request

Accuracy, all 2,198 questions
5.6%4.7% to 6.6%, 95% interval; 123 right
Answers kept at 99% confidence
Not measureda plain answer line, no confidence
Cost, $ per 1,000 questionsper 1M questions
$5.464$5,464Basis and price under Cost
Latency
172.0 smedian request, 1 question per request

Back to the leaderboard.

How we called it

Kind
Chat model
Access
our GPU (logic's server, vLLM v0.30.0 BF16, revision openbmb/MiniCPM5-2B@42433e9), one question per request, plain answer line CEVAP: X (no answer schema); sampling seed 0, temperature 1.0, top_p 0.95, thinking off; output cap 32,768
Paid as
Our own hardware
Model ID
minicpm5-2b-nothink
Parameters
2.52B dense, as the maker states it.language model with its LM head, text only; source: counted from the checkpoint's safetensors, openbmb/MiniCPM5-2B@42433e9, huggingface.co/openbmb/MiniCPM5-2B; read 12 November 2026
Hardware
1 × RTX PRO 6000 Blackwell (96 GB); BF16
Confidence from
Not measureda plain answer line, no confidence
Run
1299, 15 November 2026

Settings

Reasoning
off
Request format
1 question per request
Requests at once
32
Temperature
not recorded
Run record: harness seed, harness commit and prompt version
Harness seed
0
Harness commit
88e48d7a1055
Prompt version
ba455bb00e5c

Results

QuestionsAccuracy95% intervalRight
All questions5.6%4.7% to 6.6%123 of 2,198
Reading6.6%4.9% to 8.7%43 of 656
Grammar5.2%4.2% to 6.4%80 of 1,542

Comparison group: Σ. Compare with another setup.

Invalid answers: 1,654; read from broken replies: 0; empty replies asked again: 1 (0 came back with answers, 0 answers). Still invalid: 674 cut off at the output limit, 980 unreadable. An empty reply is asked once more; in a broken reply, any letter that is clearly readable is scored and the rest count as wrong. Invalid answers count as wrong (how).

With “Bilmiyorum” (run 1304): Invalid answers: 1,644; read from broken replies: 0; empty replies asked again: 5 (0 came back with answers, 0 answers). Still invalid: 3 still empty when asked again, 687 cut off at the output limit, 954 unreadable. (how)

Accuracy by topic

English

The Text tab's 547 kept reading questions translated into English, asked with this setup's Text tab settings. Every setup: English reading; against Turkish: English and Turkish.

Right answers
370 of 547, first English run (1094); historical paired baseline, run 1094 with Turkish
Accuracy
67.8%, the mean of 3 seeds; 66.7%–68.9% min–max of 3 seeds
Seed runs
seed 0: run 1094, 370 of 547 right; seed 1: run 1300, 377 of 547 right; seed 2: run 1301, 365 of 547 right. Seeds 1 and 2 on logic's load of 15 November (vLLM v0.30.0, max-num-seqs 16), which logic gives as run 1094's setup (#9335); the server's settings are not recorded with a run.
Non-hallucination with “I don't know”
16.8%: 33 “I don't know” of 197 not right (run 1095)
Answered with “I don't know”
89.2%: 488 of 547
Error rate with “I don't know”
33.6%: 164 of 488 answers
Cost, $ per 1,000 questionsper 1M questions, first English run
0.429429from GPU time
Latency, median request, first English run
7.29 s; 32 requests at once
English run
1094
Machine, English runs
RTX PRO 6000
Date
13 November 2026
Request format
1 question per request
Reasoning
off
Prompt version
ad6f4cc346df
Translations, SHA-256
bf0d47cd385a93050c65906977e199ef7e45d5848dbbdef2bc8d124f4d5abf68

With “I don't know” (run 1095): Invalid answers: 26; read from broken replies: 0; empty replies asked again: 0. Still invalid: 3 cut off at the output limit, 23 unreadable. (how)

Confidence

Not measured: this setup was asked for a plain answer line, so it states no confidence and reports no option probabilities. It has no answers kept at a confidence threshold, no calibration gap, ranking score (AUROC) or Brier score, and no typical score of a right or wrong answer.

Repeats

We did not repeat this setup.

Latency

Latency, median request
172.0 s (this run)
Latency, 90th percentile
546.9 s
Requests at once
32
Throughput
7.6 questions per minute of request time
Request time
289.4 minutes, over the requests whose answers were kept

Latency: the time from sending one request to receiving its answer, measured from our computer, network and any retries within the request included. Request time: the time at least one request was in flight, over the requests whose answers were kept. It is not the run's elapsed time: it leaves out requests that failed and any pause before a run was resumed.

Cost

Cost per 1,000 questionsper 1M questions
$5.464$5,464
Whole run
$12.01
Basis
Estimated from GPU time: the run's request time x the GPUs its server held x Runpod's on-demand price for an RTX PRO 6000, $2.49 per GPU-hour (read 5 November 2026). A cloud-equivalent estimate, not measured spending.
Price
$2.49 per GPU-hour, 1 GPU, for the run's request time (source, 2026-11-05)
Input tokens per question
384
Output tokens per question
10,624