Chat model
MiniCPM5-2B reasoning off · chat
reasoning off · 1 question per request
- Accuracy, all 2,198 questions
- 5.6%4.7% to 6.6%, 95% interval; 123 right
- Answers kept at 99% confidence
- Not measureda plain answer line, no confidence
- Cost, $ per 1,000 questionsper 1M questions
- $5.464$5,464Basis and price under Cost
- Latency
- 172.0 smedian request, 1 question per request
How we called it
- Kind
- Chat model
- Access
- our GPU (logic's server, vLLM v0.30.0 BF16, revision openbmb/MiniCPM5-2B@42433e9), one question per request, plain answer line CEVAP: X (no answer schema); sampling seed 0, temperature 1.0, top_p 0.95, thinking off; output cap 32,768
- Paid as
- Our own hardware
- Model ID
minicpm5-2b-nothink- Parameters
- 2.52B dense, as the maker states it.language model with its LM head, text only; source: counted from the checkpoint's safetensors, openbmb/MiniCPM5-2B@42433e9, huggingface.co/openbmb/MiniCPM5-2B; read 12 November 2026
- Hardware
- 1 × RTX PRO 6000 Blackwell (96 GB); BF16
- Confidence from
- Not measureda plain answer line, no confidence
- Run
- 1299, 15 November 2026
Settings
- Reasoning
- off
- Request format
- 1 question per request
- Requests at once
- 32
- Temperature
- not recorded
Run record: harness seed, harness commit and prompt version
- Harness seed
- 0
- Harness commit
88e48d7a1055- Prompt version
ba455bb00e5c
Results
| Questions | Accuracy | 95% interval | Right |
|---|---|---|---|
| All questions | 5.6% | 4.7% to 6.6% | 123 of 2,198 |
| Reading | 6.6% | 4.9% to 8.7% | 43 of 656 |
| Grammar | 5.2% | 4.2% to 6.4% | 80 of 1,542 |
Comparison group: Σ. Compare with another setup.
Invalid answers: 1,654; read from broken replies: 0; empty replies asked again: 1 (0 came back with answers, 0 answers). Still invalid: 674 cut off at the output limit, 980 unreadable. An empty reply is asked once more; in a broken reply, any letter that is clearly readable is scored and the rest count as wrong. Invalid answers count as wrong (how).
With “Bilmiyorum” (run 1304): Invalid answers: 1,644; read from broken replies: 0; empty replies asked again: 5 (0 came back with answers, 0 answers). Still invalid: 3 still empty when asked again, 687 cut off at the output limit, 954 unreadable. (how)
Accuracy by topic
| Topic | Part | Accuracy | Right |
|---|---|---|---|
| Word and phrase meanings | Reading | 3.8% | 5 of 131 |
| Sentence meanings | Reading | 8.4% | 13 of 154 |
| How a writer explains and develops ideas | Reading | 4.3% | 3 of 70 |
| Topic and main point of a paragraph | Reading | 15.3% | 15 of 98 |
| How a paragraph fits together | Reading | 0.9% | 1 of 106 |
| Details that support a paragraph’s main point | Reading | 6.2% | 6 of 97 |
| Word types, such as nouns and verbs | Grammar | 3.6% | 5 of 140 |
| Groups of words built around a noun | Grammar | 3.1% | 2 of 65 |
| Verbs and their forms | Grammar | 7.7% | 7 of 91 |
| Verb forms used like nouns or modifiers | Grammar | 2.8% | 2 of 72 |
| Word endings and what they do | Grammar | 5.1% | 4 of 79 |
| How words are built | Grammar | 4.3% | 3 of 69 |
| The roles of words and phrases in a sentence | Grammar | 7.6% | 11 of 145 |
| How verbs show who acts and who receives the action | Grammar | 5.4% | 4 of 74 |
| Types of sentences | Grammar | 5.9% | 5 of 85 |
| Questions combining several grammar topics | Grammar | 3.2% | 7 of 220 |
| Sound patterns and changes | Grammar | 0.0% | 0 of 77 |
| Spelling | Grammar | 6.8% | 9 of 133 |
| Punctuation | Grammar | 9.4% | 11 of 117 |
| Problems in sentence wording | Grammar | 5.7% | 10 of 175 |
English
The Text tab's 547 kept reading questions translated into English, asked with this setup's Text tab settings. Every setup: English reading; against Turkish: English and Turkish.
- Right answers
- 370 of 547, first English run (1094); historical paired baseline, run 1094 with Turkish
- Accuracy
- 67.8%, the mean of 3 seeds; 66.7%–68.9% min–max of 3 seeds
- Seed runs
- seed 0: run 1094, 370 of 547 right; seed 1: run 1300, 377 of 547 right; seed 2: run 1301, 365 of 547 right. Seeds 1 and 2 on logic's load of 15 November (vLLM v0.30.0, max-num-seqs 16), which logic gives as run 1094's setup (#9335); the server's settings are not recorded with a run.
- Non-hallucination with “I don't know”
- 16.8%: 33 “I don't know” of 197 not right (run 1095)
- Answered with “I don't know”
- 89.2%: 488 of 547
- Error rate with “I don't know”
- 33.6%: 164 of 488 answers
- Cost, $ per 1,000 questionsper 1M questions, first English run
- 0.429429from GPU time
- Latency, median request, first English run
- 7.29 s; 32 requests at once
- English run
- 1094
- Machine, English runs
- RTX PRO 6000
- Date
- 13 November 2026
- Request format
- 1 question per request
- Reasoning
- off
- Prompt version
ad6f4cc346df- Translations, SHA-256
bf0d47cd385a93050c65906977e199ef7e45d5848dbbdef2bc8d124f4d5abf68
With “I don't know” (run 1095): Invalid answers: 26; read from broken replies: 0; empty replies asked again: 0. Still invalid: 3 cut off at the output limit, 23 unreadable. (how)
Confidence
Not measured: this setup was asked for a plain answer line, so it states no confidence and reports no option probabilities. It has no answers kept at a confidence threshold, no calibration gap, ranking score (AUROC) or Brier score, and no typical score of a right or wrong answer.
Repeats
We did not repeat this setup.
Latency
- Latency, median request
- 172.0 s (this run)
- Latency, 90th percentile
- 546.9 s
- Requests at once
- 32
- Throughput
- 7.6 questions per minute of request time
- Request time
- 289.4 minutes, over the requests whose answers were kept
Latency: the time from sending one request to receiving its answer, measured from our computer, network and any retries within the request included. Request time: the time at least one request was in flight, over the requests whose answers were kept. It is not the run's elapsed time: it leaves out requests that failed and any pause before a run was resumed.
Cost
- Cost per 1,000 questionsper 1M questions
- $5.464$5,464
- Whole run
- $12.01
- Basis
- Estimated from GPU time: the run's request time x the GPUs its server held x Runpod's on-demand price for an RTX PRO 6000, $2.49 per GPU-hour (read 5 November 2026). A cloud-equivalent estimate, not measured spending.
- Price
- $2.49 per GPU-hour, 1 GPU, for the run's request time (source, 2026-11-05)
- Input tokens per question
- 384
- Output tokens per question
- 10,624