Chat model
Gemma 4 E4B default reasoning · chat
default reasoning · 1 question per request
- Accuracy, all 2,198 questions
- 57.6%55.5% to 59.6%, 95% interval; 1,265 right
- Answers kept at 99% confidence
- Not measureda plain answer line, no confidence
- Cost, $ per 1,000 questionsper 1M questions
- $0.307$307Basis and price under Cost
- Latency
- 13.4 smedian request, 1 question per request
How we called it
- Kind
- Chat model
- Access
- our GPU (logic's server, vLLM v0.30.0 BF16, revision google/gemma-4-E4B-it@4ecc2ba), one question per request, plain answer line CEVAP: X (no answer schema); sampling temperature 0.0; output cap 8,192
- Paid as
- Our own hardware
- Model ID
gemma-4-e4b-it- Parameters
- 8B dense · 4.5B effective, as the maker states it (counted from the released files: 8.00B).language model with per-layer embeddings + vision and audio encoders; effective size as Google states it; source: counted from the checkpoint's safetensors, google/gemma-4-E4B-it@4ecc2ba, huggingface.co/google/gemma-4-E4B-it; read 9 November 2026
- Hardware
- 1 × RTX PRO 6000 Blackwell (96 GB); BF16
- Confidence from
- Not measureda plain answer line, no confidence
- Run
- 1040, 12 November 2026
Settings
- Reasoning
- default
- Request format
- 1 question per request
- Requests at once
- 32
- Temperature
- not recorded
Run record: harness seed, harness commit and prompt version
- Harness seed
- not recorded
- Harness commit
6ad0eac51a4a- Prompt version
ba455bb00e5c
Results
| Questions | Accuracy | 95% interval | Right |
|---|---|---|---|
| All questions | 57.6% | 55.5% to 59.6% | 1,265 of 2,198 |
| Reading | 77.6% | 74.2% to 80.6% | 509 of 656 |
| Grammar | 49.0% | 46.5% to 51.5% | 756 of 1,542 |
Comparison group: QRSTUVWXY. Compare with another setup.
Invalid answers: 3; read from broken replies: 0; empty replies asked again: 0. Still invalid: 3 unreadable. An empty reply is asked once more; in a broken reply, any letter that is clearly readable is scored and the rest count as wrong. Invalid answers count as wrong (how).
Accuracy by topic
Vision (preliminary)
Preliminary: these numbers are not final and may change. This setup is condition A of the Vision tab, which asks the 763 questions with underlined or numbered words as text (A), as a page image (B), as both (C) and through OCR models (D); the share answered right, its 95% interval in the number's tooltip.
| Model | A · text onlyaccuracy, higher is better | B · image onlyaccuracy, higher is better | C · text plus imageaccuracy, higher is better | Best OCR routeaccuracy, higher is better |
|---|---|---|---|---|
| Gemma 4 E4B default reasoning · chat Open · 8.0B | 54.4% | 42.9% | 54.1% | Queued2 of 4 OCR routes done; 173 of 763 asked via Surya OCR 2 |
Calibration on these questions (ECE, Brier score, AUROC): not measured, a plain answer line gives no option probabilities, so accuracy only.
English
The Text tab's 547 kept reading questions translated into English, asked with this setup's Text tab settings. Every setup: English reading; against Turkish: English and Turkish.
- Right answers
- 451 of 547, first English run (889)
- Accuracy
- 82.4%
- Non-hallucination with “I don't know”
- 9.8%: 10 “I don't know” of 102 not right (run 890)
- Answered with “I don't know”
- 97.8%: 535 of 547
- Error rate with “I don't know”
- 17.2%: 92 of 535 answers
- Cost, $ per 1,000 questionsper 1M questions, first English run
- 0.199199from GPU time
- Latency, median request, first English run
- 8.39 s; 32 requests at once
- English run
- 889
- Machine, English runs
- RTX PRO 6000
- Date
- 9 November 2026
- Request format
- 1 question per request
- Reasoning
- default
- Prompt version
ad6f4cc346df- Translations, SHA-256
bf0d47cd385a93050c65906977e199ef7e45d5848dbbdef2bc8d124f4d5abf68
With “I don't know” (run 890): Invalid answers: 2; read from broken replies: 0; empty replies asked again: 0. Still invalid: 2 unreadable. (how)
Confidence
Not measured: this setup was asked for a plain answer line, so it states no confidence and reports no option probabilities. It has no answers kept at a confidence threshold, no calibration gap, ranking score (AUROC) or Brier score, and no typical score of a right or wrong answer.
Repeats
We did not repeat this setup.
Latency
- Latency, median request
- 13.4 s (this run)
- Latency, 90th percentile
- 20.4 s
- Requests at once
- 32
- Throughput
- 135.0 questions per minute of request time
- Request time
- 16.3 minutes, over the requests whose answers were kept
Latency: the time from sending one request to receiving its answer, measured from our computer, network and any retries within the request included. Request time: the time at least one request was in flight, over the requests whose answers were kept. It is not the run's elapsed time: it leaves out requests that failed and any pause before a run was resumed.
Cost
- Cost per 1,000 questionsper 1M questions
- $0.307$307
- Whole run
- $0.68
- Basis
- Estimated from GPU time: the run's request time x the GPUs its server held x Runpod's on-demand price for an RTX PRO 6000, $2.49 per GPU-hour (read 5 November 2026). A cloud-equivalent estimate, not measured spending.
- Price
- $2.49 per GPU-hour, 1 GPU, for the run's request time (source, 2026-11-05)
- Input tokens per question
- 248
- Output tokens per question
- 811