Chat model

GLM-5.3

low reasoning · 16 questions per request

74.5% of the 2,198 questions right (72.7% to 76.3%, 95% interval). Back to the leaderboard.

How we called it

Kind: Chat model
Access: OpenRouter chat API, Fireworks endpoint, fallbacks off
Paid as: Provider service
Model ID: z-ai/glm-5.3
Note: The Fireworks pin admits all of Fireworks' endpoints: OpenRouter billed 135 of the 138 requests at the spot endpoint's rates, 2 at the standard rates and 1 at the fast endpoint's rates ($1.82 in all); the cost shown uses the spot rates.
Confidence from: Written estimate
Run: 314, 28 October 2026

Settings

Reasoning: low
Request format: 16 questions per request
Requests at once: 4
Temperature: not recorded
Harness seed: 0
Harness commit: 001ce5b42634
Prompt version: 254597b8be16

Results

Questions	Accuracy	95% interval	Right
All questions	74.5%	72.7% to 76.3%	1,638 of 2,198
Reading	87.7%	84.9% to 90.0%	575 of 656
Grammar	68.9%	66.6% to 71.2%	1,063 of 1,542

Comparison group: no. Compare with another setup.

Accuracy by topic

Topic	Part	Accuracy	Right
Word and phrase meanings	Reading	84.7%	111 of 131
Sentence meanings	Reading	93.5%	144 of 154
How a writer explains and develops ideas	Reading	74.3%	52 of 70
Topic and main point of a paragraph	Reading	95.9%	94 of 98
How a paragraph fits together	Reading	81.1%	86 of 106
Details that support a paragraph’s main point	Reading	90.7%	88 of 97
Word types, such as nouns and verbs	Grammar	77.9%	109 of 140
Groups of words built around a noun	Grammar	64.6%	42 of 65
Verbs and their forms	Grammar	75.8%	69 of 91
Verb forms used like nouns or modifiers	Grammar	90.3%	65 of 72
Word endings and what they do	Grammar	78.5%	62 of 79
How words are built	Grammar	55.1%	38 of 69
The roles of words and phrases in a sentence	Grammar	73.1%	106 of 145
How verbs show who acts and who receives the action	Grammar	70.3%	52 of 74
Types of sentences	Grammar	75.3%	64 of 85
Questions combining several grammar topics	Grammar	62.7%	138 of 220
Sound patterns and changes	Grammar	33.8%	26 of 77
Spelling	Grammar	66.9%	89 of 133
Punctuation	Grammar	62.4%	73 of 117
Problems in sentence wording	Grammar	74.3%	130 of 175

Confidence

Answers kept at 99% or more: 1, of which 0 wrong (share right: 20.7% to 100.0%, 95% interval); 2,197 skipped, the model less than 99% sure
Calibration gap: 2.6 points
AUROC, all questions: 0.74 0.72 to 0.76
AUROC, reading: 0.82 0.76 to 0.87
AUROC, grammar: 0.70 0.67 to 0.73
Brier score: 0.160
Accuracy of its surest answers: most confident fifth: 93.0% right; most confident half: 86.5% right; most confident four fifths: 82.1% right
Typical score of a right answer: 80.0%
Typical score of a wrong answer: 70.0%

Repeats

We did not repeat this setup.

Latency

Latency, median request: 17.1 s (this run)
Latency, 90th percentile: 54.5 s
Requests at once: 4
Throughput: 143.5 questions per minute of request time
Request time: 15.3 minutes, over the requests whose answers were kept

Latency: the time from sending one request to receiving its answer, measured from our computer, network and any retries within the request included. Request time: the time at least one request was in flight, over the requests whose answers were kept. It is not the run's elapsed time: it leaves out requests that failed and any pause before a run was resumed.

Cost

$ per 1,000 questions: $0.81
Whole run: $1.79
Basis: Estimated from published prices and recorded token usage
Price: $0.98 input, $0.182 cached input, $3.08 output per 1M tokens (source, 2026-10-28)
Input tokens per question: 254
Output tokens per question: 187