Decision model

Laya multilingual

1 question per request

17.9% of the 2,198 questions right (16.3% to 19.5%, 95% interval). Back to the leaderboard.

How we called it

Kind: Decision model
Access: Laya server, self-hosted
Paid as: Our own hardware
Model ID: laya-multilingual
Parameters: 322M
Hardware: 1 × RTX PRO 6000 Blackwell (96 GB)
Note: Encoder with a decision head
Confidence from: Option score
Run: 46, 4 October 2026

Settings

Reasoning: off
Request format: 1 question per request
Requests at once: 1
Temperature: not recorded
Harness seed: 0
Harness commit: not recorded
Prompt version: not recorded

Results

Questions	Accuracy	95% interval	Right
All questions	17.9%	16.3% to 19.5%	393 of 2,198
Reading	17.1%	14.4% to 20.1%	112 of 656
Grammar	18.2%	16.4% to 20.2%	281 of 1,542

Comparison group: H. Compare with another setup.

Accuracy by topic

Topic	Part	Accuracy	Right
Word and phrase meanings	Reading	19.1%	25 of 131
Sentence meanings	Reading	12.3%	19 of 154
How a writer explains and develops ideas	Reading	15.7%	11 of 70
Topic and main point of a paragraph	Reading	21.4%	21 of 98
How a paragraph fits together	Reading	18.9%	20 of 106
Details that support a paragraph’s main point	Reading	16.5%	16 of 97
Word types, such as nouns and verbs	Grammar	18.6%	26 of 140
Groups of words built around a noun	Grammar	12.3%	8 of 65
Verbs and their forms	Grammar	15.4%	14 of 91
Verb forms used like nouns or modifiers	Grammar	16.7%	12 of 72
Word endings and what they do	Grammar	20.3%	16 of 79
How words are built	Grammar	15.9%	11 of 69
The roles of words and phrases in a sentence	Grammar	16.6%	24 of 145
How verbs show who acts and who receives the action	Grammar	14.9%	11 of 74
Types of sentences	Grammar	20.0%	17 of 85
Questions combining several grammar topics	Grammar	19.1%	42 of 220
Sound patterns and changes	Grammar	16.9%	13 of 77
Spelling	Grammar	21.8%	29 of 133
Punctuation	Grammar	17.1%	20 of 117
Problems in sentence wording	Grammar	21.7%	38 of 175

Confidence

Answers kept at 99% or more: 1, of which 1 wrong (share right: 0.0% to 79.3%, 95% interval); 2,197 skipped, the model less than 99% sure
Calibration gap: 27.7 points
AUROC, all questions: 0.45 0.42 to 0.49
AUROC, reading: 0.43 0.37 to 0.49
AUROC, grammar: 0.47 0.43 to 0.51
Brier score: 0.263
Accuracy of its surest answers: most confident fifth: 15.0% right; most confident half: 16.1% right; most confident four fifths: 16.5% right
Typical score of a right answer: 37.9%
Typical score of a wrong answer: 40.4%

Repeats

Run 5 times on all questions.

Answers changed in 5 runs: 0 of 2,198
Same wrong answer in all 5: 1,805 of 2,198
Wrong in at least one run: 1,805 of 2,198
Accuracy in each run: 17.9%, 17.9%, 17.9%, 17.9%, 17.9%

Latency

Latency, median request: 0.009 s (its repeated runs)
Latency, 90th percentile: 0.010 s
Requests at once: 1
Throughput: 5364.4 questions per minute of request time
Request time: 0.4 minutes, over the requests whose answers were kept

Latency: the time from sending one request to receiving its answer, measured from our computer, network and any retries within the request included. Request time: the time at least one request was in flight, over the requests whose answers were kept. It is not the run's elapsed time: it leaves out requests that failed and any pause before a run was resumed.

Cost

$ per 1,000 questions: Not measured
Basis: No dollar estimate; see hardware and time
Input tokens per question: 306
Output tokens per question: 0