Hilde · Decision models · answer key v2

Reading · topic 2

Sentence meanings

154 scored questions. Accuracy by topic compares this topic with the others.

Every setup's result

Accuracy on this topic's questions, with its 95% interval. Few questions make wide intervals.

SetupAccuracyRight
Gemini 4 Argonlow reasoning · 16 questions per request · Chat model100.0%97.6% to 100.0%154 of 154
Gemini 3.8 Flashhigh reasoning · 16 questions per request · Chat model99.4%96.4% to 99.9%153 of 154
GPT-6 Astralow reasoning · 16 questions per request · Chat model99.4%96.4% to 99.9%153 of 154
GPT-6 Astrahigh reasoning · 16 questions per request · Chat model98.7%95.4% to 99.6%152 of 154
GPT-6 Astralow reasoning · 1 question per request · Chat model98.7%95.4% to 99.6%152 of 154
Gemini 3.8 Flashlow reasoning · 1 question per request · Chat model98.7%95.4% to 99.6%152 of 154
GPT-6.1 Terrahigh reasoning · 16 questions per request · Chat model98.7%95.4% to 99.6%152 of 154
Grok 4.7high reasoning · 16 questions per request · Chat model98.7%95.4% to 99.6%152 of 154
Kimi K3.1 Prolow reasoning · 16 questions per request · Chat model98.7%95.4% to 99.6%152 of 154
GPT-6.1 Solhigh reasoning · 16 questions per request · Chat model98.1%94.4% to 99.3%151 of 154
GPT-6.1 Sollow reasoning · 16 questions per request · Chat model98.1%94.4% to 99.3%151 of 154
Claude Opus 5.5high reasoning · 16 questions per request · Chat model98.1%94.4% to 99.3%151 of 154
GPT-6.1 Sollow reasoning · 1 question per request · Chat model97.4%93.5% to 99.0%150 of 154
Claude Fable 5.1high reasoning · 16 questions per request · Chat model97.4%93.5% to 99.0%150 of 154
DeepSeek-V4.1-Flash, first releasereasoning on · 16 questions per request · Chat model97.4%93.5% to 99.0%150 of 154
GPT-6.1 Terralow reasoning · 16 questions per request · Chat model97.4%93.5% to 99.0%150 of 154
Claude Opus 5.5low reasoning · 1 question per request · Chat model96.8%92.6% to 98.6%149 of 154
Gemini 3.8 Flashlow reasoning · 16 questions per request · Chat model96.8%92.6% to 98.6%149 of 154
Claude Opus 5.5low reasoning · 16 questions per request · Chat model96.8%92.6% to 98.6%149 of 154
Claude Sonnet 5.5high reasoning · 1 question per request · Chat model96.1%91.8% to 98.2%148 of 154
Liquid d1-ultra1 question per request · Decision model96.1%91.8% to 98.2%148 of 154
Qwen3.8-27Blow reasoning · 16 questions per request · Chat model96.1%91.8% to 98.2%148 of 154
Kimi K3.1 Proreasoning none · 16 questions per request · Chat model96.1%91.8% to 98.2%148 of 154
SubQ Snap Preview1 question per request · Decision model94.8%90.1% to 97.3%146 of 154
GLM-5.3low reasoning · 16 questions per request · Chat model93.5%88.5% to 96.4%144 of 154
Gemma 4 31Breasoning off · 16 questions per request · Chat model93.5%88.5% to 96.4%144 of 154
Fastino GLiDE v1.2local · 1 question per request · Decision model93.5%88.5% to 96.4%144 of 154
GPT-6 Lunalow reasoning · 16 questions per request · Chat model93.5%88.5% to 96.4%144 of 154
Fastino GLiDE1 question per request · Decision model93.5%88.5% to 96.4%144 of 154
Drex Max 1.11 question per request · Decision model93.5%88.5% to 96.4%144 of 154
Perplexity Decider 27B v1.2local · 1 question per request · Decision model93.5%88.5% to 96.4%144 of 154
vLLM-SR d3-ultratransformers, rev 52ba9e7 · 1 question per request · Decision model92.9%87.7% to 96.0%143 of 154
Drex Max v11 question per request · Decision model92.9%87.7% to 96.0%143 of 154
Perplexity Decider 27B v1.11 question per request · Decision model92.9%87.7% to 96.0%143 of 154
GLM-5.3-Flashreasoning on · 16 questions per request · Chat model92.2%86.9% to 95.5%142 of 154
vLLM-SR d31 question per request · Decision model92.2%86.9% to 95.5%142 of 154
Mammoth-11 question per request · Decision model92.2%86.9% to 95.5%142 of 154
SubQ Snap 1.11 question per request · Decision model92.2%86.9% to 95.5%142 of 154
Clef1 question per request · Decision model92.2%86.9% to 95.5%142 of 154
Gemma 4 31Breasoning off · 1 question per request · Chat model91.6%86.1% to 95.0%141 of 154
Liquid d1.11 question per request · Decision model91.6%86.1% to 95.0%141 of 154
DeepSeek-V4.1-Flash, first releasereasoning off · 16 questions per request · Chat model90.9%85.3% to 94.5%140 of 154
Celeris-1 Decision1 question per request · Decision model90.9%85.3% to 94.5%140 of 154
Cygnet1 question per request · Decision model90.9%85.3% to 94.5%140 of 154
Claude Sonnet 5.5high reasoning · 16 questions per request · Chat model89.6%83.8% to 93.5%138 of 154
Jev 1.14.01 question per request · Decision model89.6%83.8% to 93.5%138 of 154
Winnow-12B1 question per request · Decision model89.6%83.8% to 93.5%138 of 154
Perplexity Decider 27B v11 question per request · Decision model89.0%83.0% to 93.0%137 of 154
Inception Mercury Decidevia OpenRouter · 1 question per request · Decision model89.0%83.0% to 93.0%137 of 154
Jev 1.13.01 question per request · Decision model89.0%83.0% to 93.0%137 of 154
GPT-6 LunaDecisions API, beta · 1 question per request · Decision model88.3%82.3% to 92.5%136 of 154
Qwen3.8-27Byes/no scoring · 1 question per request · Chat model87.7%81.5% to 92.0%135 of 154
Elephant-11 question per request · Decision model87.0%80.8% to 91.4%134 of 154
Claude Sonnet 5.5low reasoning · 16 questions per request · Chat model86.4%80.1% to 90.9%133 of 154
OneJev 27B1 question per request · Decision model86.4%80.1% to 90.9%133 of 154
Liquid d11 question per request · Decision model86.4%80.1% to 90.9%133 of 154
Qwen3.8-27Bvote share · reasoning off · 1 question per request · Chat model86.4%80.1% to 90.9%133 of 154
Qwen3.8-27Bstated confidence · reasoning off · 1 question per request · Chat model86.4%80.1% to 90.9%133 of 154
Clef-flash1 question per request · Decision model85.7%79.3% to 90.4%132 of 154
Open-Jev 27B v1.11 question per request · Decision model85.1%78.6% to 89.8%131 of 154
GLM-5.3-Flashreasoning off · 16 questions per request · Chat model84.4%77.9% to 89.3%130 of 154
Drex 2.01 question per request · Decision model84.4%77.9% to 89.3%130 of 154
Perplexity Decider Flash v1.2local · 1 question per request · Decision model79.9%72.8% to 85.4%123 of 154
Microsoft-Decision-1via OpenRouter · 1 question per request · Decision model78.6%71.4% to 84.3%121 of 154
Qwen3-14Breasoning off · 16 questions per request · Chat model74.7%67.3% to 80.9%115 of 154
Erk-14Breasoning off · 16 questions per request · Chat model74.0%66.6% to 80.3%114 of 154
Metask-Jev 4B1 question per request · Decision model68.8%61.1% to 75.6%106 of 154
Strands Decider 2B1 question per request · Decision model46.1%38.4% to 54.0%71 of 154
Liquid d1-2B (open weights)1 question per request · Decision model43.5%35.9% to 51.4%67 of 154
meraGPT Decider 11 question per request · Decision model29.9%23.2% to 37.5%46 of 154
GLiNER2.5-multi-Decide1 question per request · Decision model23.4%17.4% to 30.7%36 of 154
Laya1 question per request · Decision model22.7%16.8% to 30.0%35 of 154
GLiNER2.5 Multi (base)1 question per request · Chat model13.0%8.6% to 19.2%20 of 154
Laya multilingual1 question per request · Decision model12.3%8.0% to 18.5%19 of 154

Its questions

IDPageNumberAnswer keySetups rightDecision models right
134333E71 of 7435 of 37
135331B66 of 7431 of 37
136332E67 of 7431 of 37
137334A67 of 7433 of 37
143351C65 of 7430 of 37
144354B66 of 7430 of 37
145352E71 of 7435 of 37
146355E68 of 7432 of 37
147353E67 of 7432 of 37
148366D64 of 7430 of 37
149368E70 of 7434 of 37
150367E70 of 7434 of 37
151369A70 of 7433 of 37
157388C61 of 7426 of 37
158386D56 of 7423 of 37
159389D67 of 7431 of 37
160387D55 of 7427 of 37
161391D64 of 7429 of 37
162393C69 of 7433 of 37
163392E19 of 743 of 37
164394C52 of 7418 of 37
165405D68 of 7432 of 37
166408A44 of 7420 of 37
167406D66 of 7432 of 37
168409B67 of 7431 of 37
1694010C70 of 7434 of 37
170407B67 of 7431 of 37
171411B70 of 7434 of 37
172414E69 of 7433 of 37
173412B69 of 7433 of 37
174415E69 of 7433 of 37
175413C70 of 7434 of 37
176416E70 of 7434 of 37
177427C38 of 7410 of 37
1784210D69 of 7433 of 37
1794211C65 of 7431 of 37
180428B69 of 7433 of 37
1814212D59 of 7428 of 37
182429E70 of 7434 of 37
195451A68 of 7432 of 37
196454E69 of 7433 of 37
197455C68 of 7433 of 37
198452A69 of 7433 of 37
199453E69 of 7433 of 37
200456D69 of 7433 of 37
201467C66 of 7429 of 37
2024610E67 of 7431 of 37
203468A67 of 7431 of 37
2044611A48 of 7416 of 37
205469B69 of 7433 of 37
210485C21 of 741 of 37
211487D54 of 7423 of 37
212486D62 of 7432 of 37
213488E58 of 7425 of 37
214491E68 of 7432 of 37
215494E70 of 7434 of 37
216492B59 of 7427 of 37
217495A59 of 7426 of 37
218493D68 of 7432 of 37
219506C64 of 7431 of 37
220508E70 of 7434 of 37
221507D67 of 7431 of 37
222509A71 of 7434 of 37
232531B71 of 7434 of 37
233534C61 of 7426 of 37
234535D70 of 7434 of 37
235532D56 of 7428 of 37
236533C70 of 7434 of 37
237536D58 of 7425 of 37
238547E70 of 7434 of 37
239549B62 of 7428 of 37
240548C70 of 7433 of 37
2415410B68 of 7432 of 37
242551D40 of 7417 of 37
243553D56 of 7425 of 37
244554C71 of 7434 of 37
245552E57 of 7428 of 37
246565E69 of 7433 of 37
247567D70 of 7434 of 37
248566B19 of 747 of 37
253585E68 of 7432 of 37
254587E24 of 7414 of 37
255586A72 of 7435 of 37
256588B52 of 7422 of 37
257591C63 of 7431 of 37
258593D36 of 7413 of 37
259592C70 of 7434 of 37
260594C67 of 7431 of 37
261605B67 of 7432 of 37
262607E67 of 7432 of 37
263606B70 of 7434 of 37
264608E68 of 7433 of 37
265611A57 of 7427 of 37
266613D70 of 7433 of 37
267612D66 of 7431 of 37
268614D70 of 7434 of 37
269625E70 of 7434 of 37
270627D67 of 7432 of 37
271626D68 of 7432 of 37
272628E69 of 7434 of 37
273631C66 of 7431 of 37
274633A72 of 7435 of 37
275634D66 of 7430 of 37
276632B68 of 7432 of 37
277647D67 of 7431 of 37
278645B57 of 7423 of 37
279646B69 of 7433 of 37
280648C64 of 7428 of 37
281651A66 of 7431 of 37
282653C68 of 7433 of 37
283652C69 of 7433 of 37
284654E69 of 7433 of 37
285665A60 of 7428 of 37
286667D67 of 7431 of 37
287666A62 of 7429 of 37
288668C69 of 7433 of 37
1606511A68 of 7433 of 37
1607513E59 of 7426 of 37
1608514C69 of 7433 of 37
1609512B66 of 7430 of 37
3149345A64 of 7432 of 37
3150348C59 of 7426 of 37
3151346C65 of 7430 of 37
3152349D62 of 7429 of 37
3153347C52 of 7420 of 37
3154374A70 of 7433 of 37
3155371E71 of 7435 of 37
3156372E70 of 7434 of 37
3157375E66 of 7430 of 37
3158373C57 of 7427 of 37
3159434B68 of 7432 of 37
3160431C68 of 7432 of 37
3161432B65 of 7431 of 37
3162435E71 of 7435 of 37
3163436D68 of 7432 of 37
3164433A45 of 7412 of 37
31654410C62 of 7428 of 37
3166447B65 of 7429 of 37
31674411E67 of 7431 of 37
3168448E69 of 7433 of 37
3169449C70 of 7433 of 37
31704412D69 of 7433 of 37
3171471E52 of 7420 of 37
3172473B69 of 7433 of 37
3173472D28 of 7410 of 37
3174474E59 of 7428 of 37
3175525B69 of 7433 of 37
3176527B70 of 7434 of 37
3177528C70 of 7434 of 37
3178526E47 of 7417 of 37
3179571B62 of 7430 of 37
3180573C70 of 7434 of 37
3181572B69 of 7433 of 37
3182574A28 of 744 of 37