763 questions with underlined or numbered words

How the benchmark was run

Conditions

• A: clean text: the published text-track run (the same questions, prompts and options), graded against the bank as it stands now
• B: the crop(s), passage first, as the whole state; instructions 'Which option is correct?'; the question's own option letters as the choices, each described by its letter only
• C: condition A's text and options, with the crop(s) after the text; A's instructions ('Which option is the correct answer to this Turkish multiple-choice question?')
• D: one OCR model's output for the crop(s) (passage first, joined by a blank line) as the state, as its official export gives it, with layout tags removed (Unlimited-OCR-2's block boxes and page marks) and markup kept; instructions 'Which option is correct?'; option letters only, as B

Each cell is one run. It is one pass over the 763 questions, not an average of repeated runs (6 passes were resumed after an interruption; see Models and cost).

Pictures of the questions

The bank's per-question boxes (questions.bbox_json, passages.bbox_json, from its extraction run), on pages rendered from the source PDF with PyMuPDF at zoom 2.5 (180 dpi, 1600 x 1990 px), padded 4 px; on pages that print an answer key, each crop stops 6 px above the top line of the key box; crops under 280 px high are padded with white below, and crops over 1568 px on a side are scaled down to 1568 (TypeSafe's image limits; every model gets the same files). 767 images for 763 questions; 129 crops stop above an answer key box, 3 were padded and 1 scaled. Questions per topic: 1: 73, 2: 86, 3: 3, 4: 2, 5: 54, 6: 1, 7: 47, 8: 19, 9: 18, 10: 37, 11: 53, 12: 31, 13: 46, 14: 20, 15: 23, 16: 94, 17: 23, 18: 68, 19: 32, 20: 33.

We sent every crop at its own size: at most 1568 pixels a side (one crop is 1568; the rest are at most 1169). The input tokens each API reported match its documented billing grid at those sent sizes, on all 763 questions: TypeSafe's Jev, 2 tokens per 28 by 28 pixel cell plus a fixed 354 to 361 tokens (375 to 380 with two crops); Liquid's d1-ultra, 44 plus 1.5 tokens per 32 by 32 pixel patch; OpenAI's GPT-6 Luna through the Decisions API, 130 plus 1 token per 32 by 32 pixel patch with the default detail setting. A downscale before counting, to 1024 pixels or less, would have changed these counts. This shows the size at which each API counted and billed an image, not what the model saw inside it. TypeSafe documents that it rejects images outside 280 to 1568 pixels a side instead of resizing them; that is a rule about accepted inputs, not a guarantee about every internal tensor. Liquid's and OpenAI's image dimensions after tokenization are not observable to us.

Checked by eye for leaks: 20 crops, with 0 leaks found. The first detector looked only for pink or dark lines and missed the blue key boxes of some units, whose keys are printed in plain sight (page 60 shows '1-C 2-C ...'); it was replaced by a colour-blind test (any coloured or dark row across 85% of the width) before any model saw a crop; all 368 crops on key pages now stop 6 px above the box's top line (1796 or 1797 px), and no rendered page without a recorded key has such a box.

We also searched every OCR output, 3,068 in all (4 OCR models × 767 crops), for answer-key text: number-letter pairs as a key prints them (“1‑C 2‑C”) and key wording. None had key text. Six Baidu Unlimited-OCR-2 outputs each had one number-letter pair; in each, the number is the question’s own number and the letter is not its answer.

OCR models

• Chandra OCR 2 (datalab-to/chandra-ocr-2 at af93b47): chandra-ocr 0.2.0 official vLLM path (chandra.model.InferenceManager(method='vllm'), the code behind the `chandra <input> <out> --method vllm` CLI) against a vLLM 0.32.1rc1.dev613+g8977b13a1 OpenAI-compatible server (local nightly venv on 1 GPU, started with the same arguments as chandra's `chandra_vllm` launcher: --dtype bfloat16 --max-model-len 18000 --max-num-seqs 64 --max-num-batched-tokens 8192 --enable-prefix-caching --mm-processor-kwargs {min_pixels: 3136, max_pixels: 6291456}; the launcher itself would run the vllm/vllm-openai:v0.17.0 docker image, which was not used). Each crop was loaded with chandra.input.load_image (upscaled so the shorter side is >= 1536 px), resized by chandra's scale_to_fit, sent as PNG with the default 'ocr_layout' prompt (PROMPT_MAPPING['ocr_layout']); all 767 requests in one generate() call with 64 client threads. Decoding: temperature 0, top_p 0.1, max_tokens 12384 (chandra defaults); chandra's built-in retry re-generates an output with a detected repeat-token loop at temperature min(0.2*(n+1), 0.8), top_p 0.95 (up to 6 retries). The model emits layout HTML (<div data-bbox data-label> blocks); the stored text is chandra's official markdown export (BatchOutputItem.markdown = chandra.output.parse_markdown with the CLI defaults include_images=True, include_headers_footers=False, i.e. Page-Header/Page-Footer blocks dropped). dtype bfloat16. In this run chandra's client logged 39 repeat-token retry events (the retried pages' final text therefore comes from a sampled, non-greedy re-generation; chandra does not report which pages).
• dots.mocr (dots-studio/dots.mocr at e539fbb): dots_mocr parser (github.com/rednote-hilab/dots.mocr @ 23f3e5612fb8, installed editable) official vLLM path: DotsMOCRParser.parse_file(<png>, prompt_mode='prompt_layout_all_en', fitz_preprocess=True), i.e. `python3 dots_mocr/parser.py <png>` with the CLI defaults (prompt_layout_all_en, top_p 1.0, max_completion_tokens 16384, dpi 200 fitz re-render of the image, no min/max_pixels override) except temperature 0.0 instead of the CLI's 0.1 for greedy decoding; requests go to a vLLM 0.32.1rc1.dev613+g8977b13a1 OpenAI-compatible server (local nightly venv, native DotsOCRForCausalLM implementation, launched as in the model card: --gpu-memory-utilization 0.9 --chat-template-content-format string --served-model-name model --trust-remote-code; model dtype bfloat16 from config). 64 images in flight. The parser's request function was replaced by an identical copy that also records finish_reason (to detect token-limit hits). The model returns layout JSON (bbox, category, text); the stored text is the parser's own markdown export (<name>.md = format_transformer.layoutjson2md(..., no_page_hf=False): cells in reading order joined by blank lines, Formula as LaTeX, Table as HTML, Picture cells as an inline base64 PNG image link); when the JSON fails to parse, the parser's OutputCleaner fallback text is stored (noted in failures).
• Surya OCR 2 (datalab-to/surya-ocr-2 at 3b3d4cd): surya-ocr 0.22.1 official full-page OCR path (the code behind `surya_ocr <folder>`: surya.input.load.load_from_folder, then RecognitionPredictor(SuryaInferenceManager())(images, full_page=True)) against a vLLM 0.20.1 OpenAI-compatible server (vllm==0.20.1 from PyPI in an isolated venv on 1 GPU, the version of surya's vllm/vllm-openai:v0.20.1 docker image, started with the same arguments surya's VllmBackend gives that image for VLLM_GPU_TYPE=h100: --dtype bfloat16 --max-model-len 18000 --max-num-seqs 104 --max-num-batched-tokens 16384 --enable-prefix-caching --mm-processor-kwargs {min_pixels: 3136, max_pixels: 6291456} --speculative-config {method: mtp, num_speculative_tokens: 2}; docker itself was not used). The server venv pins transformers 5.17.0 (surya requires >=5.12.1): transformers >=5.18 loads this checkpoint's char-level tokenizer as the byte-level Qwen3_5Tokenizer, which corrupted every prompt and turned non-ASCII output letters (ı ğ ç ö ü ...) into U+FFFD/control bytes in a first run with vLLM nightly + transformers 5.20 (discarded). Attached via SURYA_INFERENCE_BACKEND=vllm, SURYA_INFERENCE_URL; client concurrency 96 (surya default for that sizing). Each crop is sent once as PNG (after surya's scale_to_fit) with the default full-page prompt HIGH_ACCURACY_BBOX_PROMPT; decoding: temperature 0, top_p 0.1, max_tokens 12288 (surya defaults; SURYA_FULLPAGE_REGEN off), surya's client retries a request that ends in a repeat-token loop at temperature min(0.2*(n+1), 0.8), top_p 0.95 (max 3 retries), and a page whose full-page HTML fails to parse or loops falls back to surya's layout + per-block OCR. The model emits HTML layout blocks (<div data-bbox data-label>); surya parses them into PageOCRResult.blocks in reading order with per-block inner HTML. Surya has no markdown/text export of its own (that lives in marker), so the stored text is the non-skipped blocks' `html` (surya's official per-block output, math in <math>, tables as <table>) joined in reading order with blank lines; bounding boxes and labels are dropped. dtype bfloat16. In this run: 101 client-side retry requests (non-greedy), 101 of them full-page; 2 pages used the layout + per-block fallback (12 layout/block requests).
• Baidu Unlimited-OCR-2 (baidu/Unlimited-OCR-2 at 04647c7): Model card's Transformers path (trust_remote_code; the snapshot's modeling_unlimitedocr.py), README 'Single-page inference': AutoModel.from_pretrained(..., trust_remote_code=True, use_safetensors=True, torch_dtype=torch.bfloat16).eval().cuda(), then model.infer(tokenizer, image_file=<png>, output_path=<dir>, save_results=True) per crop with all decoding defaults: prompt built automatically as '<PAGE><image>Multi page parsing.' for one page, crop_mode=None -> gundam dynamic tiling for a single page (base_size 1024, image_size 640, max_crops 16), greedy argmax (temperature 0), PageCountGuard on with max_page_tokens=20000, max_length 131072, no n-gram blocking, DASH sparse attention (dash_topm 16). eval_mode=True only disables the console token streamer; return_dict=True only returns token ids/finish reason. Batch size 1 per model copy (the path has no batching); 8 worker processes, each with its own model copy, shared GPU 2 and took the crops round-robin; seconds = first worker start (after all models loaded) to last worker end. The model emits markdown with grounding refs (<|ref|>..<|/ref|><|det|>[[boxes]]<|/det|>) and a <PAGE> marker; the stored text is the model's own save_results output result.md (its _uocr_save_results: non-image refs/boxes removed, image refs replaced by  links, '<PAGE>' line kept). torch 2.13.0+cu130, transformers 4.52.1, dtype bfloat16.

Models and cost

Every request went through the benchmark's runner (the runs record its commit: 9162663 for runs 1 to 28, on the published main; 0fbd97b for runs 29 to 55, merged into the published main in fff7c20d; 61f3824 for runs 56 to 59, merged into the published main in 4908dd9d; 737862e for runs 60 and 61, merged into the published main in 80380b72; 0ad2797 for runs 62 to 67, merged into the published main in c458470b), one question per request; the images went in each API's own format. Self-hosted models ran on 1 × RTX PRO 6000 Blackwell (96 GB); BF16; 1 × RTX PRO 6000 Blackwell (96 GB); BF16 (the published text run used a DGX Spark); 1 × RTX PRO 6000 Blackwell (96 GB); BF16; d1_server.py with the model card's system_one (transformers 5.22.1, torch 2.15.0); 1 × RTX PRO 6000 Blackwell (96 GB); BF16; drex_max_server.py with Nace's drex_max at e206b27 (transformers 5.17.0, torch 2.15.0, flash-linear-attention 0.6.0; causal_conv1d not installed, so its reference PyTorch path); 1 × RTX PRO 6000 Blackwell (96 GB); BF16; vLLM 0.32.1rc1.dev613+g8977b13a1 with --max-num-seqs 64 --gpu-memory-utilization 0.78 (another process held 18 GB of the GPU). The API image runs (B and C) cost $0.40 and the API runs of condition D $0.29, at the prices in results.json.

Engine check: we asked vLLM-SR d3's condition B again through d3_server.py on transformers, the server d3-ultra ran on (run 55): 365 of 763 right (47.8%), against 370 (48.5%) through vLLM (run 41), the run in the table. Exact McNemar test: 10 questions right only on transformers and 15 only on vLLM, p = 0.42; 47 of 763 picks differ.

Each cell holds one stored answer per question: 45,780 in all, from 66 runs. Six cells were finished by a second run after an interruption (Jev 1.14.0 B, runs 1 and 6; Liquid d1-ultra B, runs 2 and 5; GPT-6 Luna B, runs 3 and 4; GPT-6 Luna D · dots.mocr, runs 17 and 27; OneJev 27B B, runs 28 and 35; Liquid d1.1 D · dots.mocr, runs 16 and 26); these are continuations, not repeats. 45,764 answers name an option and 16 are refusals. The runs’ logs record 143 client retries: 5 in run 4 and 2 in run 9, both GPT-6 Luna after HTTP 503 errors, and 2 in run 34, d3-ultra after HTTP 500, and 54 in run 58 and 80 in run 59, both Drex Max 1.1 after HTTP 429 errors. The logs do not cover every run, and an attempt that failed and was asked again is not stored, so 45,923 is a lower bound on the requests sent. Tokens and costs come from the stored responses only: not every attempt and not an invoice. The usage and cost of attempts that were not kept are unknown, not zero.

Liquid d1-2B (open weights) answered conditions B and C four more times with the same inputs and runner (passes 2 to 5, in vision-repeats.json); every pass gave the same picks and option probabilities as the table's runs 60 and 61. The table shows the first pass, chosen as the cell before the runs.