Method

How the two benchmarks are built, run and scored

Decision and chat setups

The setups and the methodology are those of the team's decision-model benchmark (its 40 decision-model setups, run with the same runner). A setup is one model behind one endpoint with its settings (reasoning level, request size, hardware). On these pages a setup is named by its model's official name as its maker writes it, with a small tag after it: a chat model's reasoning setting and “chat”, the route where two setups of one model differ by it, a snapshot or version; then the class label (Open or Closed, with the size or the price class). Each run record below keeps the setup's full label.

Setups shown once (named “first = second”, with the first's numbers; both stay in the database and under All setups on the leaderboard):

Measures

Sources

Logic: 1,125 questions

English translations of questions from a published question bank (not named here): numerical reasoning, verbal reasoning and formal logic; set problems and every question that needs a figure are left out.

SourceParsedquestions parsedSet problemsexcludedWith figuresexcludedIn the benchmark
The question bank (published, not named here)1,8093783061,125

Math: 2,163 questions

English translations of the text-only math questions of a published question bank (not named here), in 13 topics: Basic concepts, Digits and place value, Number bases, Division and divisibility, Prime factorisation and GCD/LCM, Rational numbers, Decimals, Ordering, Simple inequalities, Absolute value, Exponents, Radicals, Factorisation.

SourceParsedquestions parsedSet problemsexcludedWith figuresexcludedIn the benchmark
The question bank (published, not named here)–––2,163

The question text is not shown; these pages show aggregate results only.

Scoring

The English questions

Every question is asked in English. The English texts were translated from the question bank by a model, with automatic checks (option keys, digits, LaTeX, tables, labels, leftover untranslated words); a text that still failed after one retry, or a question that depends on the original language's words or letters, is flagged and not asked. The answer key is the question bank's answer key.

Logic

StatusProducertranslating modelQuestions
okdeepseek-v4-flash-dspark991
okdeepseek-v4-flash-dspark+review134

Math

StatusProducertranslating modelQuestions
okdeepseek-v4-flash-dspark2,071
okdeepseek-v4-flash-dspark+review92

Runs: prompts and parameters

Logic · gemma-4-31b-it_en_t0

Name
Gemma 4 31B chat
Setup
Gemma 4 31B (chat, on our GPU) · chat model
Model
gemma-4-31B-it · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.399 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.562 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.399. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-06 17:59:58–20:45:26 UTC
GPU time
run gemma-4-31b-it_en_t0 · first answer 2026-11-06 17:28:39 UTC · last answer 2026-11-06 20:47:50 UTC · span 11951.0 s, pauses included (not charged) · charged 2023.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.3992 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 13.7 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 1 answer cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 30.7B dense · source https://huggingface.co/google/gemma-4-31B-it · read 2026-11-10 · the card states Total Parameters 30.7B (plus a vision encoder of about 550M) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 31.27B (counted from the checkpoint (31,273,088,876, the vision encoder included); the card gives 30.7B plus a vision encoder of about 550M)
Input
text, English
Endpoint
http://127.0.0.1:8021/v1/chat/completions
Started
6 Nov 2026 17:28 UTC; last answer 6 Nov 2026 20:47 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Gemma 4 31B-it FP8 weights + FP8 KV (vLLM v0.30.0, GPU 0 of our GPU host; same flags as the shared Spark host's server), thinking off (server default), temperature 0, English translations (DeepSeek + review)
Parameters
{
  "max_tokens": 8192,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · qwen3.8-27b-bf16_en_low

Name
Qwen3.8-27B low · chat
Setup
Qwen3.8-27B (chat, reasoning low, on our GPU) · chat model
Model
qwen38-27b-bf16 · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$5.495 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 2.21 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $5.495. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-06 19:58:26–20:27:18 UTC; 2026-11-06 20:41:13–21:35:53 UTC
GPU time
run qwen3.8-27b-bf16_en_low · first answer 2026-11-06 18:04:10 UTC · last answer 2026-11-06 21:40:06 UTC · span 12956.0 s, pauses included (not charged) · charged 7944.0 s in 3 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $5.4946 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 6.3 in flight (sum of request latencies / active time)
Output limit
60,000 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 27B dense · source https://huggingface.co/Qwen/Qwen3.8-27B · read 2026-11-10 · the card states Number of Parameters: 27B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 27.78B (counted from the checkpoint (27,781,427,952, the vision encoder included); the card says 27B)
Input
text, English
Endpoint
http://127.0.0.1:8022/v1/chat/completions
Started
6 Nov 2026 18:04 UTC; last answer 6 Nov 2026 21:40 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Qwen3.8-27B BF16 (vLLM v0.30.0, MTP, GPU 0 of our GPU host), thinking on at reasoning_effort low with the model's generation config sampling (1.0/0.95/20, seed 0) and a JSON answer schema, as turkce's run 318; English translations (DeepSeek + review)
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "reasoning_effort": "low"
  },
  "max_tokens": 60000,
  "response_format": {
    "json_schema": {
      "name": "answer",
      "schema": {
        "additionalProperties": false,
        "properties": {
          "answer": {
            "enum": [
              "A",
              "B",
              "C",
              "D",
              "E"
            ],
            "type": "string"
          }
        },
        "required": [
          "answer"
        ],
        "type": "object"
      },
      "strict": true
    },
    "type": "json_schema"
  },
  "seed": 0,
  "temperature": 1.0,
  "top_k": 20,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. Give your answer only as JSON {{"answer": "X"}} (X = A, B, C, D or E).

{question}

Logic · decider-flash-v1.2_en

Name
Perplexity Decider Flash v1.2
Setup
Perplexity Decider Flash v1.2 · decision model
Model
pplx-decider-flash-v1.2 · provider pplx-decider
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.004229 for the run, list · maker's list price; RTX PRO 6000 Blackwell: $0.03873 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.0156 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.039. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run decider-flash-v1.2_en · first answer 2026-11-07 13:39:34 UTC · last answer 2026-11-07 13:40:30 UTC · span 56.0 s, pauses included (not charged) · charged 56.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0387 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 4B dense · source https://huggingface.co/perplexity-ai/pplx-decider-flash-v1.2 · read 2026-11-10 · the card: built on Qwen3.5-4B, 'about one seventh of the size' of the 27B models; Ümit/turkce #4263: 4B (turkce #4263: the maker's stated total is shown); counted from the released files: 4.54B (counted from the released checkpoint: 4,539,918,336 (based on Qwen3.5-4B))
Input
text, English
Endpoint
http://127.0.0.1:19180
Started
7 Nov 2026 13:39 UTC; last answer 7 Nov 2026 13:40 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Perplexity Decider Flash v1.2 (site run 223); runner ffc307cb7; published on a DGX Spark (GB10); gpu-lanes runs it on an RTX PRO 6000
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "pplx-decider-flash-v1.2",
    "--provider",
    "pplx-decider",
    "--timeout",
    "30",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "pplx-decider-flash-v1.2",
  "perplexity_mode": null,
  "price": {
    "applies_to": "maker",
    "basis": "Perplexity's list price for the same model",
    "date": "2026-10-18",
    "extra_input_per_request": 3,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.01,
      "cache_write_input_tokens": 0.01,
      "cached_input_tokens": 0.01,
      "input_tokens": 0.01,
      "output_tokens": 0.0
    },
    "source": "https://docs.perplexity.ai/docs/getting-started/pricing",
    "text": "$0.01 input, output free per 1M tokens, plus 3 billed input tokens per request"
  },
  "prompt_version": "47f5776e71db",
  "provider": "pplx-decider",
  "questions": 1125,
  "request_seconds": 52.06132212583907,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "c459b8284bae40dff7f6cff48292343aaaed3845d1cd3dc0d1e2a1afaa9312cf",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 223,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 419507,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19180/v1/systemone
{
 "model": "pplx-decider-flash-v1.2",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · decider-flash-v1.2_en_dontknow

Name
Perplexity Decider Flash v1.2
Setup
Perplexity Decider Flash v1.2 · decision model · with the I don't know option F
Model
pplx-decider-flash-v1.2 · provider pplx-decider
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.004566 for the run, list · maker's list price; RTX PRO 6000 Blackwell: $0.03873 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.0156 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.039. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run decider-flash-v1.2_en_dontknow · first answer 2026-11-07 13:40:34 UTC · last answer 2026-11-07 13:41:30 UTC · span 56.0 s, pauses included (not charged) · charged 56.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0387 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 4B dense · source https://huggingface.co/perplexity-ai/pplx-decider-flash-v1.2 · read 2026-11-10 · the card: built on Qwen3.5-4B, 'about one seventh of the size' of the 27B models; Ümit/turkce #4263: 4B (turkce #4263: the maker's stated total is shown); counted from the released files: 4.54B (counted from the released checkpoint: 4,539,918,336 (based on Qwen3.5-4B))
Input
text, English
Endpoint
http://127.0.0.1:19180
Started
7 Nov 2026 13:40 UTC; last answer 7 Nov 2026 13:41 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Perplexity Decider Flash v1.2 (site run 223); runner ffc307cb7; published on a DGX Spark (GB10); gpu-lanes runs it on an RTX PRO 6000
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "pplx-decider-flash-v1.2",
    "--provider",
    "pplx-decider",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "pplx-decider-flash-v1.2",
  "perplexity_mode": null,
  "price": {
    "applies_to": "maker",
    "basis": "Perplexity's list price for the same model",
    "date": "2026-10-18",
    "extra_input_per_request": 3,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.01,
      "cache_write_input_tokens": 0.01,
      "cached_input_tokens": 0.01,
      "input_tokens": 0.01,
      "output_tokens": 0.0
    },
    "source": "https://docs.perplexity.ai/docs/getting-started/pricing",
    "text": "$0.01 input, output free per 1M tokens, plus 3 billed input tokens per request"
  },
  "prompt_version": "413d6775fc58",
  "provider": "pplx-decider",
  "questions": 1125,
  "request_seconds": 52.62055526894983,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "1153ff96fee1441c81a2f1bc0dda4b8a90d01397c4b1a04ba7b9d96090486453",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 223,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 453257,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19180/v1/systemone
{
 "model": "pplx-decider-flash-v1.2",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · glide-v1.2_en

Name
GLiDE v1.2
Setup
Fastino GLiDE v1.2 · decision model
Model
glide-v1.2 · provider glide
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.01561 for the run, list · maker's list price; RTX PRO 6000 Blackwell: $0.09199 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.0369 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.092. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run glide-v1.2_en · first answer 2026-11-07 13:43:06 UTC · last answer 2026-11-07 13:45:19 UTC · span 133.0 s, pauses included (not charged) · charged 133.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0920 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/fastino/GLiDE · read 2026-11-10 · the card states a 27B GLiDE decision model (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 26.09B (counted from the checkpoint (26,086,635,760: language model, vision encoder, readout); Fastino's blog says 26.09B)
Input
text, English
Endpoint
http://127.0.0.1:19110
Started
7 Nov 2026 13:43 UTC; last answer 7 Nov 2026 13:45 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Fastino GLiDE v1.2 (site run 248); runner ffc307cb7; published on a DGX Spark (GB10); gpu-lanes runs it on an RTX PRO 6000
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "glide-v1.2",
    "--provider",
    "glide",
    "--timeout",
    "60",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "glide-v1.2",
  "perplexity_mode": null,
  "price": {
    "applies_to": "maker",
    "basis": "Fastino's list price for the same model",
    "date": "2026-10-22",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.04,
      "cache_write_input_tokens": 0.04,
      "cached_input_tokens": 0.04,
      "input_tokens": 0.04,
      "output_tokens": 0.0
    },
    "source": "https://docs.fastino.ai/pricing",
    "text": "$0.04 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "glide",
  "questions": 1125,
  "request_seconds": 129.57132538210135,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "a5c2e5a548f4ccec5d3df09af8920773576e117443270d45cf416b143b8aff3a",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 248,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 390257,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19110/v1/systemone
{
 "model": "glide-v1.2",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · glide-v1.2_en_dontknow

Name
GLiDE v1.2
Setup
Fastino GLiDE v1.2 · decision model · with the I don't know option F
Model
glide-v1.2 · provider glide
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.01696 for the run, list · maker's list price; RTX PRO 6000 Blackwell: $0.09753 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.0392 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.098. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run glide-v1.2_en_dontknow · first answer 2026-11-07 13:45:25 UTC · last answer 2026-11-07 13:47:46 UTC · span 141.0 s, pauses included (not charged) · charged 141.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0975 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/fastino/GLiDE · read 2026-11-10 · the card states a 27B GLiDE decision model (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 26.09B (counted from the checkpoint (26,086,635,760: language model, vision encoder, readout); Fastino's blog says 26.09B)
Input
text, English
Endpoint
http://127.0.0.1:19110
Started
7 Nov 2026 13:45 UTC; last answer 7 Nov 2026 13:47 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Fastino GLiDE v1.2 (site run 248); runner ffc307cb7; published on a DGX Spark (GB10); gpu-lanes runs it on an RTX PRO 6000
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "glide-v1.2",
    "--provider",
    "glide",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "glide-v1.2",
  "perplexity_mode": null,
  "price": {
    "applies_to": "maker",
    "basis": "Fastino's list price for the same model",
    "date": "2026-10-22",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.04,
      "cache_write_input_tokens": 0.04,
      "cached_input_tokens": 0.04,
      "input_tokens": 0.04,
      "output_tokens": 0.0
    },
    "source": "https://docs.fastino.ai/pricing",
    "text": "$0.04 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "glide",
  "questions": 1125,
  "request_seconds": 138.09481043100823,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "5783e6415a3c91f65e883c8e8ab3ecdd639fe8df22f870a309bc368a76157e50",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 248,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 424007,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19110/v1/systemone
{
 "model": "glide-v1.2",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · d1-2b_en

Name
d1-2B
Setup
Liquid d1-2B (open weights) · decision model
Model
LiquidAI/d1-2B · provider d1-server
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.0166 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.00667 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.017. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run d1-2b_en · first answer 2026-11-07 13:49:27 UTC · last answer 2026-11-07 13:49:51 UTC · span 24.0 s, pauses included (not charged) · charged 24.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0166 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 1.92B dense · source https://huggingface.co/LiquidAI/d1-2B · read 2026-11-09 · the card: 1.92B parameters; counted from the checkpoint 1,918,181,888
Input
text, English
Endpoint
http://127.0.0.1:19260
Started
7 Nov 2026 13:49 UTC; last answer 7 Nov 2026 13:49 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Liquid d1-2B (open weights) (site run 340); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "LiquidAI/d1-2B",
    "--provider",
    "d1-server",
    "--timeout",
    "30",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "LiquidAI/d1-2B",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "47f5776e71db",
  "provider": "d1-server",
  "questions": 1125,
  "request_seconds": 20.600843607680872,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "c40af662111bd6dd20305152496df83af7635697780d49440d4d0133e31828d6",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 340,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 336401,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19260/v1/systemone
{
 "model": "LiquidAI/d1-2B",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · d1-2b_en_dontknow

Name
d1-2B
Setup
Liquid d1-2B (open weights) · decision model · with the I don't know option F
Model
LiquidAI/d1-2B · provider d1-server
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.01591 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.00639 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.016. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run d1-2b_en_dontknow · first answer 2026-11-07 13:49:55 UTC · last answer 2026-11-07 13:50:18 UTC · span 23.0 s, pauses included (not charged) · charged 23.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0159 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 1.92B dense · source https://huggingface.co/LiquidAI/d1-2B · read 2026-11-09 · the card: 1.92B parameters; counted from the checkpoint 1,918,181,888
Input
text, English
Endpoint
http://127.0.0.1:19260
Started
7 Nov 2026 13:49 UTC; last answer 7 Nov 2026 13:50 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Liquid d1-2B (open weights) (site run 340); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "LiquidAI/d1-2B",
    "--provider",
    "d1-server",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "LiquidAI/d1-2B",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "413d6775fc58",
  "provider": "d1-server",
  "questions": 1125,
  "request_seconds": 19.78920921694953,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "01f5081d8c79894b0c0f65c334ae63d92266068869a20cbd94113321b89fc436",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 340,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 366703,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19260/v1/systemone
{
 "model": "LiquidAI/d1-2B",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · strands_en

Name
Strands Decider 2B
Setup
Strands Decider 2B · decision model
Model
strands-decider-2b · provider strands
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.03113 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0125 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.031. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run strands_en · first answer 2026-11-07 13:55:31 UTC · last answer 2026-11-07 13:56:16 UTC · span 45.0 s, pauses included (not charged) · charged 45.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0311 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 2B dense · source our local copy of the model files · read 2026-11-10 · the official model name: Strands Decider 2B (turkce #4263: the maker's stated total is shown); counted from the released files: 2.29B (counted from the checkpoint: Qwen3.5-2B-Base with its LoRA and head (2,291,942,208); the Strands card states no total)
Input
text, English
Endpoint
http://127.0.0.1:19270
Started
7 Nov 2026 13:55 UTC; last answer 7 Nov 2026 13:56 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Strands Decider 2B (site run 141); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "strands-decider-2b",
    "--provider",
    "strands",
    "--timeout",
    "120",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "strands-decider-2b",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "47f5776e71db",
  "provider": "strands",
  "questions": 1125,
  "request_seconds": 41.59566023084335,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "29f6be3db6046948643e08083b3f7fa8213764cfb96ac23829239b2f6a4328c1",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 141,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 390246,
    "output_tokens": 1125
  }
}
Prompt template
POST http://127.0.0.1:19270/v1/systemone
{
 "model": "strands-decider-2b",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · strands_en_dontknow

Name
Strands Decider 2B
Setup
Strands Decider 2B · decision model · with the I don't know option F
Model
strands-decider-2b · provider strands
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.03113 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0125 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.031. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run strands_en_dontknow · first answer 2026-11-07 13:56:20 UTC · last answer 2026-11-07 13:57:05 UTC · span 45.0 s, pauses included (not charged) · charged 45.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0311 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 2B dense · source our local copy of the model files · read 2026-11-10 · the official model name: Strands Decider 2B (turkce #4263: the maker's stated total is shown); counted from the released files: 2.29B (counted from the checkpoint: Qwen3.5-2B-Base with its LoRA and head (2,291,942,208); the Strands card states no total)
Input
text, English
Endpoint
http://127.0.0.1:19270
Started
7 Nov 2026 13:56 UTC; last answer 7 Nov 2026 13:57 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Strands Decider 2B (site run 141); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "strands-decider-2b",
    "--provider",
    "strands",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "strands-decider-2b",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "413d6775fc58",
  "provider": "strands",
  "questions": 1125,
  "request_seconds": 41.1998724238947,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "81ea216b220ccc8ae70d99592d1414ea0a9e0a8d6cf33db2652974e835002960",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 141,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 423996,
    "output_tokens": 1125
  }
}
Prompt template
POST http://127.0.0.1:19270/v1/systemone
{
 "model": "strands-decider-2b",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · decider-v1.2-local_en

Name
Perplexity Decider 27B v1.2 NVIDIA RTX PRO 6000 Blackwell
Setup
Perplexity Decider 27B v1.2 (local GPU) · decision model
Model
pplx-decider-v1.2 · provider pplx-decider
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.008458 for the run, list · maker's list price; RTX PRO 6000 Blackwell: $0.1176 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.0472 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.118. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run decider-v1.2-local_en · first answer 2026-11-07 14:07:38 UTC · last answer 2026-11-07 14:10:28 UTC · span 170.0 s, pauses included (not charged) · charged 170.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1176 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/perplexity-ai/pplx-decider-v1.2 · read 2026-11-10 · the official model name: 27B (pplx-decider 27B v1.2; the card states no count of its own) (turkce #4263: the maker's stated total is shown); counted from the released files: 26.09B (counted from the released checkpoint: backbone 26,085,330,160 (its 11 safetensors shards) + decision head (readout.safetensors) 1,305,600 = 26,086,635,760; the API setups use the same count)
Input
text, English
Endpoint
http://127.0.0.1:19170
Started
7 Nov 2026 14:07 UTC; last answer 7 Nov 2026 14:10 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Perplexity Decider 27B v1.2 (local GPU) (site run 217); runner ffc307cb7; published on a DGX Spark (GB10); gpu-lanes runs it on an RTX PRO 6000
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "pplx-decider-v1.2",
    "--provider",
    "pplx-decider",
    "--timeout",
    "30",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "pplx-decider-v1.2",
  "perplexity_mode": null,
  "price": {
    "applies_to": "maker",
    "basis": "Perplexity's list price for the same model",
    "date": "2026-10-18",
    "extra_input_per_request": 3,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.02,
      "cache_write_input_tokens": 0.02,
      "cached_input_tokens": 0.02,
      "input_tokens": 0.02,
      "output_tokens": 0.0
    },
    "source": "https://docs.perplexity.ai/docs/getting-started/pricing",
    "text": "$0.02 input, output free per 1M tokens, plus 3 billed input tokens per request"
  },
  "prompt_version": "47f5776e71db",
  "provider": "pplx-decider",
  "questions": 1125,
  "request_seconds": 165.602628360386,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "32256482f5b9eb6af80f14a003c8232113f44bf7b59f64091594beb31e5a6a0e",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 217,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 419507,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19170/v1/systemone
{
 "model": "pplx-decider-v1.2",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · decider-v1.2-local_en_dontknow

Name
Perplexity Decider 27B v1.2 NVIDIA RTX PRO 6000 Blackwell
Setup
Perplexity Decider 27B v1.2 (local GPU) · decision model · with the I don't know option F
Model
pplx-decider-v1.2 · provider pplx-decider
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.009133 for the run, list · maker's list price; RTX PRO 6000 Blackwell: $0.1238 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.0497 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.124. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run decider-v1.2-local_en_dontknow · first answer 2026-11-07 14:10:34 UTC · last answer 2026-11-07 14:13:33 UTC · span 179.0 s, pauses included (not charged) · charged 179.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1238 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/perplexity-ai/pplx-decider-v1.2 · read 2026-11-10 · the official model name: 27B (pplx-decider 27B v1.2; the card states no count of its own) (turkce #4263: the maker's stated total is shown); counted from the released files: 26.09B (counted from the released checkpoint: backbone 26,085,330,160 (its 11 safetensors shards) + decision head (readout.safetensors) 1,305,600 = 26,086,635,760; the API setups use the same count)
Input
text, English
Endpoint
http://127.0.0.1:19170
Started
7 Nov 2026 14:10 UTC; last answer 7 Nov 2026 14:13 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Perplexity Decider 27B v1.2 (local GPU) (site run 217); runner ffc307cb7; published on a DGX Spark (GB10); gpu-lanes runs it on an RTX PRO 6000
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "pplx-decider-v1.2",
    "--provider",
    "pplx-decider",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "pplx-decider-v1.2",
  "perplexity_mode": null,
  "price": {
    "applies_to": "maker",
    "basis": "Perplexity's list price for the same model",
    "date": "2026-10-18",
    "extra_input_per_request": 3,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.02,
      "cache_write_input_tokens": 0.02,
      "cached_input_tokens": 0.02,
      "input_tokens": 0.02,
      "output_tokens": 0.0
    },
    "source": "https://docs.perplexity.ai/docs/getting-started/pricing",
    "text": "$0.02 input, output free per 1M tokens, plus 3 billed input tokens per request"
  },
  "prompt_version": "413d6775fc58",
  "provider": "pplx-decider",
  "questions": 1125,
  "request_seconds": 175.2436724221334,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "9f030c9df1ebfa3b360e8e9b7db26cf33885d7dc74b9dd3552bf0ea9b9f58808",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 217,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 453257,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19170/v1/systemone
{
 "model": "pplx-decider-v1.2",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · metask_en

Name
Metask-Jev-4B
Setup
Metask-Jev 4B · decision model
Model
metask-jev-4b-policy-mix · provider metask
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.1024 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0411 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.102. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run metask_en · first answer 2026-11-07 14:23:10 UTC · last answer 2026-11-07 14:25:38 UTC · span 148.0 s, pauses included (not charged) · charged 148.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1024 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 4.54B dense · source our local copy of the model files · read 2026-11-09 · the card: 4.54B parameters (Qwen3.5-4B with its LoRA merged)
Input
text, English
Endpoint
http://127.0.0.1:19250
Started
7 Nov 2026 14:23 UTC; last answer 7 Nov 2026 14:25 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Metask-Jev 4B (site run 126); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "metask-jev-4b-policy-mix",
    "--provider",
    "metask",
    "--timeout",
    "120",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "metask-jev-4b-policy-mix",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "47f5776e71db",
  "provider": "metask",
  "questions": 1125,
  "request_seconds": 144.21359020844102,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "24a7c1cd03a70339fc25b967ddcd43b9ea29e694646dafe3a7b27ea609593e5e",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 126,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 635575,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19250/v1/systemone
{
 "model": "metask-jev-4b-policy-mix",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · metask_en_dontknow

Name
Metask-Jev-4B
Setup
Metask-Jev 4B · decision model · with the I don't know option F
Model
metask-jev-4b-policy-mix · provider metask
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.1058 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0425 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.106. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run metask_en_dontknow · first answer 2026-11-07 14:25:44 UTC · last answer 2026-11-07 14:28:17 UTC · span 153.0 s, pauses included (not charged) · charged 153.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1058 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 4.54B dense · source our local copy of the model files · read 2026-11-09 · the card: 4.54B parameters (Qwen3.5-4B with its LoRA merged)
Input
text, English
Endpoint
http://127.0.0.1:19250
Started
7 Nov 2026 14:25 UTC; last answer 7 Nov 2026 14:28 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Metask-Jev 4B (site run 126); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "metask-jev-4b-policy-mix",
    "--provider",
    "metask",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "metask-jev-4b-policy-mix",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "413d6775fc58",
  "provider": "metask",
  "questions": 1125,
  "request_seconds": 148.99273813853506,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "2c806d0b8e3f41f53e0331dd09125572a2f4c561e8f70182ff7a51051b3a8161",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 126,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 683114,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19250/v1/systemone
{
 "model": "metask-jev-4b-policy-mix",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · drex-max-v1_en

Name
Drex Max
Setup
Drex Max v1 · decision model
Model
drex-max-v1 · provider drex-max
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.02537 for the run, list · maker's list price; RTX PRO 6000 Blackwell: $0.0913 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.0367 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.091. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run drex-max-v1_en · first answer 2026-11-07 14:35:08 UTC · last answer 2026-11-07 14:37:20 UTC · span 132.0 s, pauses included (not charged) · charged 132.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0913 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/nace-ai/drex-max-v1 · read 2026-11-10 · the card states a dense 27B-parameter model (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 26.09B (counted from the released checkpoint: backbone 26,085,330,160 (its 11 safetensors shards) + decision head (readout.safetensors) 1,305,600 = 26,086,635,760)
Input
text, English
Endpoint
http://127.0.0.1:19140
Started
7 Nov 2026 14:35 UTC; last answer 7 Nov 2026 14:37 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Drex Max v1 (site run 322); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "drex-max-v1",
    "--provider",
    "drex-max",
    "--timeout",
    "30",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "drex-max-v1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "maker",
    "basis": "Nace.AI's list price for the same model",
    "date": "2026-10-30",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.06,
      "cache_write_input_tokens": 0.06,
      "cached_input_tokens": 0.06,
      "input_tokens": 0.06,
      "output_tokens": 0.0
    },
    "source": "https://nace.ai/drex",
    "text": "$0.06 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "drex-max",
  "questions": 1125,
  "request_seconds": 128.37476531392895,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "8818123ec6b74f4ea011db8bd063b2ca5c866ae2215f6daf1c88d4c2d5c4cf66",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 322,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 422882,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19140/v1/systemone
{
 "model": "drex-max-v1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · drex-max-v1_en_dontknow

Name
Drex Max
Setup
Drex Max v1 · decision model · with the I don't know option F
Model
drex-max-v1 · provider drex-max
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.0274 for the run, list · maker's list price; RTX PRO 6000 Blackwell: $0.09822 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.0394 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.098. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run drex-max-v1_en_dontknow · first answer 2026-11-07 14:37:25 UTC · last answer 2026-11-07 14:39:47 UTC · span 142.0 s, pauses included (not charged) · charged 142.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0982 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/nace-ai/drex-max-v1 · read 2026-11-10 · the card states a dense 27B-parameter model (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 26.09B (counted from the released checkpoint: backbone 26,085,330,160 (its 11 safetensors shards) + decision head (readout.safetensors) 1,305,600 = 26,086,635,760)
Input
text, English
Endpoint
http://127.0.0.1:19140
Started
7 Nov 2026 14:37 UTC; last answer 7 Nov 2026 14:39 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Drex Max v1 (site run 322); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "drex-max-v1",
    "--provider",
    "drex-max",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "drex-max-v1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "maker",
    "basis": "Nace.AI's list price for the same model",
    "date": "2026-10-30",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.06,
      "cache_write_input_tokens": 0.06,
      "cached_input_tokens": 0.06,
      "input_tokens": 0.06,
      "output_tokens": 0.0
    },
    "source": "https://nace.ai/drex",
    "text": "$0.06 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "drex-max",
  "questions": 1125,
  "request_seconds": 136.4533581068972,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "aa048de7eff6ddd0eea1c1e201a899a6b3dca61b5198e00f5cf037579d0a0c67",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 322,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 456632,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19140/v1/systemone
{
 "model": "drex-max-v1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · winnow_en

Name
Winnow-12B
Setup
Winnow-12B · decision model
Model
winnow-12b · provider winnow
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.0581 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0233 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.058. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-07 14:50:00–19:32:12 UTC
GPU time
run winnow_en · first answer 2026-11-07 14:49:42 UTC · last answer 2026-11-07 19:33:18 UTC · span 17016.0 s, pauses included (not charged) · charged 84.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0581 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 12B dense · source https://huggingface.co/EldanRing/Winnow-12B · read 2026-11-10 · the official model name: Winnow-12B (turkce #4263: the maker's stated total is shown); counted from the released files: 11.91B (counted from the Q8_0 GGUF checkpoint (11,907,350,576); a fine-tune of Gemma 4 12B IT, whose card says 11.95B)
Input
text, English
Endpoint
http://127.0.0.1:19240
Started
7 Nov 2026 14:49 UTC; last answer 7 Nov 2026 19:33 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Winnow-12B (site run 128); runner ffc307cb7; server context 8,192 tokens
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "winnow-12b",
    "--provider",
    "winnow",
    "--timeout",
    "180",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "winnow-12b",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "47f5776e71db",
  "provider": "winnow",
  "questions": 1125,
  "request_seconds": 80.02928273492944,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "2d57383fc8edbe69fad587211cdb32142f8535ccce0939ec2e2d8e57b983a167",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 128,
  "status": "completed",
  "timeout": 180.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 508275,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19240/v1/systemone
{
 "model": "winnow-12b",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · glide_en

Name
GLiDE API
Setup
Fastino GLiDE · decision model
Model
fastino/GLiDE · provider fastino
GPU
—
Cost
$0.05854 for the run · provider price × reported usage
Answer time
run glide_en · first answer 2026-11-07 14:50:19 UTC · last answer 2026-11-07 14:51:38 UTC · span 79.0 s, pauses included (not charged) · charged 79.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 4.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://docs.fastino.ai/inference/systemone · read 2026-11-09 · not disclosed by maker for the hosted version we ran (6 October 2026); Fastino's 5 November blog gives 26.09B for what it serves now, a later change
Input
text, English
Endpoint
https://api.fastino.ai
Started
7 Nov 2026 14:50 UTC; last answer 7 Nov 2026 14:51 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Fastino GLiDE (site run 129); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "fastino/GLiDE",
    "--provider",
    "fastino",
    "--timeout",
    "300",
    "--concurrency",
    "4",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 4,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "fastino/GLiDE",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Fastino's GLiDE price",
    "date": "2026-10-08",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.15,
      "cache_write_input_tokens": 0.15,
      "cached_input_tokens": 0.15,
      "input_tokens": 0.15,
      "output_tokens": 0.0
    },
    "source": "https://docs.fastino.ai/pricing",
    "text": "$0.15 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "fastino",
  "questions": 1125,
  "request_seconds": 79.47210992104374,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "462c900fe86bde1d3e3e5ab98ef0670b3f9de26de41b2bdb0561e3c11896f399",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 129,
  "status": "completed",
  "timeout": 300.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 390257,
    "output_tokens": 0
  }
}
Prompt template
POST https://api.fastino.ai/v1/systemone
{
 "model": "fastino/GLiDE",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · glide_en_dontknow

Name
GLiDE API
Setup
Fastino GLiDE · decision model · with the I don't know option F
Model
fastino/GLiDE · provider fastino
GPU
—
Cost
$0.0636 for the run · provider price × reported usage
Answer time
run glide_en_dontknow · first answer 2026-11-07 14:51:43 UTC · last answer 2026-11-07 14:53:02 UTC · span 79.0 s, pauses included (not charged) · charged 79.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 4.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://docs.fastino.ai/inference/systemone · read 2026-11-09 · not disclosed by maker for the hosted version we ran (6 October 2026); Fastino's 5 November blog gives 26.09B for what it serves now, a later change
Input
text, English
Endpoint
https://api.fastino.ai
Started
7 Nov 2026 14:51 UTC; last answer 7 Nov 2026 14:53 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Fastino GLiDE (site run 129); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "fastino/GLiDE",
    "--provider",
    "fastino",
    "--timeout",
    "300",
    "--concurrency",
    "4",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 4,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "fastino/GLiDE",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Fastino's GLiDE price",
    "date": "2026-10-08",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.15,
      "cache_write_input_tokens": 0.15,
      "cached_input_tokens": 0.15,
      "input_tokens": 0.15,
      "output_tokens": 0.0
    },
    "source": "https://docs.fastino.ai/pricing",
    "text": "$0.15 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "fastino",
  "questions": 1125,
  "request_seconds": 79.42303269880358,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "219f7122c4dff9da628e67f7b00a3b99f8deed105f0b9b9cb96ce6740302e9ad",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 129,
  "status": "completed",
  "timeout": 300.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 424007,
    "output_tokens": 0
  }
}
Prompt template
POST https://api.fastino.ai/v1/systemone
{
 "model": "fastino/GLiDE",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · jev-1.14_en

Name
Jev 1.14
Setup
Jev 1.14.0 · decision model
Model
jev-1.14.0 · provider jev
GPU
—
Cost
$0.02885 for the run · provider price × reported usage
Answer time
run jev-1.14_en · first answer 2026-11-07 14:54:30 UTC · last answer 2026-11-07 14:57:44 UTC · span 194.0 s, pauses included (not charged) · charged 194.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://developers.cloudflare.com/ai/models/typesafe/jev-1.14.0/ · read 2026-11-09
Input
text, English
Endpoint
https://api.typesafe.ai
Started
7 Nov 2026 14:54 UTC; last answer 7 Nov 2026 14:57 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Jev 1.14.0 (site run 194); runner ffc307cb7; the run script reads the key from JEV_KEY_FILE; run_setup.sh takes JEV_API_KEY from the environment only
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "jev-1.14.0",
    "--provider",
    "jev",
    "--timeout",
    "30",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "jev-1.14.0",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "TypeSafe's System One price",
    "date": "2026-10-16",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.042,
      "cache_write_input_tokens": 0.042,
      "cached_input_tokens": 0.042,
      "input_tokens": 0.042,
      "output_tokens": 0.0
    },
    "source": "https://docs.typesafe.ai/models",
    "text": "$0.042 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "jev",
  "questions": 1125,
  "request_seconds": 190.23438284522854,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "6c1cb357d4d2683842cc0c7c4e3a7b1ea3af82ed0022a9d2ef074f85eaa288ad",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 194,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 686827,
    "output_tokens": 58500
  }
}
Prompt template
POST https://api.typesafe.ai/v1/systemone
{
 "model": "jev-1.14.0",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · d1-ultra_en

Name
d1-ultra
Setup
Liquid d1-ultra · decision model
Model
d1-ultra · provider liquid
GPU
—
Cost
$0.01422 for the run · provider price × reported usage
Answer time
run d1-ultra_en · first answer 2026-11-07 14:56:53 UTC · last answer 2026-11-07 15:01:24 UTC · span 271.0 s, pauses included (not charged) · charged 271.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) · source https://www.liquid.ai/blog/d1-ultra · read 2026-11-09
Input
text, English
Endpoint
https://api.liquid.ai/decisions
Started
7 Nov 2026 14:56 UTC; last answer 7 Nov 2026 15:01 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Liquid d1-ultra (site run 199); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "d1-ultra",
    "--provider",
    "liquid",
    "--timeout",
    "30",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "d1-ultra",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Liquid AI's d1-ultra price",
    "date": "2026-10-16",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.042,
      "cache_write_input_tokens": 0.042,
      "cached_input_tokens": 0.042,
      "input_tokens": 0.042,
      "output_tokens": 0.0
    },
    "source": "https://www.liquid.ai/blog/d1-ultra",
    "text": "$0.042 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "liquid",
  "questions": 1125,
  "request_seconds": 268.59644849714823,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "d42b78ebb61cb0d234c1bc94e85cd055369199898439cc791e529a9ae9cd91fa",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 199,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 338651,
    "output_tokens": 0
  }
}
Prompt template
POST https://api.liquid.ai/decisions/v1/systemone
{
 "model": "d1-ultra",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · jev-1.14_en_dontknow

Name
Jev 1.14
Setup
Jev 1.14.0 · decision model · with the I don't know option F
Model
jev-1.14.0 · provider jev
GPU
—
Cost
$0.03054 for the run · provider price × reported usage
Answer time
run jev-1.14_en_dontknow · first answer 2026-11-07 14:57:50 UTC · last answer 2026-11-07 15:01:05 UTC · span 195.0 s, pauses included (not charged) · charged 195.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://developers.cloudflare.com/ai/models/typesafe/jev-1.14.0/ · read 2026-11-09
Input
text, English
Endpoint
https://api.typesafe.ai
Started
7 Nov 2026 14:57 UTC; last answer 7 Nov 2026 15:01 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Jev 1.14.0 (site run 194); runner ffc307cb7; the run script reads the key from JEV_KEY_FILE; run_setup.sh takes JEV_API_KEY from the environment only
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "jev-1.14.0",
    "--provider",
    "jev",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "jev-1.14.0",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "TypeSafe's System One price",
    "date": "2026-10-16",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.042,
      "cache_write_input_tokens": 0.042,
      "cached_input_tokens": 0.042,
      "input_tokens": 0.042,
      "output_tokens": 0.0
    },
    "source": "https://docs.typesafe.ai/models",
    "text": "$0.042 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "jev",
  "questions": 1125,
  "request_seconds": 193.07237431663089,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "580683b08a8ed26aec3630f0d350d91982cb95c82199ee1f880951e2f4226d79",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 194,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 727057,
    "output_tokens": 66375
  }
}
Prompt template
POST https://api.typesafe.ai/v1/systemone
{
 "model": "jev-1.14.0",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · mammoth-1_en

Name
Mammoth-1
Setup
Mammoth-1 · decision model
Model
baselabs/mammoth-1 · provider baselabs
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.04319 for the run, list · maker's list price; RTX PRO 6000 Blackwell: $0.1695 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.0681 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.169. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run mammoth-1_en · first answer 2026-11-07 14:59:45 UTC · last answer 2026-11-07 15:03:50 UTC · span 245.0 s, pauses included (not charged) · charged 245.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1695 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/baselabs/mammoth-1 · read 2026-11-10 · the card states a 27B parameter decision model (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 27.36B (counted from the checkpoint: language model 26,895,998,464 + vision encoder 460,730,096; the card says a 27B decision model)
Input
text, English
Endpoint
http://127.0.0.1:19150
Started
7 Nov 2026 14:59 UTC; last answer 7 Nov 2026 15:03 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Mammoth-1 (site run 374); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "baselabs/mammoth-1",
    "--provider",
    "baselabs",
    "--timeout",
    "30",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "baselabs/mammoth-1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "maker",
    "basis": "Baseten's list price for the same model",
    "date": "2026-11-01",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.1,
      "cache_write_input_tokens": 0.1,
      "cached_input_tokens": 0.1,
      "input_tokens": 0.1,
      "output_tokens": 0.0
    },
    "source": "https://www.baseten.co/library/mammoth-1/",
    "text": "$0.10 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "baselabs",
  "questions": 1125,
  "request_seconds": 241.9871116298018,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "6a55936ee2d4cd673c007913a31f7360918e41cb56e84e7d68f5baded89da241",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 374,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 431882,
    "output_tokens": 1125
  }
}
Prompt template
POST http://127.0.0.1:19150/v1/systemone
{
 "model": "baselabs/mammoth-1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · d3-ultra_en

Name
d3-ultra
Setup
vLLM-SR d3-ultra · decision model
Model
vllm-sr/d3-ultra · provider d3-server
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.1231 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0494 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.123. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run d3-ultra_en · first answer 2026-11-07 15:00:40 UTC · last answer 2026-11-07 15:03:38 UTC · span 178.0 s, pauses included (not charged) · charged 178.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1231 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 26.09B dense · source https://huggingface.co/vllm-sr/d3-ultra · read 2026-11-09 · counted from the released checkpoint: backbone 26,085,330,160 (its 11 safetensors shards) + decision head (readout.safetensors) 1,305,600 = 26,086,635,760
Input
text, English
Endpoint
http://127.0.0.1:19130
Started
7 Nov 2026 15:00 UTC; last answer 7 Nov 2026 15:03 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup vLLM-SR d3-ultra (site run 269); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "vllm-sr/d3-ultra",
    "--provider",
    "d3-server",
    "--timeout",
    "60",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "vllm-sr/d3-ultra",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "47f5776e71db",
  "provider": "d3-server",
  "questions": 1125,
  "request_seconds": 176.07345170830376,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "4b2f23ca3126cd021b5b43da248971f54b2a3336b25cd606a6f5f5bb69255823",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 269,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 513065,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19130/v1/systemone
{
 "model": "vllm-sr/d3-ultra",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · d1-ultra_en_dontknow

Name
d1-ultra
Setup
Liquid d1-ultra · decision model · with the I don't know option F
Model
d1-ultra · provider liquid
GPU
—
Cost
$0.0155 for the run · provider price × reported usage
Answer time
run d1-ultra_en_dontknow · first answer 2026-11-07 15:01:36 UTC · last answer 2026-11-07 15:06:31 UTC · span 295.0 s, pauses included (not charged) · charged 295.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) · source https://www.liquid.ai/blog/d1-ultra · read 2026-11-09
Input
text, English
Endpoint
https://api.liquid.ai/decisions
Started
7 Nov 2026 15:01 UTC; last answer 7 Nov 2026 15:06 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Liquid d1-ultra (site run 199); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "d1-ultra",
    "--provider",
    "liquid",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "d1-ultra",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Liquid AI's d1-ultra price",
    "date": "2026-10-16",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.042,
      "cache_write_input_tokens": 0.042,
      "cached_input_tokens": 0.042,
      "input_tokens": 0.042,
      "output_tokens": 0.0
    },
    "source": "https://www.liquid.ai/blog/d1-ultra",
    "text": "$0.042 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "liquid",
  "questions": 1125,
  "request_seconds": 292.29563584085554,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "b6765f525d903e5c54d7f331d37ed9089250f0cc3e2ece68457bec476ad68669",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 199,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 368953,
    "output_tokens": 0
  }
}
Prompt template
POST https://api.liquid.ai/decisions/v1/systemone
{
 "model": "d1-ultra",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · subq-1.1_en

Name
SubQ Snap 1.1
Setup
SubQ Snap 1.1 · decision model
Model
subq-snap-1.1 · provider subq
GPU
—
Cost
$0.01678 for the run · provider price × reported usage
Answer time
run subq-1.1_en · first answer 2026-11-07 15:02:38 UTC · last answer 2026-11-07 15:09:55 UTC · span 437.0 s, pauses included (not charged) · charged 437.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://docs.subq.ai/using-the-api/models/ · read 2026-11-09
Input
text, English
Endpoint
https://api.subq.ai
Started
7 Nov 2026 15:02 UTC; last answer 7 Nov 2026 15:09 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup SubQ Snap 1.1 (site run 346); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "subq-snap-1.1",
    "--provider",
    "subq",
    "--timeout",
    "30",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "subq-snap-1.1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "SubQ's list price",
    "date": "2026-11-01",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.04,
      "cache_write_input_tokens": 0.04,
      "cached_input_tokens": 0.04,
      "input_tokens": 0.04,
      "output_tokens": 0.0
    },
    "source": "https://docs.subq.ai/using-the-api/models/",
    "text": "$0.04 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "subq",
  "questions": 1125,
  "request_seconds": 434.14323964575306,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "f1c69658e13e2b699ba74509f4581e3fb12404e16bfaf377574e1d7e4df635fb",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 346,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 419507,
    "output_tokens": 0
  }
}
Prompt template
POST https://api.subq.ai/v1/systemone
{
 "model": "subq-snap-1.1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · celeris_en

Name
celeris-1-decision
Setup
Celeris-1 Decision · decision model
Model
celeris-1-decision · provider celeris
GPU
—
Cost
$0.01733 for the run · provider price × reported usage
Answer time
run celeris_en · first answer 2026-11-07 15:02:46 UTC · last answer 2026-11-07 15:10:08 UTC · span 442.0 s, pauses included (not charged) · charged 442.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://docs.celeris.ai/decisions · read 2026-11-09
Input
text, English
Endpoint
https://inference.celeris.ai/celeris-1-decision
Started
7 Nov 2026 15:02 UTC; last answer 7 Nov 2026 15:10 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Celeris-1 Decision (site run 150); runner ffc307cb7; probabilities come rounded to two decimals
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "celeris-1-decision",
    "--provider",
    "celeris",
    "--timeout",
    "30",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "celeris-1-decision",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Celeris's decision model price",
    "date": "2026-10-10",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.04,
      "cache_write_input_tokens": 0.04,
      "cached_input_tokens": 0.04,
      "input_tokens": 0.04,
      "output_tokens": 0.0
    },
    "source": "https://docs.celeris.ai/pricing",
    "text": "$0.04 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "celeris",
  "questions": 1125,
  "request_seconds": 438.9106478684116,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "fedfd69e34d21b1311b29b162c93d01e92ab69e9f79c78b223b293bc53e1c7fd",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 150,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 433259,
    "output_tokens": 0
  }
}
Prompt template
POST https://inference.celeris.ai/celeris-1-decision/v1/systemone
{
 "model": "celeris-1-decision",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · mercury-decide_en

Name
Mercury Decide
Setup
Inception Mercury Decide · decision model
Model
inception/mercury-decide · provider openrouter
GPU
—
Cost
$0.008459 for the run · provider price × reported usage
Answer time
run mercury-decide_en · first answer 2026-11-07 15:03:22 UTC · last answer 2026-11-07 15:09:21 UTC · span 359.0 s, pauses included (not charged) · charged 359.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://www.inceptionlabs.ai/ · read 2026-11-09
Input
text, English
Endpoint
https://openrouter.ai/api
Started
7 Nov 2026 15:03 UTC; last answer 7 Nov 2026 15:09 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Inception Mercury Decide (site run 204); runner ffc307cb7; via OpenRouter; latency includes OpenRouter's time
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "inception/mercury-decide",
    "--provider",
    "openrouter",
    "--timeout",
    "30",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "inception/mercury-decide",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "OpenRouter's Mercury Decide price",
    "date": "2026-10-17",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.02,
      "cache_write_input_tokens": 0.02,
      "cached_input_tokens": 0.02,
      "input_tokens": 0.02,
      "output_tokens": 0.0
    },
    "source": "https://openrouter.ai/inception/mercury-decide",
    "text": "$0.02 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "openrouter",
  "questions": 1125,
  "request_seconds": 355.8979651177069,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "7293b9306693349e058e3d667495fd9faae5f3bf485ef05bd86c98dc5eb8afc3",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 204,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 422945,
    "output_tokens": 1125
  }
}
Prompt template
POST https://openrouter.ai/api/v1/systemone
{
 "model": "inception/mercury-decide",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · d3-ultra_en_dontknow

Name
d3-ultra
Setup
vLLM-SR d3-ultra · decision model · with the I don't know option F
Model
vllm-sr/d3-ultra · provider d3-server
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.1259 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0506 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.126. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run d3-ultra_en_dontknow · first answer 2026-11-07 15:03:45 UTC · last answer 2026-11-07 15:06:47 UTC · span 182.0 s, pauses included (not charged) · charged 182.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1259 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 26.09B dense · source https://huggingface.co/vllm-sr/d3-ultra · read 2026-11-09 · counted from the released checkpoint: backbone 26,085,330,160 (its 11 safetensors shards) + decision head (readout.safetensors) 1,305,600 = 26,086,635,760
Input
text, English
Endpoint
http://127.0.0.1:19130
Started
7 Nov 2026 15:03 UTC; last answer 7 Nov 2026 15:06 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup vLLM-SR d3-ultra (site run 269); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "vllm-sr/d3-ultra",
    "--provider",
    "d3-server",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "vllm-sr/d3-ultra",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "413d6775fc58",
  "provider": "d3-server",
  "questions": 1125,
  "request_seconds": 178.5205062504392,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "733cdff0ef056416f20f87d9f005a3b19e3a50d6aa785c07d0fab401e5e37f8b",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 269,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 572690,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19130/v1/systemone
{
 "model": "vllm-sr/d3-ultra",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · mammoth-1_en_dontknow

Name
Mammoth-1
Setup
Mammoth-1 · decision model · with the I don't know option F
Model
baselabs/mammoth-1 · provider baselabs
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.04679 for the run, list · maker's list price; RTX PRO 6000 Blackwell: $0.137 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.055 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.137. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-07 15:04:14–17:28:42 UTC
GPU time
run mammoth-1_en_dontknow · first answer 2026-11-07 15:03:57 UTC · last answer 2026-11-07 17:31:43 UTC · span 8866.0 s, pauses included (not charged) · charged 198.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1370 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/baselabs/mammoth-1 · read 2026-11-10 · the card states a 27B parameter decision model (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 27.36B (counted from the checkpoint: language model 26,895,998,464 + vision encoder 460,730,096; the card says a 27B decision model)
Input
text, English
Endpoint
http://127.0.0.1:19150
Started
7 Nov 2026 15:03 UTC; last answer 7 Nov 2026 17:31 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Mammoth-1 (site run 374); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "baselabs/mammoth-1",
    "--provider",
    "baselabs",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "baselabs/mammoth-1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "maker",
    "basis": "Baseten's list price for the same model",
    "date": "2026-11-01",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.1,
      "cache_write_input_tokens": 0.1,
      "cached_input_tokens": 0.1,
      "input_tokens": 0.1,
      "output_tokens": 0.0
    },
    "source": "https://www.baseten.co/library/mammoth-1/",
    "text": "$0.10 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "baselabs",
  "questions": 1125,
  "request_seconds": 188.99958430129027,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "240304eb024333779b2a84b2c8b76f05f66ac8c947fca940d69c0ad342a07e1e",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 374,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 467882,
    "output_tokens": 1125
  }
}
Prompt template
POST http://127.0.0.1:19150/v1/systemone
{
 "model": "baselabs/mammoth-1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · drex-max-1.1_en

Name
Drex Max v1.1
Setup
Drex Max 1.1 · decision model
Model
drex-max-v1.1 · provider nace
GPU
—
Cost
$0.02517 for the run · provider price × reported usage
Answer time
run drex-max-1.1_en · first answer 2026-11-07 15:07:01 UTC · last answer 2026-11-07 15:16:25 UTC · span 564.0 s, pauses included (not charged) · charged 564.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://console.nace.ai/docs/drex/reference/models · read 2026-11-09
Input
text, English
Endpoint
https://console.nace.ai
Started
7 Nov 2026 15:07 UTC; last answer 7 Nov 2026 15:16 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Drex Max 1.1 (site run 326); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "drex-max-v1.1",
    "--provider",
    "nace",
    "--timeout",
    "30",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "drex-max-v1.1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Nace.AI's list price (launch price $0.03 per 1M input tokens, 50% off, is what Nace bills)",
    "date": "2026-10-30",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.06,
      "cache_write_input_tokens": 0.06,
      "cached_input_tokens": 0.06,
      "input_tokens": 0.06,
      "output_tokens": 0.0
    },
    "source": "https://nace.ai/drex",
    "text": "$0.06 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "nace",
  "questions": 1125,
  "request_seconds": 561.0677336179651,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "9b3689c25a6a96cdcb90d7ac89045a9e03d406b4dd803a9e7096e8abba03af6b",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 326,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 419507,
    "output_tokens": 182669
  }
}
Prompt template
POST https://console.nace.ai/v1/systemone
{
 "model": "drex-max-v1.1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · mercury-decide_en_dontknow

Name
Mercury Decide
Setup
Inception Mercury Decide · decision model · with the I don't know option F
Model
inception/mercury-decide · provider openrouter
GPU
—
Cost
$0.009317 for the run · provider price × reported usage
Answer time
run mercury-decide_en_dontknow · first answer 2026-11-07 15:09:31 UTC · last answer 2026-11-07 15:15:32 UTC · span 361.0 s, pauses included (not charged) · charged 361.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://www.inceptionlabs.ai/ · read 2026-11-09
Input
text, English
Endpoint
https://openrouter.ai/api
Started
7 Nov 2026 15:09 UTC; last answer 7 Nov 2026 15:15 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Inception Mercury Decide (site run 204); runner ffc307cb7; via OpenRouter; latency includes OpenRouter's time
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "inception/mercury-decide",
    "--provider",
    "openrouter",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "inception/mercury-decide",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "OpenRouter's Mercury Decide price",
    "date": "2026-10-17",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.02,
      "cache_write_input_tokens": 0.02,
      "cached_input_tokens": 0.02,
      "input_tokens": 0.02,
      "output_tokens": 0.0
    },
    "source": "https://openrouter.ai/inception/mercury-decide",
    "text": "$0.02 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "openrouter",
  "questions": 1125,
  "request_seconds": 358.76765488041565,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "0a9c20ac4fe87894027ea77061f5b3239f83b570922cb81ba7138d612b3d2002",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 204,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 465860,
    "output_tokens": 1125
  }
}
Prompt template
POST https://openrouter.ai/api/v1/systemone
{
 "model": "inception/mercury-decide",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · subq-1.1_en_dontknow

Name
SubQ Snap 1.1
Setup
SubQ Snap 1.1 · decision model · with the I don't know option F
Model
subq-snap-1.1 · provider subq
GPU
—
Cost
$0.01813 for the run · provider price × reported usage
Answer time
run subq-1.1_en_dontknow · first answer 2026-11-07 15:10:06 UTC · last answer 2026-11-07 15:17:21 UTC · span 435.0 s, pauses included (not charged) · charged 435.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://docs.subq.ai/using-the-api/models/ · read 2026-11-09
Input
text, English
Endpoint
https://api.subq.ai
Started
7 Nov 2026 15:10 UTC; last answer 7 Nov 2026 15:17 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup SubQ Snap 1.1 (site run 346); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "subq-snap-1.1",
    "--provider",
    "subq",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "subq-snap-1.1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "SubQ's list price",
    "date": "2026-11-01",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.04,
      "cache_write_input_tokens": 0.04,
      "cached_input_tokens": 0.04,
      "input_tokens": 0.04,
      "output_tokens": 0.0
    },
    "source": "https://docs.subq.ai/using-the-api/models/",
    "text": "$0.04 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "subq",
  "questions": 1125,
  "request_seconds": 432.6764856518712,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "08310275970600337e96f59dcf481ed9926e16688fe00f3dd18accb81b113e0d",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 346,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 453257,
    "output_tokens": 0
  }
}
Prompt template
POST https://api.subq.ai/v1/systemone
{
 "model": "subq-snap-1.1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · celeris_en_dontknow

Name
celeris-1-decision
Setup
Celeris-1 Decision · decision model · with the I don't know option F
Model
celeris-1-decision · provider celeris
GPU
—
Cost
$0.01886 for the run · provider price × reported usage
Answer time
run celeris_en_dontknow · first answer 2026-11-07 15:10:19 UTC · last answer 2026-11-07 15:17:41 UTC · span 442.0 s, pauses included (not charged) · charged 442.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://docs.celeris.ai/decisions · read 2026-11-09
Input
text, English
Endpoint
https://inference.celeris.ai/celeris-1-decision
Started
7 Nov 2026 15:10 UTC; last answer 7 Nov 2026 15:17 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Celeris-1 Decision (site run 150); runner ffc307cb7; probabilities come rounded to two decimals
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "celeris-1-decision",
    "--provider",
    "celeris",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "celeris-1-decision",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Celeris's decision model price",
    "date": "2026-10-10",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.04,
      "cache_write_input_tokens": 0.04,
      "cached_input_tokens": 0.04,
      "input_tokens": 0.04,
      "output_tokens": 0.0
    },
    "source": "https://docs.celeris.ai/pricing",
    "text": "$0.04 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "celeris",
  "questions": 1125,
  "request_seconds": 438.9722986113047,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "b8568616b3b6a313a33fa6955d9a9b73fb621aaa3af40eb8021b576cc5b780c6",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 150,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 471509,
    "output_tokens": 0
  }
}
Prompt template
POST https://inference.celeris.ai/celeris-1-decision/v1/systemone
{
 "model": "celeris-1-decision",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · drex-max-1.1_en_dontknow

Name
Drex Max v1.1
Setup
Drex Max 1.1 · decision model · with the I don't know option F
Model
drex-max-v1.1 · provider nace
GPU
—
Cost
$0.0272 for the run · provider price × reported usage
Answer time
run drex-max-1.1_en_dontknow · first answer 2026-11-07 15:16:36 UTC · last answer 2026-11-07 15:26:26 UTC · span 590.0 s, pauses included (not charged) · charged 590.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://console.nace.ai/docs/drex/reference/models · read 2026-11-09
Input
text, English
Endpoint
https://console.nace.ai
Started
7 Nov 2026 15:16 UTC; last answer 7 Nov 2026 15:26 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Drex Max 1.1 (site run 326); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "drex-max-v1.1",
    "--provider",
    "nace",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "drex-max-v1.1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Nace.AI's list price (launch price $0.03 per 1M input tokens, 50% off, is what Nace bills)",
    "date": "2026-10-30",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.06,
      "cache_write_input_tokens": 0.06,
      "cached_input_tokens": 0.06,
      "input_tokens": 0.06,
      "output_tokens": 0.0
    },
    "source": "https://nace.ai/drex",
    "text": "$0.06 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "nace",
  "questions": 1125,
  "request_seconds": 551.6703608884709,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "37688769bf4305a4ed198badf55e71237e2a680a8b30eff4183144247378259b",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 326,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 453257,
    "output_tokens": 210115
  }
}
Prompt template
POST https://console.nace.ai/v1/systemone
{
 "model": "drex-max-v1.1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · jev-1.13_en

Name
Jev 1.13
Setup
Jev 1.13.0 · decision model
Model
jev-1.13.0 · provider jev
GPU
—
Cost
$0.02885 for the run · provider price × reported usage
Answer time
run jev-1.13_en · first answer 2026-11-07 15:26:50 UTC · last answer 2026-11-07 15:30:09 UTC · span 199.0 s, pauses included (not charged) · charged 199.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://typesafe.ai/blog/jev-1-14 · read 2026-11-09
Input
text, English
Endpoint
https://api.typesafe.ai
Started
7 Nov 2026 15:26 UTC; last answer 7 Nov 2026 15:30 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Jev 1.13.0 (site run 49); runner ffc307cb7; the run script reads the key from JEV_KEY_FILE; run_setup.sh takes JEV_API_KEY from the environment only
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "jev-1.13.0",
    "--provider",
    "jev",
    "--timeout",
    "30",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "jev-1.13.0",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "TypeSafe's System One price",
    "date": "2026-10-08",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.042,
      "cache_write_input_tokens": 0.042,
      "cached_input_tokens": 0.042,
      "input_tokens": 0.042,
      "output_tokens": 0.0
    },
    "source": "https://typesafe.ai/blog/introducing-system-one-models-and-jev",
    "text": "$0.042 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "jev",
  "questions": 1125,
  "request_seconds": 195.03095041250344,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "d6d290019c5747506a12a42385a923bb5c678093317116b5842ae675594410ff",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 49,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 686827,
    "output_tokens": 58500
  }
}
Prompt template
POST https://api.typesafe.ai/v1/systemone
{
 "model": "jev-1.13.0",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · jev-1.13_en_dontknow

Name
Jev 1.13
Setup
Jev 1.13.0 · decision model · with the I don't know option F
Model
jev-1.13.0 · provider jev
GPU
—
Cost
$0.03054 for the run · provider price × reported usage
Answer time
run jev-1.13_en_dontknow · first answer 2026-11-07 15:30:16 UTC · last answer 2026-11-07 15:33:35 UTC · span 199.0 s, pauses included (not charged) · charged 199.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://typesafe.ai/blog/jev-1-14 · read 2026-11-09
Input
text, English
Endpoint
https://api.typesafe.ai
Started
7 Nov 2026 15:30 UTC; last answer 7 Nov 2026 15:33 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Jev 1.13.0 (site run 49); runner ffc307cb7; the run script reads the key from JEV_KEY_FILE; run_setup.sh takes JEV_API_KEY from the environment only
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "jev-1.13.0",
    "--provider",
    "jev",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "jev-1.13.0",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "TypeSafe's System One price",
    "date": "2026-10-08",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.042,
      "cache_write_input_tokens": 0.042,
      "cached_input_tokens": 0.042,
      "input_tokens": 0.042,
      "output_tokens": 0.0
    },
    "source": "https://typesafe.ai/blog/introducing-system-one-models-and-jev",
    "text": "$0.042 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "jev",
  "questions": 1125,
  "request_seconds": 195.99619163922034,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "d47dc357962aa4639f9a26e38815b186200c8fa42385f597f81996b383bd73ab",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 49,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 727057,
    "output_tokens": 66375
  }
}
Prompt template
POST https://api.typesafe.ai/v1/systemone
{
 "model": "jev-1.13.0",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · d1.1_en

Name
d1.1
Setup
Liquid d1.1 · decision model
Model
d1.1 · provider liquid
GPU
—
Cost
$0.006773 for the run · provider price × reported usage
Answer time
run d1.1_en · first answer 2026-11-07 15:38:09 UTC · last answer 2026-11-07 15:42:02 UTC · span 233.0 s, pauses included (not charged) · charged 233.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://docs.liquid.ai/lfm/models/decision-models · read 2026-11-09
Input
text, English
Endpoint
https://api.liquid.ai/decisions
Started
7 Nov 2026 15:38 UTC; last answer 7 Nov 2026 15:42 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Liquid d1.1 (site run 254); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "d1.1",
    "--provider",
    "liquid",
    "--timeout",
    "30",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "d1.1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Liquid AI's price for d1.1, as its API bills it",
    "date": "2026-10-24",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.02,
      "cache_write_input_tokens": 0.02,
      "cached_input_tokens": 0.02,
      "input_tokens": 0.02,
      "output_tokens": 0.0
    },
    "source": "https://www.liquid.ai/pricing",
    "text": "$0.02 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "liquid",
  "questions": 1125,
  "request_seconds": 228.8184675835073,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "71bde2cde0871c05943d4a98576f6bfe539487e0c1ab0e59e7bc93f429bdd835",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 254,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 338651,
    "output_tokens": 0
  }
}
Prompt template
POST https://api.liquid.ai/decisions/v1/systemone
{
 "model": "d1.1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · d1.1_en_dontknow

Name
d1.1
Setup
Liquid d1.1 · decision model · with the I don't know option F
Model
d1.1 · provider liquid
GPU
—
Cost
$0.007379 for the run · provider price × reported usage
Answer time
run d1.1_en_dontknow · first answer 2026-11-07 15:42:10 UTC · last answer 2026-11-07 15:46:08 UTC · span 238.0 s, pauses included (not charged) · charged 238.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://docs.liquid.ai/lfm/models/decision-models · read 2026-11-09
Input
text, English
Endpoint
https://api.liquid.ai/decisions
Started
7 Nov 2026 15:42 UTC; last answer 7 Nov 2026 15:46 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Liquid d1.1 (site run 254); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "d1.1",
    "--provider",
    "liquid",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "d1.1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Liquid AI's price for d1.1, as its API bills it",
    "date": "2026-10-24",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.02,
      "cache_write_input_tokens": 0.02,
      "cached_input_tokens": 0.02,
      "input_tokens": 0.02,
      "output_tokens": 0.0
    },
    "source": "https://www.liquid.ai/pricing",
    "text": "$0.02 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "liquid",
  "questions": 1125,
  "request_seconds": 235.26506372028962,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "00f7b367b886ea4864976e4e3a717157c2993cb1b157b02c2722cb562d7e1973",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 254,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 368953,
    "output_tokens": 0
  }
}
Prompt template
POST https://api.liquid.ai/decisions/v1/systemone
{
 "model": "d1.1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · ms-decision-1_en

Name
Microsoft-Decision-1
Setup
Microsoft-Decision-1 · decision model
Model
microsoft/microsoft-decision-1 · provider openrouter
GPU
—
Cost
$0.0151 for the run · provider price × reported usage
Answer time
run ms-decision-1_en · first answer 2026-11-07 15:42:31 UTC · last answer 2026-11-07 15:48:39 UTC · span 368.0 s, pauses included (not charged) · charged 368.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://openrouter.ai/microsoft/microsoft-decision-1 · read 2026-11-09 · not disclosed by maker; Microsoft names only its base model (post-trained from Qwen3.5-9B) and says its weights are updated continually
Input
text, English
Endpoint
https://openrouter.ai/api
Started
7 Nov 2026 15:42 UTC; last answer 7 Nov 2026 15:48 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Microsoft-Decision-1 (site run 205); runner ffc307cb7; via OpenRouter; latency includes OpenRouter's time
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "microsoft/microsoft-decision-1",
    "--provider",
    "openrouter",
    "--timeout",
    "30",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "microsoft/microsoft-decision-1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "OpenRouter's Microsoft-Decision-1 price",
    "date": "2026-10-17",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.042,
      "cache_write_input_tokens": 0.042,
      "cached_input_tokens": 0.042,
      "input_tokens": 0.042,
      "output_tokens": 0.0
    },
    "source": "https://openrouter.ai/microsoft/microsoft-decision-1",
    "text": "$0.042 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "openrouter",
  "questions": 1125,
  "request_seconds": 364.40041960857343,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "bbbd9a7cee64a75d302a18d82bd8c06b4b7d3bcb0e25c7c8b50654f8a2970b05",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 205,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 359548,
    "output_tokens": 1125
  }
}
Prompt template
POST https://openrouter.ai/api/v1/systemone
{
 "model": "microsoft/microsoft-decision-1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · subq-preview_en

Name
SubQ Snap Preview
Setup
SubQ Snap Preview · decision model
Model
subq-snap-preview · provider subq
GPU
—
Cost
$0.02098 for the run · provider price × reported usage
Answer time
run subq-preview_en · first answer 2026-11-07 15:48:12 UTC · last answer 2026-11-07 15:55:27 UTC · span 435.0 s, pauses included (not charged) · charged 435.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://docs.subq.ai/using-the-api/models/ · read 2026-11-09
Input
text, English
Endpoint
https://api.subq.ai
Started
7 Nov 2026 15:48 UTC; last answer 7 Nov 2026 15:55 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup SubQ Snap Preview (site run 177); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "subq-snap-preview",
    "--provider",
    "subq",
    "--timeout",
    "30",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "subq-snap-preview",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "SubQ's Decisions API price",
    "date": "2026-10-13",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.05,
      "cache_write_input_tokens": 0.05,
      "cached_input_tokens": 0.05,
      "input_tokens": 0.05,
      "output_tokens": 0.0
    },
    "source": "https://docs.subq.ai/using-the-api/models/",
    "text": "$0.05 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "subq",
  "questions": 1125,
  "request_seconds": 430.7852070701774,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "55d4bbae982a2e55265ced9f8edc1c061da9567324d5365d2f9adc4d19d3c38d",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 177,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 419507,
    "output_tokens": 0
  }
}
Prompt template
POST https://api.subq.ai/v1/systemone
{
 "model": "subq-snap-preview",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · ms-decision-1_en_dontknow

Name
Microsoft-Decision-1
Setup
Microsoft-Decision-1 · decision model · with the I don't know option F
Model
microsoft/microsoft-decision-1 · provider openrouter
GPU
—
Cost
$0.01638 for the run · provider price × reported usage
Answer time
run ms-decision-1_en_dontknow · first answer 2026-11-07 15:48:48 UTC · last answer 2026-11-07 15:55:00 UTC · span 372.0 s, pauses included (not charged) · charged 372.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://openrouter.ai/microsoft/microsoft-decision-1 · read 2026-11-09 · not disclosed by maker; Microsoft names only its base model (post-trained from Qwen3.5-9B) and says its weights are updated continually
Input
text, English
Endpoint
https://openrouter.ai/api
Started
7 Nov 2026 15:48 UTC; last answer 7 Nov 2026 15:55 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Microsoft-Decision-1 (site run 205); runner ffc307cb7; via OpenRouter; latency includes OpenRouter's time
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "microsoft/microsoft-decision-1",
    "--provider",
    "openrouter",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "microsoft/microsoft-decision-1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "OpenRouter's Microsoft-Decision-1 price",
    "date": "2026-10-17",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.042,
      "cache_write_input_tokens": 0.042,
      "cached_input_tokens": 0.042,
      "input_tokens": 0.042,
      "output_tokens": 0.0
    },
    "source": "https://openrouter.ai/microsoft/microsoft-decision-1",
    "text": "$0.042 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "openrouter",
  "questions": 1125,
  "request_seconds": 366.8744044470368,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "0e2df352d40ab35d9a2007f34b4d42de4fed04c34b750e7f8d67ff7107916d38",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 205,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 389946,
    "output_tokens": 1125
  }
}
Prompt template
POST https://openrouter.ai/api/v1/systemone
{
 "model": "microsoft/microsoft-decision-1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · subq-preview_en_dontknow

Name
SubQ Snap Preview
Setup
SubQ Snap Preview · decision model · with the I don't know option F
Model
subq-snap-preview · provider subq
GPU
—
Cost
$0.02266 for the run · provider price × reported usage
Answer time
run subq-preview_en_dontknow · first answer 2026-11-07 15:55:38 UTC · last answer 2026-11-07 16:02:57 UTC · span 439.0 s, pauses included (not charged) · charged 439.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://docs.subq.ai/using-the-api/models/ · read 2026-11-09
Input
text, English
Endpoint
https://api.subq.ai
Started
7 Nov 2026 15:55 UTC; last answer 7 Nov 2026 16:02 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup SubQ Snap Preview (site run 177); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "subq-snap-preview",
    "--provider",
    "subq",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "subq-snap-preview",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "SubQ's Decisions API price",
    "date": "2026-10-13",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.05,
      "cache_write_input_tokens": 0.05,
      "cached_input_tokens": 0.05,
      "input_tokens": 0.05,
      "output_tokens": 0.0
    },
    "source": "https://docs.subq.ai/using-the-api/models/",
    "text": "$0.05 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "subq",
  "questions": 1125,
  "request_seconds": 433.4518370050937,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "5ce17213cd3318549351861972efae7fef3066a72cc4c847a1ee70f6ec96a45a",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 177,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 453257,
    "output_tokens": 0
  }
}
Prompt template
POST https://api.subq.ai/v1/systemone
{
 "model": "subq-snap-preview",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · d1_en

Name
d1
Setup
Liquid d1 · decision model
Model
d1 · provider liquid
GPU
—
Cost
$0.006773 for the run · provider price × reported usage
Answer time
run d1_en · first answer 2026-11-07 16:02:42 UTC · last answer 2026-11-07 16:06:48 UTC · span 246.0 s, pauses included (not charged) · charged 246.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://www.liquid.ai/blog/d1-decision-model · read 2026-11-09
Input
text, English
Endpoint
https://api.liquid.ai/decisions
Started
7 Nov 2026 16:02 UTC; last answer 7 Nov 2026 16:06 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Liquid d1 (site run 52); runner ffc307cb7; published as d1:free (free tier), which Liquid's API no longer serves (HTTP 404, 2 November 2026); run_english.sh's d1 lane asks for d1 with the same settings, priced at Liquid's d1 list price
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "d1",
    "--provider",
    "liquid",
    "--timeout",
    "30",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "d1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Liquid's price for d1, as its API bills it",
    "date": "2026-11-02",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.02,
      "cache_write_input_tokens": 0.02,
      "cached_input_tokens": 0.02,
      "input_tokens": 0.02,
      "output_tokens": 0.0
    },
    "source": "https://www.liquid.ai/pricing",
    "text": "$0.02 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "liquid",
  "questions": 1125,
  "request_seconds": 243.31571873032954,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "ca2b50931b9eadcff3deb42ca7483ec87ccd8ded4a645f4aafd6ff5962bcccac",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 52,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 338651,
    "output_tokens": 0
  }
}
Prompt template
POST https://api.liquid.ai/decisions/v1/systemone
{
 "model": "d1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · drex-2.0_en

Name
Drex 2.0
Setup
Drex 2.0 · decision model
Model
drex-v2.0 · provider nace
GPU
—
Cost
$0.02072 for the run · provider price × reported usage
Answer time
run drex-2.0_en · first answer 2026-11-07 16:03:32 UTC · last answer 2026-11-07 16:12:56 UTC · span 564.0 s, pauses included (not charged) · charged 564.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 35B MoE · 3B active · source https://huggingface.co/nace-ai/drex-v2.0 · read 2026-11-10 · the card states 35B parameters in total, about 3B active per token (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 35.11B / 3B active (counted from the released checkpoint (nace-ai/drex-v2.0 @ e646674): 35,107,181,936, including a vision tower of 446,571,248 that the card says is not loaded for text; mixture of experts, about 3B active per token (the card: 35B total))
Input
text, English
Endpoint
https://console.nace.ai
Started
7 Nov 2026 16:03 UTC; last answer 7 Nov 2026 16:12 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Drex 2.0 (site run 334); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "drex-v2.0",
    "--provider",
    "nace",
    "--timeout",
    "30",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "drex-v2.0",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Nace.AI's list price",
    "date": "2026-10-30",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.06,
      "cache_write_input_tokens": 0.06,
      "cached_input_tokens": 0.06,
      "input_tokens": 0.06,
      "output_tokens": 0.0
    },
    "source": "https://console.nace.ai/docs/drex/reference/pricing",
    "text": "$0.06 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "nace",
  "questions": 1125,
  "request_seconds": 559.9449750214117,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "fac2afaecf0f1eb1a9e3934c9153f0bc5bbeeef9f49f0135179796b134c90a21",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 334,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 345257,
    "output_tokens": 94881
  }
}
Prompt template
POST https://console.nace.ai/v1/systemone
{
 "model": "drex-v2.0",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · d1_en_dontknow

Name
d1
Setup
Liquid d1 · decision model · with the I don't know option F
Model
d1 · provider liquid
GPU
—
Cost
$0.007379 for the run · provider price × reported usage
Answer time
run d1_en_dontknow · first answer 2026-11-07 16:06:56 UTC · last answer 2026-11-07 16:11:12 UTC · span 256.0 s, pauses included (not charged) · charged 256.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://www.liquid.ai/blog/d1-decision-model · read 2026-11-09
Input
text, English
Endpoint
https://api.liquid.ai/decisions
Started
7 Nov 2026 16:06 UTC; last answer 7 Nov 2026 16:11 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Liquid d1 (site run 52); runner ffc307cb7; published as d1:free (free tier), which Liquid's API no longer serves (HTTP 404, 2 November 2026); run_english.sh's d1 lane asks for d1 with the same settings, priced at Liquid's d1 list price
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "d1",
    "--provider",
    "liquid",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "d1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Liquid's price for d1, as its API bills it",
    "date": "2026-11-02",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.02,
      "cache_write_input_tokens": 0.02,
      "cached_input_tokens": 0.02,
      "input_tokens": 0.02,
      "output_tokens": 0.0
    },
    "source": "https://www.liquid.ai/pricing",
    "text": "$0.02 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "liquid",
  "questions": 1125,
  "request_seconds": 250.07065323577262,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "595657ed0bd7719aa6abf286e21edde57a39b5819b41f48e00028c047ff613ef",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 52,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 368953,
    "output_tokens": 0
  }
}
Prompt template
POST https://api.liquid.ai/decisions/v1/systemone
{
 "model": "d1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · drex-2.0_en_dontknow

Name
Drex 2.0
Setup
Drex 2.0 · decision model · with the I don't know option F
Model
drex-v2.0 · provider nace
GPU
—
Cost
$0.02267 for the run · provider price × reported usage
Answer time
run drex-2.0_en_dontknow · first answer 2026-11-07 16:13:09 UTC · last answer 2026-11-07 16:22:35 UTC · span 566.0 s, pauses included (not charged) · charged 566.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 35B MoE · 3B active · source https://huggingface.co/nace-ai/drex-v2.0 · read 2026-11-10 · the card states 35B parameters in total, about 3B active per token (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 35.11B / 3B active (counted from the released checkpoint (nace-ai/drex-v2.0 @ e646674): 35,107,181,936, including a vision tower of 446,571,248 that the card says is not loaded for text; mixture of experts, about 3B active per token (the card: 35B total))
Input
text, English
Endpoint
https://console.nace.ai
Started
7 Nov 2026 16:13 UTC; last answer 7 Nov 2026 16:22 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Drex 2.0 (site run 334); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "drex-v2.0",
    "--provider",
    "nace",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "drex-v2.0",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Nace.AI's list price",
    "date": "2026-10-30",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.06,
      "cache_write_input_tokens": 0.06,
      "cached_input_tokens": 0.06,
      "input_tokens": 0.06,
      "output_tokens": 0.0
    },
    "source": "https://console.nace.ai/docs/drex/reference/pricing",
    "text": "$0.06 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "nace",
  "questions": 1125,
  "request_seconds": 562.308073193999,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "89a7b8b08a0b8e36fe7bc5957931c37728705866499235f9704ee6e8b149ec58",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 334,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 377882,
    "output_tokens": 107168
  }
}
Prompt template
POST https://console.nace.ai/v1/systemone
{
 "model": "drex-v2.0",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · deepseek-v4-flash_en_t0

Name
DeepSeek-V4-Flash chat
Setup
deepseek-v4-flash-dspark · chat model
Model
deepseek-v4-flash-dspark
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.3659 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 1.72 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.366 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run deepseek-v4-flash_en_t0 · first answer 2026-11-07 17:29:07 UTC · last answer 2026-11-07 19:12:17 UTC · span 6190.0 s, pauses included (not charged) · charged 6190.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.3659 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 2.0 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 3 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 285B MoE · 13B active · source https://fe-static.deepseek.com/chat/transparency/deepseek-V4-model-card-EN.pdf · read 2026-11-09 · DeepSeek's count for DeepSeek-V4-Flash (285B, 13B activated per token); the DeepSeek-V4-Flash-Vision-Exp checkpoint served here states no count of its own and adds a vision module
Input
text, English
Endpoint
internal address on a shared Spark host, path /v1/chat/completions
Started
7 Nov 2026 17:01 UTC; last answer 7 Nov 2026 19:12 UTC
Progress
317 answered of 1,125 asked; 0 request errors; in progress
Notes
DeepSeek-V4-Flash (shared vLLM service on two shared Spark hosts, not ours), thinking off (server default), temperature 0, English translations | INCIDENT 2026-11-07 17:01-17:30 UTC: after the reboot the runner could not store answers (responses had a new probs_json column; run_eval.py inserted without column names), so the answers generated in that window were discarded; not recoverable (vLLM logs keep no reply text). the shared Spark host's vLLM log: 362 completed chat requests from our GPU host in 17:00-17:30, an upper bound since other users of our GPU host share that address. Restarted 17:30 with the same settings; every stored answer is from that restart. | Logic: a 300-question random subset stratified by topic (work/deepseek_logic_subset.txt, seed 0: numerical 140, verbal 125, formal 35), Umit #2678; earlier answers outside it stay stored.
Parameters
{
  "max_tokens": 8192,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · reax13_en

Name
REAX 1.3
Setup
REAX 1.3 · decision model
Model
reax13 · provider reax
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.1805 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0725 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.181. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run reax13_en · first answer 2026-11-07 17:02:40 UTC · last answer 2026-11-07 17:07:01 UTC · span 261.0 s, pauses included (not charged) · charged 261.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1805 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 57.36B dense · source https://huggingface.co/decisionmodels/REAX-1.3 · read 2026-11-09 · counted from the checkpoint's two parts (57,359,724,636); the card states no count; Fastino's blog says 57.36B
Input
text, English
Endpoint
http://127.0.0.1:19100
Started
7 Nov 2026 17:02 UTC; last answer 7 Nov 2026 17:07 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup REAX 1.3 (site run 827); runner ffc307cb7; Docker with the model card's protections; the published run dropped no-new-privileges (an approved exception): servers.sh refuses to start it without the protections
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "reax13",
    "--provider",
    "reax",
    "--timeout",
    "1800",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "reax13",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "47f5776e71db",
  "provider": "reax",
  "questions": 1125,
  "request_seconds": 249.48370815399946,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "362085311e0aa8337b40e3caf798098e818c0e287f8bd4960d88078a6072814b",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 827,
  "status": "completed",
  "timeout": 1800.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 0,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19100/v1/systemone
{
 "model": "reax13",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · reax13_en_dontknow

Name
REAX 1.3
Setup
REAX 1.3 · decision model · with the I don't know option F
Model
reax13 · provider reax
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.2165 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0869 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.216. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run reax13_en_dontknow · first answer 2026-11-07 17:07:09 UTC · last answer 2026-11-07 17:12:22 UTC · span 313.0 s, pauses included (not charged) · charged 313.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.2165 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 57.36B dense · source https://huggingface.co/decisionmodels/REAX-1.3 · read 2026-11-09 · counted from the checkpoint's two parts (57,359,724,636); the card states no count; Fastino's blog says 57.36B
Input
text, English
Endpoint
http://127.0.0.1:19100
Started
7 Nov 2026 17:07 UTC; last answer 7 Nov 2026 17:12 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup REAX 1.3 (site run 827); runner ffc307cb7; Docker with the model card's protections; the published run dropped no-new-privileges (an approved exception): servers.sh refuses to start it without the protections
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "reax13",
    "--provider",
    "reax",
    "--timeout",
    "1800",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "reax13",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "413d6775fc58",
  "provider": "reax",
  "questions": 1125,
  "request_seconds": 307.11514147099905,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "5d6eb60e98b25591873a119f87868f39b83e456ab0e7c96568ac972a44d2e361",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 827,
  "status": "completed",
  "timeout": 1800.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 0,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19100/v1/systemone
{
 "model": "reax13",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · decider-v1.2-web_en

Name
Perplexity Decider 27B v1.2 API · with web
Setup
Perplexity Decider 27B v1.2 (API, with web) · decision model
Model
pplx-decider-v1.2 · provider perplexity
GPU
—
Cost
$1.158 for the run · provider price × reported usage
Answer time
run decider-v1.2-web_en · first answer 2026-11-07 17:21:42 UTC · last answer 2026-11-07 17:32:09 UTC · span 627.0 s, pauses included (not charged) · charged 627.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/perplexity-ai/pplx-decider-v1.2 · read 2026-11-10 · the official model name: 27B (pplx-decider 27B v1.2; the card states no count of its own) (turkce #4263: the maker's stated total is shown); counted from the released files: 26.09B (counted from the released checkpoint: backbone 26,085,330,160 (its 11 safetensors shards) + decision head (readout.safetensors) 1,305,600 = 26,086,635,760; the API setups use the same count)
Input
text, English
Endpoint
https://api.perplexity.ai
Started
7 Nov 2026 17:21 UTC; last answer 7 Nov 2026 17:32 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Perplexity Decider 27B v1.2 (API, with web) (site run 640); runner ffc307cb7; web grounding: one Perplexity search per question ($0.001 each)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "pplx-decider-v1.2",
    "--provider",
    "perplexity",
    "--timeout",
    "30",
    "--perplexity-mode",
    "web_grounding",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "pplx-decider-v1.2",
  "perplexity_mode": "web_grounding",
  "price": {
    "applies_to": "billed",
    "basis": "Perplexity's Decisions API price with web grounding",
    "date": "2026-11-05",
    "extra_input_per_request": 0,
    "per_request": 0.001,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.02,
      "cache_write_input_tokens": 0.02,
      "cached_input_tokens": 0.02,
      "input_tokens": 0.02,
      "output_tokens": 0.0
    },
    "source": "https://docs.perplexity.ai/docs/getting-started/pricing",
    "text": "$0.02 input, output free per 1M tokens, plus $0.001 per request (one web search)"
  },
  "prompt_version": "47f5776e71db",
  "provider": "perplexity",
  "questions": 1125,
  "request_seconds": 618.0730732509935,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "cf3770cb9288b573b4c1aea137fbedebb3bacaf7eadbea8479c5e1776b7c45b0",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 640,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 1663650,
    "output_tokens": 1125
  }
}
Prompt template
POST https://api.perplexity.ai/v1/decisions
{
 "model": "pplx-decider-v1.2",
 "mode": "web_grounding",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · decider-v1.2-web_en_dontknow

Name
Perplexity Decider 27B v1.2 API · with web
Setup
Perplexity Decider 27B v1.2 (API, with web) · decision model · with the I don't know option F
Model
pplx-decider-v1.2 · provider perplexity
GPU
—
Cost
$1.153 for the run · provider price × reported usage
Answer time
run decider-v1.2-web_en_dontknow · first answer 2026-11-07 17:32:24 UTC · last answer 2026-11-07 17:43:47 UTC · span 683.0 s, pauses included (not charged) · charged 683.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 0.8 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/perplexity-ai/pplx-decider-v1.2 · read 2026-11-10 · the official model name: 27B (pplx-decider 27B v1.2; the card states no count of its own) (turkce #4263: the maker's stated total is shown); counted from the released files: 26.09B (counted from the released checkpoint: backbone 26,085,330,160 (its 11 safetensors shards) + decision head (readout.safetensors) 1,305,600 = 26,086,635,760; the API setups use the same count)
Input
text, English
Endpoint
https://api.perplexity.ai
Started
7 Nov 2026 17:32 UTC; last answer 7 Nov 2026 17:43 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Perplexity Decider 27B v1.2 (API, with web) (site run 640); runner ffc307cb7; web grounding: one Perplexity search per question ($0.001 each)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "pplx-decider-v1.2",
    "--provider",
    "perplexity",
    "--timeout",
    "30",
    "--perplexity-mode",
    "web_grounding",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "pplx-decider-v1.2",
  "perplexity_mode": "web_grounding",
  "price": {
    "applies_to": "billed",
    "basis": "Perplexity's Decisions API price with web grounding",
    "date": "2026-11-05",
    "extra_input_per_request": 0,
    "per_request": 0.001,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.02,
      "cache_write_input_tokens": 0.02,
      "cached_input_tokens": 0.02,
      "input_tokens": 0.02,
      "output_tokens": 0.0
    },
    "source": "https://docs.perplexity.ai/docs/getting-started/pricing",
    "text": "$0.02 input, output free per 1M tokens, plus $0.001 per request (one web search)"
  },
  "prompt_version": "413d6775fc58",
  "provider": "perplexity",
  "questions": 1125,
  "request_seconds": 566.5075323920066,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "2067aaed809f35ea96d79c925a8a72631ef79a92947e569be011354189cec1f6",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 640,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 1410565,
    "output_tokens": 1125
  }
}
Prompt template
POST https://api.perplexity.ai/v1/decisions
{
 "model": "pplx-decider-v1.2",
 "mode": "web_grounding",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · d3_en

Name
d3
Setup
vLLM-SR d3 · decision model
Model
vllm-sr/d3 · provider vllm-sr
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.06433 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0258 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.064. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run d3_en · first answer 2026-11-07 17:44:23 UTC · last answer 2026-11-07 17:45:56 UTC · span 93.0 s, pauses included (not charged) · charged 93.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0643 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 26.09B dense · source https://huggingface.co/vllm-sr/d3 · read 2026-11-09 · counted from the released checkpoint: backbone 26,085,330,160 (its 11 safetensors shards) + decision head (readout.safetensors) 1,305,600 = 26,086,635,760
Input
text, English
Endpoint
http://127.0.0.1:19120
Started
7 Nov 2026 17:44 UTC; last answer 7 Nov 2026 17:45 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup vLLM-SR d3 (site run 259); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "vllm-sr/d3",
    "--provider",
    "vllm-sr",
    "--timeout",
    "30",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "vllm-sr/d3",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "47f5776e71db",
  "provider": "vllm-sr",
  "questions": 1125,
  "request_seconds": 89.77727661399695,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "a5f04a7c50d89f6db10e30cb1751d77588c8de1ecbc3cef9f825705f6b6f8edf",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 259,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 428507,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19120/v1/systemone
{
 "model": "vllm-sr/d3",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · d3_en_dontknow

Name
d3
Setup
vLLM-SR d3 · decision model · with the I don't know option F
Model
vllm-sr/d3 · provider vllm-sr
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.06917 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0278 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.069. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run d3_en_dontknow · first answer 2026-11-07 17:46:00 UTC · last answer 2026-11-07 17:47:40 UTC · span 100.0 s, pauses included (not charged) · charged 100.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0692 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 26.09B dense · source https://huggingface.co/vllm-sr/d3 · read 2026-11-09 · counted from the released checkpoint: backbone 26,085,330,160 (its 11 safetensors shards) + decision head (readout.safetensors) 1,305,600 = 26,086,635,760
Input
text, English
Endpoint
http://127.0.0.1:19120
Started
7 Nov 2026 17:46 UTC; last answer 7 Nov 2026 17:47 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup vLLM-SR d3 (site run 259); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "vllm-sr/d3",
    "--provider",
    "vllm-sr",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "vllm-sr/d3",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "413d6775fc58",
  "provider": "vllm-sr",
  "questions": 1125,
  "request_seconds": 97.08732003800378,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "5d0d395b822834851227ebc924390a04327e1c414613b2a05c6a26ae795951bc",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 259,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 462257,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19120/v1/systemone
{
 "model": "vllm-sr/d3",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · clef_en

Name
Clef
Setup
Clef · decision model
Model
clef · provider clef
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.1196 for the run, list · maker's list price; RTX PRO 6000 Blackwell: $0.1584 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.0636 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.158. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run clef_en · first answer 2026-11-07 18:00:42 UTC · last answer 2026-11-07 18:04:31 UTC · span 229.0 s, pauses included (not charged) · charged 229.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1584 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/Cloudflare/clef · read 2026-11-10 · the card states a 27B model (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 27.48B (counted from the checkpoint: backbone 27,356,728,560 + joint head 128,056,324; the card says a 27B model)
Input
text, English
Endpoint
http://127.0.0.1:19200
Started
7 Nov 2026 18:00 UTC; last answer 7 Nov 2026 18:04 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Clef (site run 125); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "clef",
    "--provider",
    "clef",
    "--timeout",
    "120",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "clef",
  "perplexity_mode": null,
  "price": {
    "applies_to": "maker",
    "basis": "Cloudflare's list price for the same model",
    "date": "2026-10-18",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.24,
      "cache_write_input_tokens": 0.24,
      "cached_input_tokens": 0.24,
      "input_tokens": 0.24,
      "output_tokens": 0.0
    },
    "source": "https://developers.cloudflare.com/workers-ai/models/clef/",
    "text": "$0.24 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "clef",
  "questions": 1125,
  "request_seconds": 224.55437063500813,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "0510fb79b157548e6ea633cbbb4c5a41974e4edf81af30438d0d46d4d9a74bb6",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 125,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 498331,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19200/v1/systemone
{
 "model": "clef",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · clef_en_dontknow

Name
Clef
Setup
Clef · decision model · with the I don't know option F
Model
clef · provider clef
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.1304 for the run, list · maker's list price; RTX PRO 6000 Blackwell: $0.1619 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.065 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.162. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run clef_en_dontknow · first answer 2026-11-07 18:04:37 UTC · last answer 2026-11-07 18:08:31 UTC · span 234.0 s, pauses included (not charged) · charged 234.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1619 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/Cloudflare/clef · read 2026-11-10 · the card states a 27B model (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 27.48B (counted from the checkpoint: backbone 27,356,728,560 + joint head 128,056,324; the card says a 27B model)
Input
text, English
Endpoint
http://127.0.0.1:19200
Started
7 Nov 2026 18:04 UTC; last answer 7 Nov 2026 18:08 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Clef (site run 125); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "clef",
    "--provider",
    "clef",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "clef",
  "perplexity_mode": null,
  "price": {
    "applies_to": "maker",
    "basis": "Cloudflare's list price for the same model",
    "date": "2026-10-18",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.24,
      "cache_write_input_tokens": 0.24,
      "cached_input_tokens": 0.24,
      "input_tokens": 0.24,
      "output_tokens": 0.0
    },
    "source": "https://developers.cloudflare.com/workers-ai/models/clef/",
    "text": "$0.24 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "clef",
  "questions": 1125,
  "request_seconds": 229.5907855990099,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "78eac839d422c357427f12455854c890ac15f743dc9e4f8cb3ff904f1c3aa901",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 125,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 543331,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19200/v1/systemone
{
 "model": "clef",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · decider-v1.2_en

Name
Perplexity Decider 27B v1.2 API
Setup
Perplexity Decider 27B v1.2 (API) · decision model
Model
pplx-decider-v1.2 · provider perplexity
GPU
—
Cost
$0.00839 for the run · provider price × reported usage
Answer time
run decider-v1.2_en · first answer 2026-11-07 18:21:56 UTC · last answer 2026-11-07 18:28:28 UTC · span 392.0 s, pauses included (not charged) · charged 392.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/perplexity-ai/pplx-decider-v1.2 · read 2026-11-10 · the official model name: 27B (pplx-decider 27B v1.2; the card states no count of its own) (turkce #4263: the maker's stated total is shown); counted from the released files: 26.09B (counted from the released checkpoint: backbone 26,085,330,160 (its 11 safetensors shards) + decision head (readout.safetensors) 1,305,600 = 26,086,635,760; the API setups use the same count)
Input
text, English
Endpoint
https://api.perplexity.ai
Started
7 Nov 2026 18:21 UTC; last answer 7 Nov 2026 18:28 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Perplexity Decider 27B v1.2 (API) (site run 633); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "pplx-decider-v1.2",
    "--provider",
    "perplexity",
    "--timeout",
    "30",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "pplx-decider-v1.2",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Perplexity's Decisions API price",
    "date": "2026-11-05",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.02,
      "cache_write_input_tokens": 0.02,
      "cached_input_tokens": 0.02,
      "input_tokens": 0.02,
      "output_tokens": 0.0
    },
    "source": "https://docs.perplexity.ai/docs/getting-started/pricing",
    "text": "$0.02 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "perplexity",
  "questions": 1125,
  "request_seconds": 385.91272553100316,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "41dc41225f79516b737570d8eb94c3c6b2dd2320874002f771b6b75d6213a8fb",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 633,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 419507,
    "output_tokens": 1125
  }
}
Prompt template
POST https://api.perplexity.ai/v1/decisions
{
 "model": "pplx-decider-v1.2",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · onejev_en

Name
OneJev-27B
Setup
OneJev 27B · decision model
Model
OmniJev/OneJev-27B · provider onejev
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.09683 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0389 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.097. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run onejev_en · first answer 2026-11-07 18:23:01 UTC · last answer 2026-11-07 18:25:21 UTC · span 140.0 s, pauses included (not charged) · charged 140.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0968 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/OmniJev/OneJev-27B · read 2026-11-10 · the card states the 27B model of OneJev (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 27.36B (counted from the checkpoint: its index total_parameters 27,356,728,560)
Input
text, English
Endpoint
http://127.0.0.1:19190
Started
7 Nov 2026 18:23 UTC; last answer 7 Nov 2026 18:25 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup OneJev 27B (site run 264); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "OmniJev/OneJev-27B",
    "--provider",
    "onejev",
    "--timeout",
    "30",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "OmniJev/OneJev-27B",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "47f5776e71db",
  "provider": "onejev",
  "questions": 1125,
  "request_seconds": 136.54746135798723,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "24dc89795c24bddc48d9b89c1318e39ae8718eaf4891d2cc2d439b2d855c91a7",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 264,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 431882,
    "output_tokens": 1125
  }
}
Prompt template
POST http://127.0.0.1:19190/v1/systemone
{
 "model": "OmniJev/OneJev-27B",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · onejev_en_dontknow

Name
OneJev-27B
Setup
OneJev 27B · decision model · with the I don't know option F
Model
OmniJev/OneJev-27B · provider onejev
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.1038 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0417 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.104. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run onejev_en_dontknow · first answer 2026-11-07 18:25:25 UTC · last answer 2026-11-07 18:27:55 UTC · span 150.0 s, pauses included (not charged) · charged 150.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1038 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/OmniJev/OneJev-27B · read 2026-11-10 · the card states the 27B model of OneJev (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 27.36B (counted from the checkpoint: its index total_parameters 27,356,728,560)
Input
text, English
Endpoint
http://127.0.0.1:19190
Started
7 Nov 2026 18:25 UTC; last answer 7 Nov 2026 18:27 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup OneJev 27B (site run 264); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "OmniJev/OneJev-27B",
    "--provider",
    "onejev",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "OmniJev/OneJev-27B",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "413d6775fc58",
  "provider": "onejev",
  "questions": 1125,
  "request_seconds": 146.19797822599867,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "13f893cc642e1b2d8b0f752e52bdd3e1ae6dbea1c6579bab21afdd5a74bc51ee",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 264,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 467882,
    "output_tokens": 1125
  }
}
Prompt template
POST http://127.0.0.1:19190/v1/systemone
{
 "model": "OmniJev/OneJev-27B",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · decider-v1.2_en_dontknow

Name
Perplexity Decider 27B v1.2 API
Setup
Perplexity Decider 27B v1.2 (API) · decision model · with the I don't know option F
Model
pplx-decider-v1.2 · provider perplexity
GPU
—
Cost
$0.009065 for the run · provider price × reported usage
Answer time
run decider-v1.2_en_dontknow · first answer 2026-11-07 18:28:35 UTC · last answer 2026-11-07 18:34:33 UTC · span 358.0 s, pauses included (not charged) · charged 358.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/perplexity-ai/pplx-decider-v1.2 · read 2026-11-10 · the official model name: 27B (pplx-decider 27B v1.2; the card states no count of its own) (turkce #4263: the maker's stated total is shown); counted from the released files: 26.09B (counted from the released checkpoint: backbone 26,085,330,160 (its 11 safetensors shards) + decision head (readout.safetensors) 1,305,600 = 26,086,635,760; the API setups use the same count)
Input
text, English
Endpoint
https://api.perplexity.ai
Started
7 Nov 2026 18:28 UTC; last answer 7 Nov 2026 18:34 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Perplexity Decider 27B v1.2 (API) (site run 633); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "pplx-decider-v1.2",
    "--provider",
    "perplexity",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "pplx-decider-v1.2",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Perplexity's Decisions API price",
    "date": "2026-11-05",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.02,
      "cache_write_input_tokens": 0.02,
      "cached_input_tokens": 0.02,
      "input_tokens": 0.02,
      "output_tokens": 0.0
    },
    "source": "https://docs.perplexity.ai/docs/getting-started/pricing",
    "text": "$0.02 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "perplexity",
  "questions": 1125,
  "request_seconds": 352.5518577140065,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "5dd0aafe249c88f24f440cb57a8d4fdf266958c3a1fcd17a7f9d3fb2ec9b713e",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 633,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 453257,
    "output_tokens": 1125
  }
}
Prompt template
POST https://api.perplexity.ai/v1/decisions
{
 "model": "pplx-decider-v1.2",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · elephant-1_en

Name
Elephant-1
Setup
Elephant-1 · decision model
Model
baselabs/elephant-1 · provider baselabs
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.01512 for the run, list · maker's list price; RTX PRO 6000 Blackwell: $0.06571 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.0264 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.066. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run elephant-1_en · first answer 2026-11-07 18:35:29 UTC · last answer 2026-11-07 18:37:04 UTC · span 95.0 s, pauses included (not charged) · charged 95.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0657 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/baselabs/elephant-1 · read 2026-11-10 · the card states a 9B parameter decision model (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 9.41B (counted from the checkpoint: language model 8,953,803,264 + vision encoder 456,010,480; the card says a 9B decision model)
Input
text, English
Endpoint
http://127.0.0.1:19160
Started
7 Nov 2026 18:35 UTC; last answer 7 Nov 2026 18:37 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Elephant-1 (site run 373); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "baselabs/elephant-1",
    "--provider",
    "baselabs",
    "--timeout",
    "30",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "baselabs/elephant-1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "maker",
    "basis": "Baseten's list price for the same model",
    "date": "2026-11-01",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.035,
      "cache_write_input_tokens": 0.035,
      "cached_input_tokens": 0.035,
      "input_tokens": 0.035,
      "output_tokens": 0.0
    },
    "source": "https://www.baseten.co/library/elephant-1/",
    "text": "$0.035 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "baselabs",
  "questions": 1125,
  "request_seconds": 92.06893664297513,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "61cdfbeac91276ed79f01bf06ba62bdafc119404248d2ed064ecfb2271fd450c",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 373,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 431882,
    "output_tokens": 1125
  }
}
Prompt template
POST http://127.0.0.1:19160/v1/systemone
{
 "model": "baselabs/elephant-1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · elephant-1_en_dontknow

Name
Elephant-1
Setup
Elephant-1 · decision model · with the I don't know option F
Model
baselabs/elephant-1 · provider baselabs
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.01638 for the run, list · maker's list price; RTX PRO 6000 Blackwell: $0.06433 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.0258 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.064. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run elephant-1_en_dontknow · first answer 2026-11-07 18:37:07 UTC · last answer 2026-11-07 18:38:40 UTC · span 93.0 s, pauses included (not charged) · charged 93.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0643 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/baselabs/elephant-1 · read 2026-11-10 · the card states a 9B parameter decision model (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 9.41B (counted from the checkpoint: language model 8,953,803,264 + vision encoder 456,010,480; the card says a 9B decision model)
Input
text, English
Endpoint
http://127.0.0.1:19160
Started
7 Nov 2026 18:37 UTC; last answer 7 Nov 2026 18:38 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Elephant-1 (site run 373); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "baselabs/elephant-1",
    "--provider",
    "baselabs",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "baselabs/elephant-1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "maker",
    "basis": "Baseten's list price for the same model",
    "date": "2026-11-01",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.035,
      "cache_write_input_tokens": 0.035,
      "cached_input_tokens": 0.035,
      "input_tokens": 0.035,
      "output_tokens": 0.0
    },
    "source": "https://www.baseten.co/library/elephant-1/",
    "text": "$0.035 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "baselabs",
  "questions": 1125,
  "request_seconds": 89.42109007900399,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "d5b9716659a949996da62b101c2a2b3219d904bdfd18ae78dcebb536ea62c816",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 373,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 467882,
    "output_tokens": 1125
  }
}
Prompt template
POST http://127.0.0.1:19160/v1/systemone
{
 "model": "baselabs/elephant-1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · open-jev_en

Name
Open-Jev-27B-v1.1
Setup
Open-Jev 27B v1.1 · decision model
Model
open-jev-27b-v1.1 · provider open-jev
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.3313 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.133 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.331. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run open-jev_en · first answer 2026-11-07 18:45:33 UTC · last answer 2026-11-07 18:53:32 UTC · span 479.0 s, pauses included (not charged) · charged 479.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.3313 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/ZefanCai/Open-Jev-27B-v1.1 · read 2026-11-10 · the official model name: Open-Jev 27B (turkce #4263: the maker's stated total is shown); counted from the released files: 27.80B (the model as run: Qwen3.8-27B 27,781,427,952 + its released adapter and head 15,471,617 = 27,796,899,569 (the Open-Jev repo holds only the adapter and head))
Input
text, English
Endpoint
http://127.0.0.1:19230
Started
7 Nov 2026 18:45 UTC; last answer 7 Nov 2026 18:53 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Open-Jev 27B v1.1 (site run 45); runner ffc307cb7; server --max-length 4096 tokens
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "open-jev-27b-v1.1",
    "--provider",
    "open-jev",
    "--timeout",
    "30",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "open-jev-27b-v1.1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "47f5776e71db",
  "provider": "open-jev",
  "questions": 1125,
  "request_seconds": 474.0322139910022,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "e9ac57ee83a83d8d68e0b62936c782dd71837b615a2262d084444ba230dfc0b7",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 45,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 1678303,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19230/v1/systemone
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · open-jev_en_dontknow

Name
Open-Jev-27B-v1.1
Setup
Open-Jev 27B v1.1 · decision model · with the I don't know option F
Model
open-jev-27b-v1.1 · provider open-jev
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.4219 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.169 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.422. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run open-jev_en_dontknow · first answer 2026-11-07 18:53:46 UTC · last answer 2026-11-07 19:03:56 UTC · span 610.0 s, pauses included (not charged) · charged 610.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.4219 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/ZefanCai/Open-Jev-27B-v1.1 · read 2026-11-10 · the official model name: Open-Jev 27B (turkce #4263: the maker's stated total is shown); counted from the released files: 27.80B (the model as run: Qwen3.8-27B 27,781,427,952 + its released adapter and head 15,471,617 = 27,796,899,569 (the Open-Jev repo holds only the adapter and head))
Input
text, English
Endpoint
http://127.0.0.1:19230
Started
7 Nov 2026 18:53 UTC; last answer 7 Nov 2026 19:03 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Open-Jev 27B v1.1 (site run 45); runner ffc307cb7; server --max-length 4096 tokens
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "open-jev-27b-v1.1",
    "--provider",
    "open-jev",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "open-jev-27b-v1.1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "413d6775fc58",
  "provider": "open-jev",
  "questions": 1125,
  "request_seconds": 604.4510684630186,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "8b364470989c08fb04ae395434a244c440835d598f9ea44955fe4306110a6495",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 45,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 2153877,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19230/v1/systemone
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · cygnet_en

Name
Cygnet
Setup
Cygnet · decision model
Model
cygnet · provider cygnet
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.04468 for the run, list · public host's list price (same weights); RTX PRO 6000 Blackwell: $0.03458 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.0139 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.035. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run cygnet_en · first answer 2026-11-07 19:27:06 UTC · last answer 2026-11-07 19:27:56 UTC · span 50.0 s, pauses included (not charged) · charged 50.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0346 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 11.95B dense · source https://huggingface.co/google/gemma-4-12B-it · read 2026-11-10 · the card states Gemma 4 12B IT unchanged; its card: 11.95B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 11.96B (counted from the checkpoint (11,959,730,224): Gemma 4 12B IT unchanged, whose card says 11.95B)
Input
text, English
Endpoint
http://127.0.0.1:19220
Started
7 Nov 2026 19:27 UTC; last answer 7 Nov 2026 19:27 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Cygnet (site run 127); runner ffc307cb7; vLLM 0.30.0 container (port + 5) and cygnet-recipe's shim
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "cygnet",
    "--provider",
    "cygnet",
    "--timeout",
    "180",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "cygnet",
  "perplexity_mode": null,
  "price": {
    "applies_to": "weights",
    "basis": "SiliconFlow's list price for the same weights (Gemma 4 12B IT; precision not stated, ours BF16)",
    "date": "2026-11-05",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.1,
      "cache_write_input_tokens": 0.1,
      "cached_input_tokens": 0.1,
      "input_tokens": 0.1,
      "output_tokens": 0.3
    },
    "source": "https://www.siliconflow.com/models/gemma-4-12b-it",
    "text": "$0.10 input, $0.30 output per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "cygnet",
  "questions": 1125,
  "request_seconds": 46.42101441999512,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "f8c81568cd57056cbb8c59b6225d60cf2ff99ddc79762f68b0ca0aa2dc20349a",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 127,
  "status": "completed",
  "timeout": 180.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 443384,
    "output_tokens": 1125
  }
}
Prompt template
POST http://127.0.0.1:19220/v1/systemone
{
 "model": "cygnet",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · cygnet_en_dontknow

Name
Cygnet
Setup
Cygnet · decision model · with the I don't know option F
Model
cygnet · provider cygnet
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.04816 for the run, list · public host's list price (same weights); RTX PRO 6000 Blackwell: $0.03528 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.0142 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.035. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run cygnet_en_dontknow · first answer 2026-11-07 19:27:59 UTC · last answer 2026-11-07 19:28:50 UTC · span 51.0 s, pauses included (not charged) · charged 51.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0353 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 11.95B dense · source https://huggingface.co/google/gemma-4-12B-it · read 2026-11-10 · the card states Gemma 4 12B IT unchanged; its card: 11.95B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 11.96B (counted from the checkpoint (11,959,730,224): Gemma 4 12B IT unchanged, whose card says 11.95B)
Input
text, English
Endpoint
http://127.0.0.1:19220
Started
7 Nov 2026 19:27 UTC; last answer 7 Nov 2026 19:28 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Cygnet (site run 127); runner ffc307cb7; vLLM 0.30.0 container (port + 5) and cygnet-recipe's shim
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "cygnet",
    "--provider",
    "cygnet",
    "--timeout",
    "180",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "cygnet",
  "perplexity_mode": null,
  "price": {
    "applies_to": "weights",
    "basis": "SiliconFlow's list price for the same weights (Gemma 4 12B IT; precision not stated, ours BF16)",
    "date": "2026-11-05",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.1,
      "cache_write_input_tokens": 0.1,
      "cached_input_tokens": 0.1,
      "input_tokens": 0.1,
      "output_tokens": 0.3
    },
    "source": "https://www.siliconflow.com/models/gemma-4-12b-it",
    "text": "$0.10 input, $0.30 output per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "cygnet",
  "questions": 1125,
  "request_seconds": 48.38162024001758,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "5ba4e68bd4997ba6ce760ab9a693d8b804be54707cb59980c1e165f91cdb919c",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 127,
  "status": "completed",
  "timeout": 180.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 478259,
    "output_tokens": 1125
  }
}
Prompt template
POST http://127.0.0.1:19220/v1/systemone
{
 "model": "cygnet",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · winnow_en_dontknow

Name
Winnow-12B
Setup
Winnow-12B · decision model · with the I don't know option F
Model
winnow-12b · provider winnow
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.06087 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0244 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.061. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run winnow_en_dontknow · first answer 2026-11-07 19:33:22 UTC · last answer 2026-11-07 19:34:50 UTC · span 88.0 s, pauses included (not charged) · charged 88.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0609 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 12B dense · source https://huggingface.co/EldanRing/Winnow-12B · read 2026-11-10 · the official model name: Winnow-12B (turkce #4263: the maker's stated total is shown); counted from the released files: 11.91B (counted from the Q8_0 GGUF checkpoint (11,907,350,576); a fine-tune of Gemma 4 12B IT, whose card says 11.95B)
Input
text, English
Endpoint
http://127.0.0.1:19240
Started
7 Nov 2026 19:33 UTC; last answer 7 Nov 2026 19:34 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Winnow-12B (site run 128); runner ffc307cb7; server context 8,192 tokens
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "winnow-12b",
    "--provider",
    "winnow",
    "--timeout",
    "180",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "winnow-12b",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "413d6775fc58",
  "provider": "winnow",
  "questions": 1125,
  "request_seconds": 84.56830128500951,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "603dfef675488829ca8b9a41d1cba8dca09b2235e08ad9f454d7ea233d52b30d",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 128,
  "status": "completed",
  "timeout": 180.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 548751,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19240/v1/systemone
{
 "model": "winnow-12b",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · clef-flash_en

Name
Clef-Flash
Setup
Clef-flash · decision model
Model
clef-flash · provider clef
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.01894 for the run, list · maker's list price; RTX PRO 6000 Blackwell: $0.0913 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.0367 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.091. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run clef-flash_en · first answer 2026-11-07 19:39:27 UTC · last answer 2026-11-07 19:41:39 UTC · span 132.0 s, pauses included (not charged) · charged 132.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0913 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/Cloudflare/clef-flash · read 2026-11-10 · the card states a 9B model (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 9.53B (counted from the checkpoint: backbone 9,409,813,744 + joint head; the card says a 9B model)
Input
text, English
Endpoint
http://127.0.0.1:19210
Started
7 Nov 2026 19:39 UTC; last answer 7 Nov 2026 19:41 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Clef-flash (site run 139); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "clef-flash",
    "--provider",
    "clef",
    "--timeout",
    "60",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "clef-flash",
  "perplexity_mode": null,
  "price": {
    "applies_to": "maker",
    "basis": "Cloudflare's list price for the same model",
    "date": "2026-10-18",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.038,
      "cache_write_input_tokens": 0.038,
      "cached_input_tokens": 0.038,
      "input_tokens": 0.038,
      "output_tokens": 0.0
    },
    "source": "https://developers.cloudflare.com/workers-ai/models/clef-flash/",
    "text": "$0.038 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "clef",
  "questions": 1125,
  "request_seconds": 123.11124927998571,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "900a59a69facb8efdbd7aabfdfbac8e603193f4658b2a38ec77f6373f02d3ce6",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 139,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 498331,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19210/v1/systemone
{
 "model": "clef-flash",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · clef-flash_en_dontknow

Name
Clef-Flash
Setup
Clef-flash · decision model · with the I don't know option F
Model
clef-flash · provider clef
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.02065 for the run, list · maker's list price; RTX PRO 6000 Blackwell: $0.09061 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.0364 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.091. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run clef-flash_en_dontknow · first answer 2026-11-07 19:41:43 UTC · last answer 2026-11-07 19:43:54 UTC · span 131.0 s, pauses included (not charged) · charged 131.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0906 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/Cloudflare/clef-flash · read 2026-11-10 · the card states a 9B model (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 9.53B (counted from the checkpoint: backbone 9,409,813,744 + joint head; the card says a 9B model)
Input
text, English
Endpoint
http://127.0.0.1:19210
Started
7 Nov 2026 19:41 UTC; last answer 7 Nov 2026 19:43 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Clef-flash (site run 139); runner ffc307cb7
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "clef-flash",
    "--provider",
    "clef",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "clef-flash",
  "perplexity_mode": null,
  "price": {
    "applies_to": "maker",
    "basis": "Cloudflare's list price for the same model",
    "date": "2026-10-18",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.038,
      "cache_write_input_tokens": 0.038,
      "cached_input_tokens": 0.038,
      "input_tokens": 0.038,
      "output_tokens": 0.0
    },
    "source": "https://developers.cloudflare.com/workers-ai/models/clef-flash/",
    "text": "$0.038 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "clef",
  "questions": 1125,
  "request_seconds": 123.32337547702264,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "7b0b6f4682833840615f8a9d06ebf81be87f6c3b05322fbea442e788ae4557c3",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": 139,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 543331,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19210/v1/systemone
{
 "model": "clef-flash",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · deepseek-v4.1-flash-1017_en

Name
DeepSeek-V4.1-Flash high · chat · 1017
Setup
DeepSeek-V4.1-Flash 1017 · chat model
Model
deepseek/deepseek-v4.1-flash-1017 · provider openrouter
GPU
—
Cost
$1.853 for the run · provider price × reported usage
Answer time
run deepseek-v4.1-flash-1017_en · first answer 2026-11-08 10:41:16 UTC · last answer 2026-11-08 15:35:35 UTC · span 17659.0 s, pauses included (not charged) · charged 2854.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 7.4 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 4 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 552B MoE · 16B active · source https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash · read 2026-11-10 · Ümit #4254: from the V4.1 Flash model card (open weights on Hugging Face): 552B backbone total, 16B active (decode; 8B in prefill); the card also lists 196B of conditional memory, not counted in the total
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
8 Nov 2026 10:41 UTC; last answer 8 Nov 2026 15:35 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
DeepSeek-V4.1-Flash 1017 through OpenRouter (deepseek/deepseek-v4.1-flash-1017), provider pinned to deepseek, fallbacks off; reasoning {"effort": "high"}; temperature 1.0; max_tokens 32768; one question per request; parameters 16B / 552B active / total (Tinker models table). English translations. DeepSeek-V4.1-Flash 1017 also ran on our GPUs for other benchmarks (Reading, Backgammon); those are separate setups under the same model name, not verified to be identical to this API setup (Umit #3101).
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "deepseek"
    ]
  },
  "reasoning": {
    "effort": "high"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · glm-5.3-flash_en

Name
GLM-5.3-Flash high · chat
Setup
GLM-5.3-Flash · chat model
Model
z-ai/glm-5.3-flash · provider openrouter
GPU
—
Cost
$1.189 for the run · provider price × reported usage
Answer time
run glm-5.3-flash_en · first answer 2026-11-08 10:41:16 UTC · last answer 2026-11-08 16:47:42 UTC · span 21986.0 s, pauses included (not charged) · charged 7054.0 s in 3 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 6.8 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 12 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 320B MoE · 18B active · from the run's notes (Tinker models table); also https://huggingface.co/zai-org/GLM-5.3-Flash · read 2026-11-09
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
8 Nov 2026 10:41 UTC; last answer 8 Nov 2026 16:47 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
GLM-5.3-Flash through OpenRouter (z-ai/glm-5.3-flash), provider pinned to z-ai/fp8, fallbacks off; reasoning {"effort": "high"}; temperature 1.0; max_tokens 32768; one question per request; parameters 18B / 320B active / total (Tinker models table). English translations.
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "z-ai/fp8"
    ]
  },
  "reasoning": {
    "effort": "high"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · qwen3.5-397b-a17b_en

Name
Qwen3.5-397B-A17B on · chat
Setup
Qwen3.5-397B-A17B · chat model
Model
qwen/qwen3.5-397b-a17b · provider openrouter
GPU
—
Cost
$15.1 for the run · provider price × reported usage
Answer time
run qwen3.5-397b-a17b_en · first answer 2026-11-08 10:42:52 UTC · last answer 2026-11-08 22:03:37 UTC · span 40845.0 s, pauses included (not charged) · charged 13283.0 s in 3 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 7.9 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 1 empty reply asked once more
Model size
Parameters: 397B MoE · 17B active · from the run's notes (Tinker models table); also https://huggingface.co/Qwen/Qwen3.5-397B-A17B · read 2026-11-09
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
8 Nov 2026 10:41 UTC; last answer 8 Nov 2026 22:03 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Qwen3.5-397B-A17B through OpenRouter (qwen/qwen3.5-397b-a17b), provider pinned to alibaba, fallbacks off; reasoning {"enabled": true}; temperature 1.0; max_tokens 32768; one question per request; parameters 17B / 397B active / total (Tinker models table). English translations.
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "alibaba"
    ]
  },
  "reasoning": {
    "enabled": true
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · mimo-v2.6-flash_en

Name
MiMo-V2.6-Flash on · chat
Setup
MiMo V2.6 Flash · chat model
Model
xiaomi/mimo-v2.6-flash · provider openrouter
GPU
—
Cost
$0.03404 for the run · provider price × reported usage
Answer time
run mimo-v2.6-flash_en · first answer 2026-11-08 10:41:30 UTC · last answer 2026-11-08 10:52:11 UTC · span 641.0 s, pauses included (not charged) · charged 641.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 3.1 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 1 answer cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 309B MoE · 15B active · from the run's notes (OpenRouter model card); also https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL · read 2026-11-09
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
8 Nov 2026 10:41 UTC; last answer 8 Nov 2026 10:52 UTC
Progress
25 answered of 1,125 asked; 0 request errors; smoke test at 32,768 tokens only; the full run uses 65,536 tokens (run mimo-v2.6-flash-64k_en)
Notes
MiMo V2.6 Flash through OpenRouter (xiaomi/mimo-v2.6-flash), provider pinned to xiaomi/fp8, fallbacks off; reasoning {"enabled": true}; temperature 1.0; max_tokens 32768; one question per request; parameters 15B / 309B active / total (OpenRouter model card). English translations.
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "xiaomi/fp8"
    ]
  },
  "reasoning": {
    "enabled": true
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · minimax-m3.1-flash_en

Name
MiniMax-M3.1-Flash high · chat
Setup
MiniMax-M3.1-Flash · chat model
Model
minimax/minimax-m3.1-flash · provider openrouter
GPU
—
Cost
$0.9036 for the run · provider price × reported usage
Answer time
run minimax-m3.1-flash_en · first answer 2026-11-08 10:41:14 UTC · last answer 2026-11-08 15:31:58 UTC · span 17444.0 s, pauses included (not charged) · charged 2505.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 7.1 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 2 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 428B MoE · 23B active · from the run's notes (NVIDIA NIM model card); also https://huggingface.co/MiniMaxAI/MiniMax-M3.1-Flash · read 2026-11-09 · the card: approximately 428B total, 23B activated per token
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
8 Nov 2026 10:41 UTC; last answer 8 Nov 2026 15:31 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
MiniMax-M3.1-Flash through OpenRouter (minimax/minimax-m3.1-flash), provider pinned to minimax, fallbacks off; reasoning {"effort": "high"}; temperature 1.0; max_tokens 32768; one question per request; parameters 23B / 428B active / total (NVIDIA NIM model card). English translations.
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "minimax"
    ]
  },
  "reasoning": {
    "effort": "high"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · inkling-small_en

Name
Inkling-Small high · chat · Jul 2026
Setup
Inkling Small · chat model
Model
thinkingmachines/inkling-small · provider openrouter
GPU
—
Cost
$0.201 for the run · provider price × reported usage
Answer time
run inkling-small_en · first answer 2026-11-08 10:50:01 UTC · last answer 2026-11-08 11:16:45 UTC · span 1604.0 s, pauses included (not charged) · charged 1604.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 4.7 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 1 answer cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 276B MoE · 12B active · source https://thinkingmachines.ai/model-card/inkling-small/ · read 2026-11-10 · the card: 276B total, 12B active (Ümit #4245: the card's count, not our checkpoint count of 265.96B by turkce, bus #3592)
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
8 Nov 2026 10:48 UTC; last answer 8 Nov 2026 11:16 UTC
Progress
25 answered of 1,125 asked; 0 request errors; smoke test at 32,768 tokens only; the full run uses 65,536 tokens (run inkling-small-64k_en)
Notes
Inkling Small through OpenRouter (thinkingmachines/inkling-small), provider pinned to deepinfra/fp8, fallbacks off; reasoning {"effort": "high"}; temperature 1.0; max_tokens 32768; one question per request; parameters 12B / 276B active / total (Thinking Machines (Tinker models table)). English translations.
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "deepinfra/fp8"
    ]
  },
  "reasoning": {
    "effort": "high"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · mimo-v2.6-flash-64k_en

Name
MiMo-V2.6-Flash on · chat
Setup
MiMo V2.6 Flash · chat model
Model
xiaomi/mimo-v2.6-flash · provider openrouter
GPU
—
Cost
$1.564 for the run · provider price × reported usage
Answer time
run mimo-v2.6-flash-64k_en · first answer 2026-11-08 14:51:55 UTC · last answer 2026-11-08 22:13:08 UTC · span 26473.0 s, pauses included (not charged) · charged 14411.0 s in 5 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 7.5 in flight (sum of request latencies / active time)
Output limit
65,536 tokens · 23 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 309B MoE · 15B active · from the run's notes (OpenRouter model card); also https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL · read 2026-11-09
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
8 Nov 2026 14:51 UTC; last answer 8 Nov 2026 22:13 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
MiMo V2.6 Flash through OpenRouter (xiaomi/mimo-v2.6-flash), provider pinned to xiaomi/fp8, fallbacks off; reasoning {"enabled": true}; temperature 1.0; max_tokens 65536; one question per request; parameters 15B / 309B active / total (OpenRouter model card). English translations.
Parameters
{
  "max_tokens": 65536,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "xiaomi/fp8"
    ]
  },
  "reasoning": {
    "enabled": true
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · inkling-small-64k_en

Name
Inkling-Small high · chat · Jul 2026
Setup
Inkling Small · chat model
Model
thinkingmachines/inkling-small · provider openrouter
GPU
—
Cost
$6.176 for the run · provider price × reported usage
Answer time
run inkling-small-64k_en · first answer 2026-11-08 14:52:13 UTC · last answer 2026-11-08 18:45:12 UTC · span 13979.0 s, pauses included (not charged) · charged 4217.0 s in 3 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 7.6 in flight (sum of request latencies / active time)
Output limit
65,536 tokens · 0 answers cut off at it with no letter (counted wrong) · 1 empty reply asked once more
Model size
Parameters: 276B MoE · 12B active · source https://thinkingmachines.ai/model-card/inkling-small/ · read 2026-11-10 · the card: 276B total, 12B active (Ümit #4245: the card's count, not our checkpoint count of 265.96B by turkce, bus #3592)
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
8 Nov 2026 14:51 UTC; last answer 8 Nov 2026 18:45 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Inkling Small through OpenRouter (thinkingmachines/inkling-small), provider pinned to deepinfra/fp8, fallbacks off; reasoning {"effort": "high"}; temperature 1.0; max_tokens 65536; one question per request; parameters 12B / 276B active / total (Thinking Machines (Tinker models table)). English translations.
Parameters
{
  "max_tokens": 65536,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "deepinfra/fp8"
    ]
  },
  "reasoning": {
    "effort": "high"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · minimax-m3.1-flash_en_dontknow

Name
MiniMax-M3.1-Flash high · chat
Setup
MiniMax-M3.1-Flash · chat model · with the I don't know option F
Model
minimax/minimax-m3.1-flash · provider openrouter
GPU
—
Cost
$0.6921 for the run · provider price × reported usage
Answer time
run minimax-m3.1-flash_en_dontknow · first answer 2026-11-08 15:32:01 UTC · last answer 2026-11-08 16:09:59 UTC · span 2278.0 s, pauses included (not charged) · charged 2278.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 7.9 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 1 answer cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 428B MoE · 23B active · from the run's notes (NVIDIA NIM model card); also https://huggingface.co/MiniMaxAI/MiniMax-M3.1-Flash · read 2026-11-09 · the card: approximately 428B total, 23B activated per token
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
8 Nov 2026 15:31 UTC; last answer 8 Nov 2026 16:09 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
MiniMax-M3.1-Flash through OpenRouter (minimax/minimax-m3.1-flash), provider pinned to minimax, fallbacks off; reasoning {"effort": "high"}; temperature 1.0; max_tokens 32768; one question per request; parameters 23B / 428B active / total (NVIDIA NIM model card). English translations.
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "minimax"
    ]
  },
  "reasoning": {
    "effort": "high"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · deepseek-v4.1-flash-1017_en_dontknow

Name
DeepSeek-V4.1-Flash high · chat · 1017
Setup
DeepSeek-V4.1-Flash 1017 · chat model · with the I don't know option F
Model
deepseek/deepseek-v4.1-flash-1017 · provider openrouter
GPU
—
Cost
$1.924 for the run · provider price × reported usage
Answer time
run deepseek-v4.1-flash-1017_en_dontknow · first answer 2026-11-08 15:35:39 UTC · last answer 2026-11-08 16:21:03 UTC · span 2724.0 s, pauses included (not charged) · charged 2724.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 7.8 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 14 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 552B MoE · 16B active · source https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash · read 2026-11-10 · Ümit #4254: from the V4.1 Flash model card (open weights on Hugging Face): 552B backbone total, 16B active (decode; 8B in prefill); the card also lists 196B of conditional memory, not counted in the total
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
8 Nov 2026 15:35 UTC; last answer 8 Nov 2026 16:21 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
DeepSeek-V4.1-Flash 1017 through OpenRouter (deepseek/deepseek-v4.1-flash-1017), provider pinned to deepseek, fallbacks off; reasoning {"effort": "high"}; temperature 1.0; max_tokens 32768; one question per request; parameters 16B / 552B active / total (Tinker models table). English translations. DeepSeek-V4.1-Flash 1017 also ran on our GPUs for other benchmarks (Reading, Backgammon); those are separate setups under the same model name, not verified to be identical to this API setup (Umit #3101).
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "deepseek"
    ]
  },
  "reasoning": {
    "effort": "high"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · glm-5.3-flash_en_dontknow

Name
GLM-5.3-Flash high · chat
Setup
GLM-5.3-Flash · chat model · with the I don't know option F
Model
z-ai/glm-5.3-flash · provider openrouter
GPU
—
Cost
$1.116 for the run · provider price × reported usage
Answer time
run glm-5.3-flash_en_dontknow · first answer 2026-11-08 16:47:49 UTC · last answer 2026-11-08 18:16:15 UTC · span 5306.0 s, pauses included (not charged) · charged 5306.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 7.6 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 11 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 320B MoE · 18B active · from the run's notes (Tinker models table); also https://huggingface.co/zai-org/GLM-5.3-Flash · read 2026-11-09
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
8 Nov 2026 16:47 UTC; last answer 8 Nov 2026 18:16 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
GLM-5.3-Flash through OpenRouter (z-ai/glm-5.3-flash), provider pinned to z-ai/fp8, fallbacks off; reasoning {"effort": "high"}; temperature 1.0; max_tokens 32768; one question per request; parameters 18B / 320B active / total (Tinker models table). English translations.
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "z-ai/fp8"
    ]
  },
  "reasoning": {
    "effort": "high"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · inkling-small-64k_en_dontknow

Name
Inkling-Small high · chat · Jul 2026
Setup
Inkling Small · chat model · with the I don't know option F
Model
thinkingmachines/inkling-small · provider openrouter
GPU
—
Cost
$5.77 for the run · provider price × reported usage
Answer time
run inkling-small-64k_en_dontknow · first answer 2026-11-08 17:33:43 UTC · last answer 2026-11-08 18:44:34 UTC · span 4251.0 s, pauses included (not charged) · charged 4251.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 8.0 in flight (sum of request latencies / active time)
Output limit
65,536 tokens · 0 answers cut off at it with no letter (counted wrong) · 2 empty replies asked once more
Model size
Parameters: 276B MoE · 12B active · source https://thinkingmachines.ai/model-card/inkling-small/ · read 2026-11-10 · the card: 276B total, 12B active (Ümit #4245: the card's count, not our checkpoint count of 265.96B by turkce, bus #3592)
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
8 Nov 2026 17:33 UTC; last answer 8 Nov 2026 18:44 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Inkling Small through OpenRouter (thinkingmachines/inkling-small), provider pinned to deepinfra/fp8, fallbacks off; reasoning {"effort": "high"}; temperature 1.0; max_tokens 65536; one question per request; parameters 12B / 276B active / total (Thinking Machines (Tinker models table)). English translations.
Parameters
{
  "max_tokens": 65536,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "deepinfra/fp8"
    ]
  },
  "reasoning": {
    "effort": "high"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · qwen3.5-397b-a17b_en_dontknow

Name
Qwen3.5-397B-A17B on · chat
Setup
Qwen3.5-397B-A17B · chat model · with the I don't know option F
Model
qwen/qwen3.5-397b-a17b · provider openrouter
GPU
—
Cost
$14.73 for the run · provider price × reported usage
Answer time
run qwen3.5-397b-a17b_en_dontknow · first answer 2026-11-08 22:04:23 UTC · last answer 2026-11-09 01:35:34 UTC · span 12671.0 s, pauses included (not charged) · charged 12671.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 8.0 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 397B MoE · 17B active · from the run's notes (Tinker models table); also https://huggingface.co/Qwen/Qwen3.5-397B-A17B · read 2026-11-09
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
8 Nov 2026 22:03 UTC; last answer 9 Nov 2026 01:35 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Qwen3.5-397B-A17B through OpenRouter (qwen/qwen3.5-397b-a17b), provider pinned to alibaba, fallbacks off; reasoning {"enabled": true}; temperature 1.0; max_tokens 32768; one question per request; parameters 17B / 397B active / total (Tinker models table). English translations.
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "alibaba"
    ]
  },
  "reasoning": {
    "enabled": true
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · mimo-v2.6-flash-64k_en_dontknow

Name
MiMo-V2.6-Flash on · chat
Setup
MiMo V2.6 Flash · chat model · with the I don't know option F
Model
xiaomi/mimo-v2.6-flash · provider openrouter
GPU
—
Cost
$1.486 for the run · provider price × reported usage
Answer time
run mimo-v2.6-flash-64k_en_dontknow · first answer 2026-11-08 22:13:20 UTC · last answer 2026-11-09 01:24:52 UTC · span 11492.0 s, pauses included (not charged) · charged 11492.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 7.8 in flight (sum of request latencies / active time)
Output limit
65,536 tokens · 18 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 309B MoE · 15B active · from the run's notes (OpenRouter model card); also https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL · read 2026-11-09
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
8 Nov 2026 22:13 UTC; last answer 9 Nov 2026 01:24 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
MiMo V2.6 Flash through OpenRouter (xiaomi/mimo-v2.6-flash), provider pinned to xiaomi/fp8, fallbacks off; reasoning {"enabled": true}; temperature 1.0; max_tokens 65536; one question per request; parameters 15B / 309B active / total (OpenRouter model card). English translations.
Parameters
{
  "max_tokens": 65536,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "xiaomi/fp8"
    ]
  },
  "reasoning": {
    "enabled": true
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · celeris-1.1_en

Name
celeris-1.1-decision
Setup
Celeris-1.1 Decision · decision model
Model
celeris-1.1-decision · provider celeris
GPU
—
Cost
$0.001733 for the run · provider price × reported usage
Answer time
run celeris-1.1_en · first answer 2026-11-09 12:09:37 UTC · last answer 2026-11-09 12:16:53 UTC · span 436.0 s, pauses included (not charged) · charged 436.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://docs.celeris.ai/decisions · read 2026-11-10
Input
text, English
Endpoint
https://inference.celeris.ai/celeris-1-decision
Started
9 Nov 2026 12:09 UTC; last answer 9 Nov 2026 12:16 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Celeris-1.1 Decision (site run None); runner ffc307cb7; probabilities come rounded to two decimals; the published runner has no price for celeris-1.1-decision, so runs carry no cost_usd (tokens × $0.004/1M)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "celeris-1.1-decision",
    "--provider",
    "celeris",
    "--timeout",
    "30",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "celeris-1.1-decision",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Celeris's decision model price",
    "date": "2026-11-09",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.004,
      "cache_write_input_tokens": 0.004,
      "cached_input_tokens": 0.004,
      "input_tokens": 0.004,
      "output_tokens": 0.0
    },
    "source": "https://docs.celeris.ai/pricing",
    "text": "$0.004 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "celeris",
  "questions": 1125,
  "request_seconds": 429.36234423697897,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "fe8ebfd5c9c4c125faf60aff20add8b677a99df6b47657fafcafcfb86e38e3f6",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 433259,
    "output_tokens": 0
  }
}
Prompt template
POST https://inference.celeris.ai/celeris-1-decision/v1/systemone
{
 "model": "celeris-1.1-decision",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · lfm2.5-8b-a1b_en

Name
LFM2.5-8B-A1B on · chat
Setup
LFM2.5-8B-A1B (chat, thinking, on our GPU) · chat model
Model
lfm2.5-8b-a1b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$5.247 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 2.11 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $5.247. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes†
GPU time
run lfm2.5-8b-a1b_en · first answer 2026-11-09 12:11:39 UTC · last answer 2026-11-09 14:18:05 UTC · span 7586.0 s, pauses included (not charged) · charged 7586.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $5.2470 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)†
Requests in flight
≈ 26.6 in flight (sum of request latencies / active time)†
Output limit
32,768 tokens · 83 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 8.3B MoE · 1.5B active · source https://huggingface.co/LiquidAI/LFM2.5-8B-A1B · read 2026-11-10 · the card states 8.3B total / 1.5B active (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 8.47B / 1.69B active (MoE, 4 of 32 experts; checkpoint 8,467,856,832, active 1,685,667,264 incl. tied embeddings (1.42B without); card: 8.3B total, 1.5B active)
Input
text, English
Endpoint
http://127.0.0.1:18601/v1/chat/completions
Started
9 Nov 2026 12:10 UTC; last answer 9 Nov 2026 14:18 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
LFM2.5-8B-A1B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from LiquidAI/LFM2.5-8B-A1B@dd83f5cb02aa2938b66bba9a9998d38d0517c1f3, vLLM reasoning parser qwen3, one RTX PRO 6000 on our GPU host alone; thinking always on (reasoning model; no switch, no levels); sampling temperature 0.2, top_k 80, repetition_penalty 1.05, from generation_config.json; seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 1.5B / 8.3B active / total (LiquidAI model card). English translations (DeepSeek + review). Phases: up to 8 requests at a time from the runner until 12:31:02 UTC (86 Logic answers stored; the 8 in flight then were aborted, not stored, and asked again), then up to 32 (the server, vLLM --max-num-seqs 16, runs at most 16 at once and queues the rest, so latency per request includes queue time). Same image, revision, server arguments and request settings in both phases; vLLM does not guarantee identical outputs across batch sizes (no batch-invariance flag), so the two phases are not claimed answer-equivalent. Interrupted attempts: the runner was stopped while 8 requests were in flight (not a quiescent checkpoint); those 8 attempts produced no stored answer, their questions were sent again after the restart, and their token usage is unknown (not counted as zero; it is not in the answer counts). The GPU time of this run (first to last answer per stretch) covers the wall-clock time they used, including the ~4-minute server restart. No thinking-off setup (bus #6516/#6924/#8187/#8202): the current served chat template exposes no supported switch for a separate thinking-off run; no separate off variant has been run or scored. (Checked 15 Nov in the pinned template: its only option, preserve_thinking, keeps or drops earlier turns' thinking.)
Parameters
{
  "max_tokens": 32768,
  "repetition_penalty": 1.05,
  "seed": 0,
  "temperature": 0.2,
  "top_k": 80
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · celeris-1.1_en_dontknow

Name
celeris-1.1-decision
Setup
Celeris-1.1 Decision · decision model · with the I don't know option F
Model
celeris-1.1-decision · provider celeris
GPU
—
Cost
$0.001886 for the run · provider price × reported usage
Answer time
run celeris-1.1_en_dontknow · first answer 2026-11-09 12:17:04 UTC · last answer 2026-11-09 12:24:17 UTC · span 433.0 s, pauses included (not charged) · charged 433.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://docs.celeris.ai/decisions · read 2026-11-10
Input
text, English
Endpoint
https://inference.celeris.ai/celeris-1-decision
Started
9 Nov 2026 12:17 UTC; last answer 9 Nov 2026 12:24 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Celeris-1.1 Decision (site run None); runner ffc307cb7; probabilities come rounded to two decimals; the published runner has no price for celeris-1.1-decision, so runs carry no cost_usd (tokens × $0.004/1M)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "celeris-1.1-decision",
    "--provider",
    "celeris",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "celeris-1.1-decision",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Celeris's decision model price",
    "date": "2026-11-09",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.004,
      "cache_write_input_tokens": 0.004,
      "cached_input_tokens": 0.004,
      "input_tokens": 0.004,
      "output_tokens": 0.0
    },
    "source": "https://docs.celeris.ai/pricing",
    "text": "$0.004 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "celeris",
  "questions": 1125,
  "request_seconds": 428.1363685370525,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "735307cd40f62e82fd79ace1309ede3b8e4477fa266cdcd0fbb799afa2acad47",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 471509,
    "output_tokens": 0
  }
}
Prompt template
POST https://inference.celeris.ai/celeris-1-decision/v1/systemone
{
 "model": "celeris-1.1-decision",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · lfm2.5-8b-a1b_en_dontknow

Name
LFM2.5-8B-A1B on · chat
Setup
LFM2.5-8B-A1B (chat, thinking, on our GPU) · chat model · with the I don't know option F
Model
lfm2.5-8b-a1b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$3.157 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 1.27 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $3.157. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run lfm2.5-8b-a1b_en_dontknow · first answer 2026-11-09 14:44:10 UTC · last answer 2026-11-09 16:00:15 UTC · span 4565.0 s, pauses included (not charged) · charged 4565.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $3.1575 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.5 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 29 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 8.3B MoE · 1.5B active · source https://huggingface.co/LiquidAI/LFM2.5-8B-A1B · read 2026-11-10 · the card states 8.3B total / 1.5B active (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 8.47B / 1.69B active (MoE, 4 of 32 experts; checkpoint 8,467,856,832, active 1,685,667,264 incl. tied embeddings (1.42B without); card: 8.3B total, 1.5B active)
Input
text, English
Endpoint
http://127.0.0.1:18601/v1/chat/completions
Started
9 Nov 2026 14:43 UTC; last answer 9 Nov 2026 16:00 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
LFM2.5-8B-A1B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from LiquidAI/LFM2.5-8B-A1B@dd83f5cb02aa2938b66bba9a9998d38d0517c1f3, vLLM reasoning parser qwen3, one RTX PRO 6000 on our GPU host alone; thinking always on (reasoning model; no switch, no levels); sampling temperature 0.2, top_k 80, repetition_penalty 1.05, from generation_config.json; seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 1.69B / 8.47B active / total (MoE, 4 of 32 experts; checkpoint 8,467,856,832, active 1,685,667,264 incl. tied embeddings, 1.42B without; card: 8.3B total, 1.5B active). English translations (DeepSeek + review). No thinking-off setup (bus #6516/#6924/#8187/#8202): the current served chat template exposes no supported switch for a separate thinking-off run; no separate off variant has been run or scored. (Checked 15 Nov in the pinned template: its only option, preserve_thinking, keeps or drops earlier turns' thinking.)
Parameters
{
  "max_tokens": 32768,
  "repetition_penalty": 1.05,
  "seed": 0,
  "temperature": 0.2,
  "top_k": 80
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · gemma-4-e4b-it_en

Name
Gemma 4 E4B chat
Setup
Gemma 4 E4B (chat, on our GPU) · chat model
Model
gemma-4-e4b-it · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.9476 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.381 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.948. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gemma-4-e4b-it_en · first answer 2026-11-09 16:39:08 UTC · last answer 2026-11-09 17:01:58 UTC · span 1370.0 s, pauses included (not charged) · charged 1370.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.9476 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 29.9 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 3 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 8B dense · 4.5B effective · source https://huggingface.co/google/gemma-4-E4B-it · read 2026-11-10 · the card states 4.5B effective (8B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 8.00B (counted from the checkpoint (7,996,157,418: language model 7.52B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 8B with embeddings, 4.5B effective; not a mixture of experts)
Input
text, English
Endpoint
http://127.0.0.1:18602/v1/chat/completions
Started
9 Nov 2026 16:39 UTC; last answer 9 Nov 2026 17:01 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Gemma 4 E4B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from google/gemma-4-E4B-it@4ecc2ba63bcac23ba4a767fc87025b1450f84c9c, vLLM reasoning parser gemma4, one RTX PRO 6000 on our GPU host alone; thinking off (template default, as the Gemma 4 31B run); sampling temperature 0, as the Gemma 4 31B chat run; no seed (as the Gemma 4 31B run); max_tokens 8,192; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 8.00B / 8.00B active / total (dense; checkpoint 7,996,157,418 incl. vision 0.17B and audio 0.31B towers; card: 8B with per-layer embeddings, 4.5B effective). English translations (DeepSeek + review).
Parameters
{
  "max_tokens": 8192,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · gemma-4-e4b-it_en_dontknow

Name
Gemma 4 E4B chat
Setup
Gemma 4 E4B (chat, on our GPU) · chat model · with the I don't know option F
Model
gemma-4-e4b-it · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.8487 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.341 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.849. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gemma-4-e4b-it_en_dontknow · first answer 2026-11-09 17:17:10 UTC · last answer 2026-11-09 17:37:37 UTC · span 1227.0 s, pauses included (not charged) · charged 1227.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.8487 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.6 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 1 answer cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 8B dense · 4.5B effective · source https://huggingface.co/google/gemma-4-E4B-it · read 2026-11-10 · the card states 4.5B effective (8B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 8.00B (counted from the checkpoint (7,996,157,418: language model 7.52B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 8B with embeddings, 4.5B effective; not a mixture of experts)
Input
text, English
Endpoint
http://127.0.0.1:18602/v1/chat/completions
Started
9 Nov 2026 17:17 UTC; last answer 9 Nov 2026 17:37 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Gemma 4 E4B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from google/gemma-4-E4B-it@4ecc2ba63bcac23ba4a767fc87025b1450f84c9c, vLLM reasoning parser gemma4, one RTX PRO 6000 on our GPU host alone; thinking off (template default, as the Gemma 4 31B run); sampling temperature 0, as the Gemma 4 31B chat run; no seed (as the Gemma 4 31B run); max_tokens 8,192; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 8.00B / 8.00B active / total (dense; checkpoint 7,996,157,418 incl. vision 0.17B and audio 0.31B towers; card: 8B with per-layer embeddings, 4.5B effective). English translations (DeepSeek + review).
Parameters
{
  "max_tokens": 8192,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · lfm3-1.7b_en

Name
LFM3-1.7B on · chat
Setup
LFM3-1.7B (chat, thinking, on our GPU) · chat model
Model
lfm3-1.7b · provider llama.cpp → vLLM
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$28.05 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 11.3 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $28.052. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-09 19:42:04–19:59:28 UTC; 2026-11-09 20:21:36–20:35:11 UTC; 2026-11-09 21:33:08–21:53:52 UTC; 2026-11-10 00:37:59–11:47:24 UTC; 2026-11-12 11:47:49–11:58:10 UTC; 2026-11-12 17:15:47–17:49:57 UTC†
GPU time
run lfm3-1.7b_en · first answer 2026-11-09 17:59:51 UTC · last answer 2026-11-12 18:01:27 UTC · span 259296.0 s, pauses included (not charged) · charged 40557.0 s in 7 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $28.0519 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)†
Requests in flight
≈ 23.9 in flight (sum of request latencies / active time)†
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 1.73B dense · from the run's notes (dense; BF16 GGUF checkpoint 1,728,801,600, text only; card: 1.7B); also https://huggingface.co/LiquidAI/LFM3-1.7B-GGUF · read 2026-11-09 · the released checkpoint's tensors without registered buffers: 1,728,801,536 (the served BF16 GGUF also stores a 64-element RoPE buffer, 1,728,801,600); text only; card: 1.7B
Input
text, English
Endpoint
http://127.0.0.1:18605/v1/chat/completions
Started
9 Nov 2026 17:59 UTC; last answer 12 Nov 2026 18:01 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete, but replaced by run lfm3-1.7b-vllm_en (see its notes for why)
Notes
Engine: llama.cpp b12020 for answers 1–528, then vLLM nightly be47189f (from 12 Nov 17:49 UTC); same checkpoint, template and sampling. LFM3-1.7B, served with llama.cpp server b12020 (commit 0ec4027, CUDA build), official BF16 GGUF LiquidAI/LFM3-1.7B-GGUF@cd0c74c575b5c416f3011b5b4c298077f26ca8c2 LFM3-1.7B-BF16.gguf (vLLM v0.30.0 has no Lfm3ForCausalLM), chat template from LiquidAI/LFM3-1.7B@0dc4268b561499c648c7ba7c9466bf3d8d01517a, llama.cpp --reasoning-format deepseek, one RTX PRO 6000 on our GPU host alone; thinking on (enable_thinking true, the template default; on/off only); sampling temperature 0.1, top_k 50, repeat_penalty 1.1, repeat_last_n 69632, top_p 1, min_p 0, from generation_config.json; repetition_penalty as llama.cpp repeat_penalty over the whole context, llama.cpp's own top_p/min_p defaults switched off; seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 1.73B / 1.73B active / total (dense; BF16 GGUF checkpoint 1,728,801,600, text only; card: 1.7B). English translations (DeepSeek + review). Was paused (Umit #3883, 2026-11-10T00:38Z) until 12 Nov: skipped for now, stopped at 206 answers; kept, not ranked. Cause: llama.cpp main thread CPU-bound (repeat_penalty over the whole context for every token, 32 slots: ~118 tokens/s in total). Resumed 2026-11-12T11:31Z from answer 207 (Logic) / 26 (Math): 4 llama-server instances × 8 slots (same build, GGUF, flags, sampling, per-slot context 69,632, 32 requests in flight as before) behind a forwarder on the same port, for throughput only (zeki #5247/#5265); its latency and throughput figures mix the two topologies (1 × 32 slots before, 4 × 8 after). PAUSED 2026-11-12T17:17:39Z by logic under Umit's $30 cumulative cap for LFM3-1.7B (via zeki #5540): GPU-time cost by the site's rule (first to last answer less gaps >10 min, × $2.49/h) $27.62 at the pause; new dispatch stopped; the 32 requests then in flight were dropped (no reply stored, not scored); stored answers kept. Resume only after a verified engine check and a human OK. Cap removed by Can #5612 (2026-11-12): the $30 cap above no longer applies; the run continues on the vLLM nightly (ENGINE BOUNDARY below), target: finished within a day. ENGINE BOUNDARY (2026-11-12T17:49:37+00:00): Engine switched from llama.cpp b12020 (commit 0ec4027, official BF16 GGUF LiquidAI/LFM3-1.7B-GGUF@cd0c74c575b5c416f3011b5b4c298077f26ca8c2, 4 x 8 slots) to vLLM 0.34.0rc1.dev302+gbe47189f3 (nightly, commit be47189f37f6, CUDA 13.5 image; docker vllm/vllm-openai@sha256:7f0618da95bac2f1cc97fcd1791380d7c1a06d24a4998da45aeb7fbe6c49179e), BF16 serving LiquidAI/LFM3-1.7B@0dc4268b561499c648c7ba7c9466bf3d8d01517a (BF16 safetensors, tensors byte-identical to the GGUF), reasoning parser qwen3, 32 sequences at once, on GPU 0 of our GPU host alone. Unchanged: checkpoint, precision, chat template (sha256 f45005c5…, thinking on; prompt token ids identical), sampling (temperature 0.1, top_k 50, top_p 1, min_p 0, repetition penalty 1.1 over prompt and output in both engines: vLLM repetition_penalty in place of llama.cpp repeat_penalty with repeat_last_n over the whole context), seed 0, max_tokens 32,768, prompt and scoring. Differs: the random stream (vLLM Gumbel-max vs llama.cpp mt19937, so answers are not reproducible across the boundary), GPU kernels, and the reasoning splitter (qwen3 vs deepseek; the letter is read from the reply only). Why: llama.cpp was too slow (CPU-bound repeat penalty) for the one-day target; permission Umit via zeki #5540, cap removed by Can #5612; qualification work/lfm3_vllm_qual.md. Latency and throughput figures of this run mix the two engines. The 528 answers stored before it (last at 2026-11-12T17:15:47+00:00) came from endpoint http://127.0.0.1:18605/v1/chat/completions with parameters {"chat_template_kwargs": {"enable_thinking": true}, "max_tokens": 32768, "min_p": 0.0, "repeat_last_n": 69632, "repeat_penalty": 1.1, "seed": 0, "temperature": 0.1, "top_k": 50, "top_p": 1.0}; every answer stored after it (answer 529 onward, including re-asks of the 13 failed requests) comes from endpoint http://127.0.0.1:18605/v1/chat/completions with parameters {"chat_template_kwargs": {"enable_thinking": true}, "max_tokens": 32768, "min_p": 0.0, "repetition_penalty": 1.1, "seed": 0, "temperature": 0.1, "top_k": 50, "top_p": 1.0}. SUPERSEDED by lfm3-1.7b-vllm_en (engine mix llama.cpp + vLLM; cost/latency not comparable), Umit #6250.
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "repetition_penalty": 1.1,
  "seed": 0,
  "temperature": 0.1,
  "top_k": 50,
  "top_p": 1.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · gemma-4-e2b-it_en

Name
Gemma 4 E2B chat
Setup
Gemma 4 E2B (chat, on our GPU) · chat model
Model
gemma-4-e2b-it · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.5775 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.232 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.578. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gemma-4-e2b-it_en · first answer 2026-11-10 00:41:13 UTC · last answer 2026-11-10 00:55:08 UTC · span 835.0 s, pauses included (not charged) · charged 835.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.5775 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 30.0 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 15 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 5.1B dense · 2.3B effective · source https://huggingface.co/google/gemma-4-E2B-it · read 2026-11-10 · the card states 2.3B effective (5.1B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 5.12B (counted from the checkpoint (5,123,178,979: language model 4.65B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 5.1B with embeddings, 2.3B effective; not a mixture of experts)
Input
text, English
Endpoint
http://127.0.0.1:18606/v1/chat/completions
Started
10 Nov 2026 00:41 UTC; last answer 10 Nov 2026 00:55 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Gemma 4 E2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from google/gemma-4-E2B-it@4adf15748de9e1a0097b6274cee9c15836c41c37, vLLM reasoning parser gemma4, one RTX PRO 6000 on our GPU host alone; thinking off (template default, as the Gemma 4 31B run); sampling temperature 0, as the Gemma 4 31B chat run; no seed (as the Gemma 4 31B run); max_tokens 8,192; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 5.12B / 5.12B active / total (dense; checkpoint 5,123,178,979 incl. vision 0.17B and audio 0.31B towers; card: 5.1B with per-layer embeddings, 2.3B effective). English translations (DeepSeek + review).
Parameters
{
  "max_tokens": 8192,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · gemma-4-e2b-it_en_dontknow

Name
Gemma 4 E2B chat
Setup
Gemma 4 E2B (chat, on our GPU) · chat model · with the I don't know option F
Model
gemma-4-e2b-it · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.5782 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.232 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.578. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gemma-4-e2b-it_en_dontknow · first answer 2026-11-10 01:07:23 UTC · last answer 2026-11-10 01:21:19 UTC · span 836.0 s, pauses included (not charged) · charged 836.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.5782 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 30.6 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 22 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 5.1B dense · 2.3B effective · source https://huggingface.co/google/gemma-4-E2B-it · read 2026-11-10 · the card states 2.3B effective (5.1B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 5.12B (counted from the checkpoint (5,123,178,979: language model 4.65B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 5.1B with embeddings, 2.3B effective; not a mixture of experts)
Input
text, English
Endpoint
http://127.0.0.1:18606/v1/chat/completions
Started
10 Nov 2026 01:07 UTC; last answer 10 Nov 2026 01:21 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Gemma 4 E2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from google/gemma-4-E2B-it@4adf15748de9e1a0097b6274cee9c15836c41c37, vLLM reasoning parser gemma4, one RTX PRO 6000 on our GPU host alone; thinking off (template default, as the Gemma 4 31B run); sampling temperature 0, as the Gemma 4 31B chat run; no seed (as the Gemma 4 31B run); max_tokens 8,192; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 5.12B / 5.12B active / total (dense; checkpoint 5,123,178,979 incl. vision 0.17B and audio 0.31B towers; card: 5.1B with per-layer embeddings, 2.3B effective). English translations (DeepSeek + review).
Parameters
{
  "max_tokens": 8192,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · gemma-4-12b-it_en

Name
Gemma 4 12B Unified chat
Setup
Gemma 4 12B (chat, on our GPU) · chat model
Model
gemma-4-12b-it · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.453 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.584 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.453. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gemma-4-12b-it_en · first answer 2026-11-10 01:40:33 UTC · last answer 2026-11-10 02:15:34 UTC · span 2101.0 s, pauses included (not charged) · charged 2101.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.4532 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 28.1 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 46 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 11.95B dense · source https://huggingface.co/google/gemma-4-12B-it · read 2026-11-10 · the card states Total Parameters 11.95B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 11.96B (counted from the checkpoint (11,959,730,224: language model 11.91B, vision embedder 0.05B, audio embedder 0.002B); the card: 11.95B)
Input
text, English
Endpoint
http://127.0.0.1:18607/v1/chat/completions
Started
10 Nov 2026 01:40 UTC; last answer 10 Nov 2026 02:15 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Gemma 4 12B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from google/gemma-4-12B-it@43c9d014b231474fa7212dbb48a762566c2b255d, vLLM reasoning parser gemma4, one RTX PRO 6000 on our GPU host alone; thinking off (template default, as the Gemma 4 31B run); sampling temperature 0, as the Gemma 4 31B chat run; no seed (as the Gemma 4 31B run); max_tokens 8,192; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 11.96B / 11.96B active / total (dense; checkpoint 11,959,730,224 incl. vision embedder 0.05B and audio embedder 0.002B; card: 11.95B). English translations (DeepSeek + review).
Parameters
{
  "max_tokens": 8192,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · gemma-4-12b-it_en_dontknow

Name
Gemma 4 12B Unified chat
Setup
Gemma 4 12B (chat, on our GPU) · chat model · with the I don't know option F
Model
gemma-4-12b-it · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.376 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.552 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.376. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gemma-4-12b-it_en_dontknow · first answer 2026-11-10 02:41:03 UTC · last answer 2026-11-10 03:14:12 UTC · span 1989.0 s, pauses included (not charged) · charged 1989.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.3757 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 30.0 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 45 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 11.95B dense · source https://huggingface.co/google/gemma-4-12B-it · read 2026-11-10 · the card states Total Parameters 11.95B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 11.96B (counted from the checkpoint (11,959,730,224: language model 11.91B, vision embedder 0.05B, audio embedder 0.002B); the card: 11.95B)
Input
text, English
Endpoint
http://127.0.0.1:18607/v1/chat/completions
Started
10 Nov 2026 02:40 UTC; last answer 10 Nov 2026 03:14 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Gemma 4 12B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from google/gemma-4-12B-it@43c9d014b231474fa7212dbb48a762566c2b255d, vLLM reasoning parser gemma4, one RTX PRO 6000 on our GPU host alone; thinking off (template default, as the Gemma 4 31B run); sampling temperature 0, as the Gemma 4 31B chat run; no seed (as the Gemma 4 31B run); max_tokens 8,192; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 11.96B / 11.96B active / total (dense; checkpoint 11,959,730,224 incl. vision embedder 0.05B and audio embedder 0.002B; card: 11.95B). English translations (DeepSeek + review).
Parameters
{
  "max_tokens": 8192,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · glide-27b_en

Name
GLiDE 27B
Setup
GLiDE 27B (open) · decision model
Model
glide-27b · provider glide
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.0373 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.175 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.037 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run glide-27b_en · first answer 2026-11-10 08:52:52 UTC · last answer 2026-11-10 09:03:23 UTC · span 631.0 s, pauses included (not charged) · charged 631.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0373 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/fastino/GLiDE-27B · read 2026-11-10 · the card states a 27B model (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 26.09B (counted from the checkpoint (26,086,635,760: language model 25,624,600,064, vision encoder 460,730,096, readout 1,305,600; the same count as GLiDE v1.2, changed weights); recurrent depth: layers 24–47 run in 2 passes, their parameters counted once; the card says a 27B model)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
10 Nov 2026 08:52 UTC; last answer 10 Nov 2026 09:03 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup GLiDE 27B (open) (site run None); runner ffc307cb7; fastino/GLiDE-27B@4d32743ae96b74f0116f050d37742bbd43c04e7d (Fastino Community License); recurrent depth (decision_config midloop): layers 24–47 run in 2 passes (glide/recurrent.py), parameters counted once; on a shared Spark host (DGX Spark, GB10), served by the glide package the repository ships, one request at a time (GLIDE_MAX_BATCH=1): systemd-run --user --unit=logic-glide-27b --working-directory=~/glide-family/models/GLiDE-27B@4d32743ae96b74f0116f050d37742bbd43c04e7d -E GLIDE_MAX_BATCH=1 -E PYTHONUNBUFFERED=1 -E HF_HUB_OFFLINE=1 ~/envs/glide-family/bin/python -m glide.server --model ~/glide-family/models/GLiDE-27B@4d32743ae96b74f0116f050d37742bbd43c04e7d --host <a shared Spark host> --port 19310; env ~/envs/glide-family (CPython 3.12.13, torch 2.14.0+cu130, triton 3.8.0, transformers 5.17.0, flash-linear-attention 0.5.2); no price: no maker price for this checkpoint and no GPU-time rate for a DGX Spark in the runner
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "glide-27b",
    "--provider",
    "glide",
    "--timeout",
    "60",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "glide-27b",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "glide",
  "questions": 1125,
  "request_seconds": 624.1857343681622,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "9c12bc39fe0dfc2454b00f399128fed140f67a7822210760829c151a7ed4be56",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 390257,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19310/v1/systemone
{
 "model": "glide-27b",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · glide-27b_en_dontknow

Name
GLiDE 27B
Setup
GLiDE 27B (open) · decision model · with the I don't know option F
Model
glide-27b · provider glide
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.03896 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.183 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.039 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run glide-27b_en_dontknow · first answer 2026-11-10 09:19:53 UTC · last answer 2026-11-10 09:30:52 UTC · span 659.0 s, pauses included (not charged) · charged 659.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0390 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/fastino/GLiDE-27B · read 2026-11-10 · the card states a 27B model (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 26.09B (counted from the checkpoint (26,086,635,760: language model 25,624,600,064, vision encoder 460,730,096, readout 1,305,600; the same count as GLiDE v1.2, changed weights); recurrent depth: layers 24–47 run in 2 passes, their parameters counted once; the card says a 27B model)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
10 Nov 2026 09:19 UTC; last answer 10 Nov 2026 09:30 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup GLiDE 27B (open) (site run None); runner ffc307cb7; fastino/GLiDE-27B@4d32743ae96b74f0116f050d37742bbd43c04e7d (Fastino Community License); recurrent depth (decision_config midloop): layers 24–47 run in 2 passes (glide/recurrent.py), parameters counted once; on a shared Spark host (DGX Spark, GB10), served by the glide package the repository ships, one request at a time (GLIDE_MAX_BATCH=1): systemd-run --user --unit=logic-glide-27b --working-directory=~/glide-family/models/GLiDE-27B@4d32743ae96b74f0116f050d37742bbd43c04e7d -E GLIDE_MAX_BATCH=1 -E PYTHONUNBUFFERED=1 -E HF_HUB_OFFLINE=1 ~/envs/glide-family/bin/python -m glide.server --model ~/glide-family/models/GLiDE-27B@4d32743ae96b74f0116f050d37742bbd43c04e7d --host <a shared Spark host> --port 19310; env ~/envs/glide-family (CPython 3.12.13, torch 2.14.0+cu130, triton 3.8.0, transformers 5.17.0, flash-linear-attention 0.5.2); no price: no maker price for this checkpoint and no GPU-time rate for a DGX Spark in the runner
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "glide-27b",
    "--provider",
    "glide",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "glide-27b",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "glide",
  "questions": 1125,
  "request_seconds": 651.2592161455541,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "c50f25b0d0f35a8be97b8b31502b72bfd11b7810aa52a2f020ee19a3b6ba49cb",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 424007,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19310/v1/systemone
{
 "model": "glide-27b",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · glide-9b_en

Name
GLiDE 9B
Setup
GLiDE 9B (open) · decision model
Model
glide-9b · provider glide
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.01496 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0703 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.015 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run glide-9b_en · first answer 2026-11-10 11:45:37 UTC · last answer 2026-11-10 11:49:50 UTC · span 253.0 s, pauses included (not charged) · charged 253.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0150 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 7.94B dense · source https://huggingface.co/fastino/GLiDE-9B/tree/ab54df5acba0eebf7df45560ffdaacd64e32df3b · read 2026-11-10 · counted from the checkpoint (7,937,729,024: language model 7,936,684,544, readout 1,044,480; text only); recurrent depth: layers 12–23 run in 2 passes, their parameters counted once; the card: 7.94B (1.04M decision readout)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
10 Nov 2026 11:45 UTC; last answer 10 Nov 2026 11:49 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup GLiDE 9B (open) (site run None); runner 9ba1b409d; fastino/GLiDE-9B@ab54df5acba0eebf7df45560ffdaacd64e32df3b (Fastino Community License); recurrent depth (decision_config midloop): layers 12–23 run in 2 passes (glide/recurrent.py), parameters counted once; on a shared Spark host (DGX Spark, GB10), served by the glide package the repository ships, one request at a time (GLIDE_MAX_BATCH=1): systemd-run --user --unit=logic-glide-9b --working-directory=~/glide-family/models/GLiDE-9B@ab54df5acba0eebf7df45560ffdaacd64e32df3b -E GLIDE_MAX_BATCH=1 -E PYTHONUNBUFFERED=1 -E HF_HUB_OFFLINE=1 ~/envs/glide-family/bin/python -m glide.server --model ~/glide-family/models/GLiDE-9B@ab54df5acba0eebf7df45560ffdaacd64e32df3b --host <a shared Spark host> --port 19320; env ~/envs/glide-family (CPython 3.12.13, torch 2.14.0+cu130, triton 3.8.0, transformers 5.17.0, flash-linear-attention 0.5.2); no price: no maker price for this checkpoint and no GPU-time rate for a DGX Spark in the runner
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "glide-9b",
    "--provider",
    "glide",
    "--timeout",
    "60",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "3b14502cefed852b2c6ac3a9139c7574549374b6",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "glide-9b",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "glide",
  "questions": 1125,
  "request_seconds": 237.44628156535327,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "daf7d0af79d1b0ad39341ddf33ec68842d54627efbcb836ed0e3236c1469cc01",
  "runner": "benchmark_ollama.py",
  "runner_commit": "9ba1b409d68af1a8b3c846f54993e438b9db212a",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 390257,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19320/v1/systemone
{
 "model": "glide-9b",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · glide-9b_en_dontknow

Name
GLiDE 9B
Setup
GLiDE 9B (open) · decision model · with the I don't know option F
Model
glide-9b · provider glide
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.01537 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0722 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.015 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run glide-9b_en_dontknow · first answer 2026-11-10 11:56:27 UTC · last answer 2026-11-10 12:00:47 UTC · span 260.0 s, pauses included (not charged) · charged 260.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0154 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 7.94B dense · source https://huggingface.co/fastino/GLiDE-9B/tree/ab54df5acba0eebf7df45560ffdaacd64e32df3b · read 2026-11-10 · counted from the checkpoint (7,937,729,024: language model 7,936,684,544, readout 1,044,480; text only); recurrent depth: layers 12–23 run in 2 passes, their parameters counted once; the card: 7.94B (1.04M decision readout)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
10 Nov 2026 11:56 UTC; last answer 10 Nov 2026 12:00 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup GLiDE 9B (open) (site run None); runner 9ba1b409d; fastino/GLiDE-9B@ab54df5acba0eebf7df45560ffdaacd64e32df3b (Fastino Community License); recurrent depth (decision_config midloop): layers 12–23 run in 2 passes (glide/recurrent.py), parameters counted once; on a shared Spark host (DGX Spark, GB10), served by the glide package the repository ships, one request at a time (GLIDE_MAX_BATCH=1): systemd-run --user --unit=logic-glide-9b --working-directory=~/glide-family/models/GLiDE-9B@ab54df5acba0eebf7df45560ffdaacd64e32df3b -E GLIDE_MAX_BATCH=1 -E PYTHONUNBUFFERED=1 -E HF_HUB_OFFLINE=1 ~/envs/glide-family/bin/python -m glide.server --model ~/glide-family/models/GLiDE-9B@ab54df5acba0eebf7df45560ffdaacd64e32df3b --host <a shared Spark host> --port 19320; env ~/envs/glide-family (CPython 3.12.13, torch 2.14.0+cu130, triton 3.8.0, transformers 5.17.0, flash-linear-attention 0.5.2); no price: no maker price for this checkpoint and no GPU-time rate for a DGX Spark in the runner
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "glide-9b",
    "--provider",
    "glide",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "3b14502cefed852b2c6ac3a9139c7574549374b6",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "glide-9b",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "glide",
  "questions": 1125,
  "request_seconds": 245.89707675614045,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "9f79ca75df8baa79f94fb08cbaaf294c278b4abc89ab9934c45310c8f1820698",
  "runner": "benchmark_ollama.py",
  "runner_commit": "9ba1b409d68af1a8b3c846f54993e438b9db212a",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 424007,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19320/v1/systemone
{
 "model": "glide-9b",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · glide-2b_en

Name
GLiDE 2B
Setup
GLiDE 2B (open) · decision model
Model
glide-2b · provider glide
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.002542 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0119 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.003 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run glide-2b_en · first answer 2026-11-10 12:41:00 UTC · last answer 2026-11-10 12:41:43 UTC · span 43.0 s, pauses included (not charged) · charged 43.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0025 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 1.88B dense · source https://huggingface.co/fastino/GLiDE-2B/tree/00f122c120cdeed82284106cb77d9bab5a50ad16 · read 2026-11-10 · counted from the checkpoint (1,882,347,328: language model 1,881,825,088, readout 522,240; text only); the card: 1.88B
Input
text, English
Endpoint
internal address on a shared Spark host
Started
10 Nov 2026 12:41 UTC; last answer 10 Nov 2026 12:41 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup GLiDE 2B (open) (site run None); runner 9ba1b409d; fastino/GLiDE-2B@00f122c120cdeed82284106cb77d9bab5a50ad16 (Fastino Community License) on a shared Spark host (DGX Spark, GB10), served by the glide package the repository ships, one request at a time (GLIDE_MAX_BATCH=1): systemd-run --user --unit=logic-glide-2b --working-directory=~/glide-family/models/GLiDE-2B@00f122c120cdeed82284106cb77d9bab5a50ad16 -E GLIDE_MAX_BATCH=1 -E PYTHONUNBUFFERED=1 -E HF_HUB_OFFLINE=1 ~/envs/glide-family/bin/python -m glide.server --model ~/glide-family/models/GLiDE-2B@00f122c120cdeed82284106cb77d9bab5a50ad16 --host <a shared Spark host> --port 19330; env ~/envs/glide-family (CPython 3.12.13, torch 2.14.0+cu130, triton 3.8.0, transformers 5.17.0, flash-linear-attention 0.5.2); no price: no maker price for this checkpoint and no GPU-time rate for a DGX Spark in the runner
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "glide-2b",
    "--provider",
    "glide",
    "--timeout",
    "60",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "3b14502cefed852b2c6ac3a9139c7574549374b6",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "glide-2b",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "glide",
  "questions": 1125,
  "request_seconds": 39.59819752917974,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "f8cc38d7a78a04e36ab5cb2eccff884cc6d8588320afe701668565379f2f30dd",
  "runner": "benchmark_ollama.py",
  "runner_commit": "9ba1b409d68af1a8b3c846f54993e438b9db212a",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 390257,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19330/v1/systemone
{
 "model": "glide-2b",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · glide-2b_en_dontknow

Name
GLiDE 2B
Setup
GLiDE 2B (open) · decision model · with the I don't know option F
Model
glide-2b · provider glide
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.002601 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0122 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.003 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run glide-2b_en_dontknow · first answer 2026-11-10 12:42:53 UTC · last answer 2026-11-10 12:43:37 UTC · span 44.0 s, pauses included (not charged) · charged 44.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0026 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 1.88B dense · source https://huggingface.co/fastino/GLiDE-2B/tree/00f122c120cdeed82284106cb77d9bab5a50ad16 · read 2026-11-10 · counted from the checkpoint (1,882,347,328: language model 1,881,825,088, readout 522,240; text only); the card: 1.88B
Input
text, English
Endpoint
internal address on a shared Spark host
Started
10 Nov 2026 12:42 UTC; last answer 10 Nov 2026 12:43 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup GLiDE 2B (open) (site run None); runner 9ba1b409d; fastino/GLiDE-2B@00f122c120cdeed82284106cb77d9bab5a50ad16 (Fastino Community License) on a shared Spark host (DGX Spark, GB10), served by the glide package the repository ships, one request at a time (GLIDE_MAX_BATCH=1): systemd-run --user --unit=logic-glide-2b --working-directory=~/glide-family/models/GLiDE-2B@00f122c120cdeed82284106cb77d9bab5a50ad16 -E GLIDE_MAX_BATCH=1 -E PYTHONUNBUFFERED=1 -E HF_HUB_OFFLINE=1 ~/envs/glide-family/bin/python -m glide.server --model ~/glide-family/models/GLiDE-2B@00f122c120cdeed82284106cb77d9bab5a50ad16 --host <a shared Spark host> --port 19330; env ~/envs/glide-family (CPython 3.12.13, torch 2.14.0+cu130, triton 3.8.0, transformers 5.17.0, flash-linear-attention 0.5.2); no price: no maker price for this checkpoint and no GPU-time rate for a DGX Spark in the runner
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "glide-2b",
    "--provider",
    "glide",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "3b14502cefed852b2c6ac3a9139c7574549374b6",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "glide-2b",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "glide",
  "questions": 1125,
  "request_seconds": 41.00952412612969,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "58d5c110edc47ac5e5cd4a6ced516fd1658ca15b7e341364f454ee08e1ef6f73",
  "runner": "benchmark_ollama.py",
  "runner_commit": "9ba1b409d68af1a8b3c846f54993e438b9db212a",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 424007,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19330/v1/systemone
{
 "model": "glide-2b",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · glide-0.8b_en

Name
GLiDE 0.8B
Setup
GLiDE 0.8B (open) · decision model
Model
glide-0.8b · provider glide
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.001773 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.00833 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.002 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run glide-0.8b_en · first answer 2026-11-10 12:52:10 UTC · last answer 2026-11-10 12:52:40 UTC · span 30.0 s, pauses included (not charged) · charged 30.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0018 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 0.75B dense · source https://huggingface.co/fastino/GLiDE-0.8B/tree/9eacdeec79fa48ec52a8d6f54b331376b4bbfaf0 · read 2026-11-10 · counted from the checkpoint (752,654,144: language model 752,393,024, readout 261,120; text only); the card: 0.75B
Input
text, English
Endpoint
internal address on a shared Spark host
Started
10 Nov 2026 12:52 UTC; last answer 10 Nov 2026 12:52 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup GLiDE 0.8B (open) (site run None); runner 9ba1b409d; fastino/GLiDE-0.8B@9eacdeec79fa48ec52a8d6f54b331376b4bbfaf0 (Fastino Community License) on a shared Spark host (DGX Spark, GB10), served by the glide package the repository ships, one request at a time (GLIDE_MAX_BATCH=1): systemd-run --user --unit=logic-glide-0.8b --working-directory=~/glide-family/models/GLiDE-0.8B@9eacdeec79fa48ec52a8d6f54b331376b4bbfaf0 -E GLIDE_MAX_BATCH=1 -E PYTHONUNBUFFERED=1 -E HF_HUB_OFFLINE=1 ~/envs/glide-family/bin/python -m glide.server --model ~/glide-family/models/GLiDE-0.8B@9eacdeec79fa48ec52a8d6f54b331376b4bbfaf0 --host <a shared Spark host> --port 19340; env ~/envs/glide-family (CPython 3.12.13, torch 2.14.0+cu130, triton 3.8.0, transformers 5.17.0, flash-linear-attention 0.5.2); no price: no maker price for this checkpoint and no GPU-time rate for a DGX Spark in the runner
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "glide-0.8b",
    "--provider",
    "glide",
    "--timeout",
    "60",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "3b14502cefed852b2c6ac3a9139c7574549374b6",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "glide-0.8b",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "glide",
  "questions": 1125,
  "request_seconds": 26.96682221913943,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "ded7d8d4c796fc903e149c91dedb51024436dfae20b10794118e2a52308b25c0",
  "runner": "benchmark_ollama.py",
  "runner_commit": "9ba1b409d68af1a8b3c846f54993e438b9db212a",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 390257,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19340/v1/systemone
{
 "model": "glide-0.8b",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · glide-0.8b_en_dontknow

Name
GLiDE 0.8B
Setup
GLiDE 0.8B (open) · decision model · with the I don't know option F
Model
glide-0.8b · provider glide
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.001714 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.00806 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.002 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run glide-0.8b_en_dontknow · first answer 2026-11-10 12:53:35 UTC · last answer 2026-11-10 12:54:04 UTC · span 29.0 s, pauses included (not charged) · charged 29.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0017 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 0.75B dense · source https://huggingface.co/fastino/GLiDE-0.8B/tree/9eacdeec79fa48ec52a8d6f54b331376b4bbfaf0 · read 2026-11-10 · counted from the checkpoint (752,654,144: language model 752,393,024, readout 261,120; text only); the card: 0.75B
Input
text, English
Endpoint
internal address on a shared Spark host
Started
10 Nov 2026 12:53 UTC; last answer 10 Nov 2026 12:54 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup GLiDE 0.8B (open) (site run None); runner 9ba1b409d; fastino/GLiDE-0.8B@9eacdeec79fa48ec52a8d6f54b331376b4bbfaf0 (Fastino Community License) on a shared Spark host (DGX Spark, GB10), served by the glide package the repository ships, one request at a time (GLIDE_MAX_BATCH=1): systemd-run --user --unit=logic-glide-0.8b --working-directory=~/glide-family/models/GLiDE-0.8B@9eacdeec79fa48ec52a8d6f54b331376b4bbfaf0 -E GLIDE_MAX_BATCH=1 -E PYTHONUNBUFFERED=1 -E HF_HUB_OFFLINE=1 ~/envs/glide-family/bin/python -m glide.server --model ~/glide-family/models/GLiDE-0.8B@9eacdeec79fa48ec52a8d6f54b331376b4bbfaf0 --host <a shared Spark host> --port 19340; env ~/envs/glide-family (CPython 3.12.13, torch 2.14.0+cu130, triton 3.8.0, transformers 5.17.0, flash-linear-attention 0.5.2); no price: no maker price for this checkpoint and no GPU-time rate for a DGX Spark in the runner
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "glide-0.8b",
    "--provider",
    "glide",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "3b14502cefed852b2c6ac3a9139c7574549374b6",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "glide-0.8b",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "glide",
  "questions": 1125,
  "request_seconds": 26.82365612903959,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "215285816a2372b22ff8958262e16faa097ff17a3afa83aa8e5d34aeef597f04",
  "runner": "benchmark_ollama.py",
  "runner_commit": "9ba1b409d68af1a8b3c846f54993e438b9db212a",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 424007,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19340/v1/systemone
{
 "model": "glide-0.8b",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · glide-0.4b_en

Name
GLiDE 0.4B
Setup
GLiDE 0.4B (open) · decision model
Model
glide-0.4b · provider glide
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.001064 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.005 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.001 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run glide-0.4b_en · first answer 2026-11-10 13:00:38 UTC · last answer 2026-11-10 13:00:56 UTC · span 18.0 s, pauses included (not charged) · charged 18.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0011 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.8 in flight (sum of request latencies / active time)
Model size
Parameters: 0.40B dense · source https://huggingface.co/fastino/GLiDE-0.4B/tree/6bda7ab91aeb36dcad6d27828a8871e548c7fc71 · read 2026-11-10 · counted from the checkpoint (395,042,816: encoder 394,781,696, readout 261,120; an encoder, ModernBERT architecture); the card: 0.40B
Input
text, English
Endpoint
internal address on a shared Spark host
Started
10 Nov 2026 13:00 UTC; last answer 10 Nov 2026 13:00 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup GLiDE 0.4B (open) (site run None); runner 9ba1b409d; fastino/GLiDE-0.4B@6bda7ab91aeb36dcad6d27828a8871e548c7fc71 (Fastino Community License) on a shared Spark host (DGX Spark, GB10), served by the glide package the repository ships, one request at a time (GLIDE_MAX_BATCH=1): systemd-run --user --unit=logic-glide-0.4b --working-directory=~/glide-family/models/GLiDE-0.4B@6bda7ab91aeb36dcad6d27828a8871e548c7fc71 -E GLIDE_MAX_BATCH=1 -E PYTHONUNBUFFERED=1 -E HF_HUB_OFFLINE=1 ~/envs/glide-family/bin/python -m glide.server --model ~/glide-family/models/GLiDE-0.4B@6bda7ab91aeb36dcad6d27828a8871e548c7fc71 --host <a shared Spark host> --port 19350; env ~/envs/glide-family (CPython 3.12.13, torch 2.14.0+cu130, triton 3.8.0, transformers 5.17.0, flash-linear-attention 0.5.2); no price: no maker price for this checkpoint and no GPU-time rate for a DGX Spark in the runner
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "glide-0.4b",
    "--provider",
    "glide",
    "--timeout",
    "60",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "3b14502cefed852b2c6ac3a9139c7574549374b6",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "glide-0.4b",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "glide",
  "questions": 1125,
  "request_seconds": 14.922689840517705,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "a988663f1e5da53616a4cfb370fba050520ad9c1920911420879dda0ede71a75",
  "runner": "benchmark_ollama.py",
  "runner_commit": "9ba1b409d68af1a8b3c846f54993e438b9db212a",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 415137,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19350/v1/systemone
{
 "model": "glide-0.4b",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · glide-0.4b_en_dontknow

Name
GLiDE 0.4B
Setup
GLiDE 0.4B (open) · decision model · with the I don't know option F
Model
glide-0.4b · provider glide
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.001123 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.00528 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.001 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run glide-0.4b_en_dontknow · first answer 2026-11-10 13:01:32 UTC · last answer 2026-11-10 13:01:51 UTC · span 19.0 s, pauses included (not charged) · charged 19.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0011 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.8 in flight (sum of request latencies / active time)
Model size
Parameters: 0.40B dense · source https://huggingface.co/fastino/GLiDE-0.4B/tree/6bda7ab91aeb36dcad6d27828a8871e548c7fc71 · read 2026-11-10 · counted from the checkpoint (395,042,816: encoder 394,781,696, readout 261,120; an encoder, ModernBERT architecture); the card: 0.40B
Input
text, English
Endpoint
internal address on a shared Spark host
Started
10 Nov 2026 13:01 UTC; last answer 10 Nov 2026 13:01 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup GLiDE 0.4B (open) (site run None); runner 9ba1b409d; fastino/GLiDE-0.4B@6bda7ab91aeb36dcad6d27828a8871e548c7fc71 (Fastino Community License) on a shared Spark host (DGX Spark, GB10), served by the glide package the repository ships, one request at a time (GLIDE_MAX_BATCH=1): systemd-run --user --unit=logic-glide-0.4b --working-directory=~/glide-family/models/GLiDE-0.4B@6bda7ab91aeb36dcad6d27828a8871e548c7fc71 -E GLIDE_MAX_BATCH=1 -E PYTHONUNBUFFERED=1 -E HF_HUB_OFFLINE=1 ~/envs/glide-family/bin/python -m glide.server --model ~/glide-family/models/GLiDE-0.4B@6bda7ab91aeb36dcad6d27828a8871e548c7fc71 --host <a shared Spark host> --port 19350; env ~/envs/glide-family (CPython 3.12.13, torch 2.14.0+cu130, triton 3.8.0, transformers 5.17.0, flash-linear-attention 0.5.2); no price: no maker price for this checkpoint and no GPU-time rate for a DGX Spark in the runner
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "glide-0.4b",
    "--provider",
    "glide",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "3b14502cefed852b2c6ac3a9139c7574549374b6",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "glide-0.4b",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "glide",
  "questions": 1125,
  "request_seconds": 15.786791075923247,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "9581ebafe9ab2b7fc4379f738624883a9107c9d864f207842737b5cda7ec81aa",
  "runner": "benchmark_ollama.py",
  "runner_commit": "9ba1b409d68af1a8b3c846f54993e438b9db212a",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 450012,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19350/v1/systemone
{
 "model": "glide-0.4b",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · gpt-6-luna_en

Name
GPT-6 Luna max · chat
Setup
GPT-6 Luna · chat model
Model
openai/gpt-6-luna · provider openrouter
GPU
—
Cost
not measured
Answer time
run gpt-6-luna_en · first answer 2026-11-10 14:04:26 UTC · last answer 2026-11-10 14:12:15 UTC · span 469.0 s, pauses included (not charged) · charged 469.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 7.0 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 1 answer cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: not disclosed by maker (closed weights) · source https://openrouter.ai/openai/gpt-6-luna · read 2026-11-10
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
10 Nov 2026 14:04 UTC; last answer 10 Nov 2026 14:12 UTC
Progress
55 answered of 1,125 asked; 0 request errors; paused for now at 55 of 1,125 (zeki #4129, 2026-11-10T14:13Z)
Notes
GPT-6 Luna through OpenRouter (openai/gpt-6-luna), provider pinned to openai, fallbacks off; reasoning {"effort": "max"}; temperature 1.0; max_tokens 32768; one question per request; parameters not disclosed (closed weights). English translations. PAUSED (zeki #4129, 2026-11-10T14:13Z): skipped for now, stopped at 55 answers; paid smoke stopped, no scoped budget yet; kept, not ranked
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "openai"
    ]
  },
  "reasoning": {
    "effort": "max"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · claude-haiku-5.5_en

Name
Claude Haiku 5.5 max · chat
Setup
Claude Haiku 5.5 · chat model
Model
anthropic/claude-haiku-5.5 · provider openrouter
GPU
—
Cost
not measured
Answer time
run claude-haiku-5.5_en · first answer 2026-11-10 14:05:06 UTC · last answer 2026-11-10 14:12:08 UTC · span 422.0 s, pauses included (not charged) · charged 422.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 7.7 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 15 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: not disclosed by maker (closed weights) · source https://openrouter.ai/anthropic/claude-haiku-5.5 · read 2026-11-10
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
10 Nov 2026 14:05 UTC; last answer 10 Nov 2026 14:12 UTC
Progress
47 answered of 1,125 asked; 0 request errors; paused for now at 47 of 1,125 (zeki #4129, 2026-11-10T14:13Z)
Notes
Claude Haiku 5.5 through OpenRouter (anthropic/claude-haiku-5.5), provider pinned to anthropic, fallbacks off; reasoning {"effort": "max"}; temperature 1.0; max_tokens 32768; one question per request; parameters not disclosed (closed weights). English translations. PAUSED (zeki #4129, 2026-11-10T14:13Z): skipped for now, stopped at 47 answers; paid smoke stopped, no scoped budget yet; kept, not ranked
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "anthropic"
    ]
  },
  "reasoning": {
    "effort": "max"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · muse-ember_en

Name
Muse Ember max · chat
Setup
Muse Ember · chat model
Model
meta/muse-ember · provider openrouter
GPU
—
Cost
$2.048 for the run · provider price × reported usage
Answer time
run muse-ember_en · first answer 2026-11-10 14:05:03 UTC · last answer 2026-11-10 14:51:54 UTC · span 2811.0 s, pauses included (not charged) · charged 1000.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 7.6 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 20 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: not disclosed by maker (closed weights) · source https://openrouter.ai/meta/muse-ember · read 2026-11-10
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
10 Nov 2026 14:05 UTC; last answer 10 Nov 2026 14:51 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Muse Ember through OpenRouter (meta/muse-ember), provider pinned to meta, fallbacks off; reasoning {"effort": "max"}; temperature 1.0; max_tokens 32768; one question per request; parameters not disclosed (closed weights). English translations.
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "meta"
    ]
  },
  "reasoning": {
    "effort": "max"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · muse-ember_en_dontknow

Name
Muse Ember max · chat
Setup
Muse Ember · chat model · with the I don't know option F
Model
meta/muse-ember · provider openrouter
GPU
—
Cost
$2.051 for the run · provider price × reported usage
Answer time
run muse-ember_en_dontknow · first answer 2026-11-10 14:51:56 UTC · last answer 2026-11-10 15:08:06 UTC · span 970.0 s, pauses included (not charged) · charged 970.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 7.9 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 13 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: not disclosed by maker (closed weights) · source https://openrouter.ai/meta/muse-ember · read 2026-11-10
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
10 Nov 2026 14:51 UTC; last answer 10 Nov 2026 15:08 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Muse Ember through OpenRouter (meta/muse-ember), provider pinned to meta, fallbacks off; reasoning {"effort": "max"}; temperature 1.0; max_tokens 32768; one question per request; parameters not disclosed (closed weights). English translations.
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "meta"
    ]
  },
  "reasoning": {
    "effort": "max"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · knowline-31b_en

Name
KnowLine-31B-Gen1.5
Setup
KnowLine 31B Gen1.5 · decision model
Model
PelaAI/KnowLine-31B-Gen1.5 · provider knowline
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.06911 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.325 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.069 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run knowline-31b_en · first answer 2026-11-10 15:26:45 UTC · last answer 2026-11-10 15:46:14 UTC · span 1169.0 s, pauses included (not charged) · charged 1169.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0691 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 31B dense · source https://huggingface.co/PelaAI/KnowLine-31B-Gen1.5 · read 2026-11-10 · the card states about 31B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 31.27B (counted from the checkpoint (31,273,088,876: language model 30,697,345,340, vision tower 575,743,536); a fine-tune of Gemma 4 31B IT; the card: about 31B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
10 Nov 2026 15:26 UTC; last answer 10 Nov 2026 15:46 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup KnowLine 31B Gen1.5 (site run None); runner 9ba1b409d; PelaAI/KnowLine-31B-Gen1.5@8e56aa1b236a6dd1064dcfd22ba1e5e360e4c140 (Apache-2.0 weights, MIT code; Gemma 4 31B IT fine-tune, readout temperature folded into the weights) behind the card's serve_knowline.sh: SGLang 0.5.21 (BF16, --enable-fp32-lm-head) on port+1000 and the repository's knowline_server.py front end (read twice for prompts up to 3,000 tokens): cd ~/decision-models/KnowLine-31B-Gen1.5@8e56aa1b236a6dd1064dcfd22ba1e5e360e4c140 && PATH=~/envs/knowline-sglang/bin:$PATH HOST=0.0.0.0 bash serve_knowline.sh ~/decision-models/KnowLine-31B-Gen1.5@8e56aa1b236a6dd1064dcfd22ba1e5e360e4c140 0 19360 (MEM default 0.85), env ~/envs/knowline-sglang (sglang==0.5.21, transformers==5.12.1, torch 2.13.0+cu130); vision on (the server's default); bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'knowline' (System One at /v1/systemone; model field ignored)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "PelaAI/KnowLine-31B-Gen1.5",
    "--provider",
    "knowline",
    "--timeout",
    "120",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "3b14502cefed852b2c6ac3a9139c7574549374b6",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "PelaAI/KnowLine-31B-Gen1.5",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "knowline",
  "questions": 1125,
  "request_seconds": 1160.884435379412,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "18d06a40fc4cd18fbf4c04e53ea4a1ef0783b8b032e4bda8ffb0a870e5517e9b",
  "runner": "benchmark_ollama.py",
  "runner_commit": "9ba1b409d68af1a8b3c846f54993e438b9db212a",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 836143,
    "output_tokens": 1125
  }
}
Prompt template
POST http://spark2:19360/v1/systemone
{
 "model": "PelaAI/KnowLine-31B-Gen1.5",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · knowline-31b_en_dontknow

Name
KnowLine-31B-Gen1.5
Setup
KnowLine 31B Gen1.5 · decision model · with the I don't know option F
Model
PelaAI/KnowLine-31B-Gen1.5 · provider knowline
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.07413 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.348 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.074 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run knowline-31b_en_dontknow · first answer 2026-11-10 16:15:37 UTC · last answer 2026-11-10 16:36:31 UTC · span 1254.0 s, pauses included (not charged) · charged 1254.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0741 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 31B dense · source https://huggingface.co/PelaAI/KnowLine-31B-Gen1.5 · read 2026-11-10 · the card states about 31B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 31.27B (counted from the checkpoint (31,273,088,876: language model 30,697,345,340, vision tower 575,743,536); a fine-tune of Gemma 4 31B IT; the card: about 31B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
10 Nov 2026 16:15 UTC; last answer 10 Nov 2026 16:36 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup KnowLine 31B Gen1.5 (site run None); runner 9ba1b409d; PelaAI/KnowLine-31B-Gen1.5@8e56aa1b236a6dd1064dcfd22ba1e5e360e4c140 (Apache-2.0 weights, MIT code; Gemma 4 31B IT fine-tune, readout temperature folded into the weights) behind the card's serve_knowline.sh: SGLang 0.5.21 (BF16, --enable-fp32-lm-head) on port+1000 and the repository's knowline_server.py front end (read twice for prompts up to 3,000 tokens): cd ~/decision-models/KnowLine-31B-Gen1.5@8e56aa1b236a6dd1064dcfd22ba1e5e360e4c140 && PATH=~/envs/knowline-sglang/bin:$PATH HOST=0.0.0.0 bash serve_knowline.sh ~/decision-models/KnowLine-31B-Gen1.5@8e56aa1b236a6dd1064dcfd22ba1e5e360e4c140 0 19360 (MEM default 0.85), env ~/envs/knowline-sglang (sglang==0.5.21, transformers==5.12.1, torch 2.13.0+cu130); vision on (the server's default); bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'knowline' (System One at /v1/systemone; model field ignored)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "PelaAI/KnowLine-31B-Gen1.5",
    "--provider",
    "knowline",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "3b14502cefed852b2c6ac3a9139c7574549374b6",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "PelaAI/KnowLine-31B-Gen1.5",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "knowline",
  "questions": 1125,
  "request_seconds": 1246.8962870760588,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "35de90853b816c59ee21aedfa8b3a57e042234e532fec67d36b5c2368db094c1",
  "runner": "benchmark_ollama.py",
  "runner_commit": "9ba1b409d68af1a8b3c846f54993e438b9db212a",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 910393,
    "output_tokens": 1125
  }
}
Prompt template
POST http://spark2:19360/v1/systemone
{
 "model": "PelaAI/KnowLine-31B-Gen1.5",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · knowline-4b_en

Name
KnowLine 4B Gen5.5
Setup
KnowLine 4B Gen5.5 · decision model
Model
PelaAI/KnowLine-4B-Gen5.5 · provider knowline
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.006621 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0311 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.007 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run knowline-4b_en · first answer 2026-11-11 11:03:25 UTC · last answer 2026-11-11 11:05:17 UTC · span 112.0 s, pauses included (not charged) · charged 112.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0066 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 4B dense · source https://huggingface.co/PelaAI/KnowLine-4B-Gen5.5 · read 2026-11-10 · the card states 4B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 4.66B (counted from the checkpoint (4,659,865,088: language model 4,205,751,296, MTP layer 120,599,552 (model-extra-from-base.safetensors), vision tower 333,514,240); from Qwen3.5-4B; the card: 4B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
11 Nov 2026 11:03 UTC; last answer 11 Nov 2026 11:05 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup KnowLine 4B Gen5.5 (site run None); runner 5fb51788d; PelaAI/KnowLine-4B-Gen5.5@779944d2cc8ffa19cef386aee83c83bcb967ade2 (Apache-2.0 weights, MIT code; Qwen3.5-4B LoRA average, temperature folded into the weights) behind the card's serve_knowline.sh: SGLang 0.5.21 (FP8 at load) on port+1000 and knowline_server.py (read twice up to 3,000 tokens): cd ~/decision-models/KnowLine-4B-Gen5.5@779944d2cc8ffa19cef386aee83c83bcb967ade2 && PATH=~/envs/knowline-sglang/bin:$PATH HOST=0.0.0.0 bash serve_knowline.sh ~/decision-models/KnowLine-4B-Gen5.5@779944d2cc8ffa19cef386aee83c83bcb967ade2 0 19400 (MEM default 0.72), env ~/envs/knowline-sglang; vision on (the server's default); bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'knowline'
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "PelaAI/KnowLine-4B-Gen5.5",
    "--provider",
    "knowline",
    "--timeout",
    "60",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "b55c22654bc72e59b7a430a92a37731267ddb6de",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "PelaAI/KnowLine-4B-Gen5.5",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "knowline",
  "questions": 1125,
  "request_seconds": 109.76755630149273,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "0d331bb8651b9443f00a82549280c2b9f2b2c82f013e4cd2d9bfe5a390b3b769",
  "runner": "benchmark_ollama.py",
  "runner_commit": "5fb51788d3992285ed8f0cf76fd6e8716353b0d9",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 836764,
    "output_tokens": 1125
  }
}
Prompt template
POST http://spark2:19400/v1/systemone
{
 "model": "PelaAI/KnowLine-4B-Gen5.5",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · knowline-4b_en_dontknow

Name
KnowLine 4B Gen5.5
Setup
KnowLine 4B Gen5.5 · decision model · with the I don't know option F
Model
PelaAI/KnowLine-4B-Gen5.5 · provider knowline
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.006798 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0319 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.007 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run knowline-4b_en_dontknow · first answer 2026-11-11 11:08:13 UTC · last answer 2026-11-11 11:10:08 UTC · span 115.0 s, pauses included (not charged) · charged 115.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0068 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 4B dense · source https://huggingface.co/PelaAI/KnowLine-4B-Gen5.5 · read 2026-11-10 · the card states 4B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 4.66B (counted from the checkpoint (4,659,865,088: language model 4,205,751,296, MTP layer 120,599,552 (model-extra-from-base.safetensors), vision tower 333,514,240); from Qwen3.5-4B; the card: 4B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
11 Nov 2026 11:08 UTC; last answer 11 Nov 2026 11:10 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup KnowLine 4B Gen5.5 (site run None); runner 5fb51788d; PelaAI/KnowLine-4B-Gen5.5@779944d2cc8ffa19cef386aee83c83bcb967ade2 (Apache-2.0 weights, MIT code; Qwen3.5-4B LoRA average, temperature folded into the weights) behind the card's serve_knowline.sh: SGLang 0.5.21 (FP8 at load) on port+1000 and knowline_server.py (read twice up to 3,000 tokens): cd ~/decision-models/KnowLine-4B-Gen5.5@779944d2cc8ffa19cef386aee83c83bcb967ade2 && PATH=~/envs/knowline-sglang/bin:$PATH HOST=0.0.0.0 bash serve_knowline.sh ~/decision-models/KnowLine-4B-Gen5.5@779944d2cc8ffa19cef386aee83c83bcb967ade2 0 19400 (MEM default 0.72), env ~/envs/knowline-sglang; vision on (the server's default); bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'knowline'
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "PelaAI/KnowLine-4B-Gen5.5",
    "--provider",
    "knowline",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "b55c22654bc72e59b7a430a92a37731267ddb6de",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "PelaAI/KnowLine-4B-Gen5.5",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "knowline",
  "questions": 1125,
  "request_seconds": 109.44185549329268,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "a2d245c4e176679cdee634dc41dfbca41243af91872a3bd5a4fc36084de7fef2",
  "runner": "benchmark_ollama.py",
  "runner_commit": "5fb51788d3992285ed8f0cf76fd6e8716353b0d9",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 904264,
    "output_tokens": 1125
  }
}
Prompt template
POST http://spark2:19400/v1/systemone
{
 "model": "PelaAI/KnowLine-4B-Gen5.5",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · startlux-2b_en

Name
StartLux-Decision-v1.1-2B-Instant
Setup
StartLux-Decision v1.1 2B · decision model
Model
startlux-models/StartLux-Decision-v1.1-2B-Instant · provider startlux
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.002838 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0133 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.003 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run startlux-2b_en · first answer 2026-11-11 12:23:32 UTC · last answer 2026-11-11 12:24:20 UTC · span 48.0 s, pauses included (not charged) · charged 48.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0028 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 2B dense · source https://huggingface.co/startlux-models/StartLux-Decision-v1.1-2B-Instant · read 2026-11-10 · the card states 2B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 2.27B (counted from the checkpoint (2,274,069,824: language model 1,881,825,088, MTP layer 60,828,160, vision tower 331,416,576; dense, no experts); from Qwen3.5-2B; the card: 2B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
11 Nov 2026 12:23 UTC; last answer 11 Nov 2026 12:24 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup StartLux-Decision v1.1 2B (site run None); runner 5fb51788d; startlux-models/StartLux-Decision-v1.1-2B-Instant@f1e6cd147dda82c9ffa3e8e725d5c1f175566fb1 (weights CC BY-NC 4.0, non-commercial: run for this non-commercial benchmark; code Apache-2.0; Qwen3.5-2B, per-type temperatures from decision_config.json (choice 1.2937), prefix-bidirectional vLLM plugin) behind the card's server (github.com/StartLuxLabs/StartLux-Decision at a99977d): ~/envs/startlux/bin/startlux-decide serve --model ~/decision-models/StartLux-Decision-v1.1-2B-Instant@f1e6cd147dda82c9ffa3e8e725d5c1f175566fb1 --launch-vllm --gpus 0 --host 0.0.0.0 --port 19440 (env ~/envs/startlux: vllm==0.31.0, torch 2.13.0+cu130, the repository at a99977d with [vllm]; vLLM 0.31.0 because 0.19.1, the version of the card's published scores, has no aarch64 build; the plugin is validated on both); vision on (the server's default); bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'startlux' (System One at /v1/systemone, model ignored)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "startlux-models/StartLux-Decision-v1.1-2B-Instant",
    "--provider",
    "startlux",
    "--timeout",
    "60",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "b55c22654bc72e59b7a430a92a37731267ddb6de",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "startlux-models/StartLux-Decision-v1.1-2B-Instant",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "startlux",
  "questions": 1125,
  "request_seconds": 45.58553473860957,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "ccdb9fbcd8db96874e542de84145227f7a3b0f4e04da59dfc5712f14f258246f",
  "runner": "benchmark_ollama.py",
  "runner_commit": "5fb51788d3992285ed8f0cf76fd6e8716353b0d9",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 394757,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19440/v1/systemone
{
 "model": "startlux-models/StartLux-Decision-v1.1-2B-Instant",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · startlux-2b_en_dontknow

Name
StartLux-Decision-v1.1-2B-Instant
Setup
StartLux-Decision v1.1 2B · decision model · with the I don't know option F
Model
startlux-models/StartLux-Decision-v1.1-2B-Instant · provider startlux
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.002897 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0136 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.003 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run startlux-2b_en_dontknow · first answer 2026-11-11 12:25:49 UTC · last answer 2026-11-11 12:26:38 UTC · span 49.0 s, pauses included (not charged) · charged 49.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0029 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 2B dense · source https://huggingface.co/startlux-models/StartLux-Decision-v1.1-2B-Instant · read 2026-11-10 · the card states 2B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 2.27B (counted from the checkpoint (2,274,069,824: language model 1,881,825,088, MTP layer 60,828,160, vision tower 331,416,576; dense, no experts); from Qwen3.5-2B; the card: 2B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
11 Nov 2026 12:25 UTC; last answer 11 Nov 2026 12:26 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup StartLux-Decision v1.1 2B (site run None); runner 5fb51788d; startlux-models/StartLux-Decision-v1.1-2B-Instant@f1e6cd147dda82c9ffa3e8e725d5c1f175566fb1 (weights CC BY-NC 4.0, non-commercial: run for this non-commercial benchmark; code Apache-2.0; Qwen3.5-2B, per-type temperatures from decision_config.json (choice 1.2937), prefix-bidirectional vLLM plugin) behind the card's server (github.com/StartLuxLabs/StartLux-Decision at a99977d): ~/envs/startlux/bin/startlux-decide serve --model ~/decision-models/StartLux-Decision-v1.1-2B-Instant@f1e6cd147dda82c9ffa3e8e725d5c1f175566fb1 --launch-vllm --gpus 0 --host 0.0.0.0 --port 19440 (env ~/envs/startlux: vllm==0.31.0, torch 2.13.0+cu130, the repository at a99977d with [vllm]; vLLM 0.31.0 because 0.19.1, the version of the card's published scores, has no aarch64 build; the plugin is validated on both); vision on (the server's default); bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'startlux' (System One at /v1/systemone, model ignored)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "startlux-models/StartLux-Decision-v1.1-2B-Instant",
    "--provider",
    "startlux",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "b55c22654bc72e59b7a430a92a37731267ddb6de",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "startlux-models/StartLux-Decision-v1.1-2B-Instant",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "startlux",
  "questions": 1125,
  "request_seconds": 46.36886011622846,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "a2c28ca4c2d5b2844913900e6861fbb7906cbc9d78252b621d52e3c864162fe3",
  "runner": "benchmark_ollama.py",
  "runner_commit": "5fb51788d3992285ed8f0cf76fd6e8716353b0d9",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 426257,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19440/v1/systemone
{
 "model": "startlux-models/StartLux-Decision-v1.1-2B-Instant",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · qwen3.8-flash_en

Name
Qwen3.8 Flash off · chat
Setup
Qwen3.8 Flash · chat model
Model
qwen/qwen3.8-flash · provider openrouter
GPU
—
Cost
$1.548 for the run · provider price × reported usage
Answer time
run qwen3.8-flash_en · first answer 2026-11-11 15:10:45 UTC · last answer 2026-11-13 15:29:25 UTC · span 173920.0 s, pauses included (not charged) · charged 4109.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 7.9 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: not disclosed by maker (closed weights) · source https://openrouter.ai/qwen/qwen3.8-flash · read 2026-11-13
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
11 Nov 2026 15:10 UTC; last answer 13 Nov 2026 15:29 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Qwen3.8 Flash through OpenRouter (qwen/qwen3.8-flash), provider pinned to alibaba, fallbacks off; reasoning {"enabled": false}; temperature 1.0; max_tokens 32768; one question per request; parameters not disclosed (closed weights). English translations.
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "alibaba"
    ]
  },
  "reasoning": {
    "enabled": false
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · qwen3-14b_en

Name
Qwen3-14B off · chat
Setup
Qwen3-14B (chat, reasoning off, on our GPU) · chat model
Model
qwen3-14b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.993 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.801 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.993. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run qwen3-14b_en · first answer 2026-11-11 17:10:55 UTC · last answer 2026-11-11 17:58:57 UTC · span 2882.0 s, pauses included (not charged) · charged 2882.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.9934 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 16.8 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 123 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 14.8B dense · source https://huggingface.co/Qwen/Qwen3-14B · read 2026-11-11 · dense; the maker's stated figure (14.8B, the card) is shown, as for every model (turkce #4263); checkpoint count 14.77B
Input
text, English
Endpoint
http://127.0.0.1:19500/v1/chat/completions
Started
11 Nov 2026 17:10 UTC; last answer 11 Nov 2026 17:58 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Qwen3-14B as Reading's setup: Qwen/Qwen3-14B@40c069824f4251a91eefaf281ebe4c544efd3e18 BF16, served on our GPU host (one RTX PRO 6000) via docker vLLM 0.30.0 (vllm/vllm-openai:v0.30.0) from the HF cache as Reading's runs, max-model-len 16,384, gpu-memory-utilization 0.85; reasoning off as Reading's requests (chat_template_kwargs enable_thinking=false, thinking=false), temperature 0, seed 0; max_tokens 8192; one question per request (32 requests in flight; 8 for the first 424 answers), line prompt 'ANSWER: X', no system message and no JSON schema (Reading: system prompt, JSON schema, 16 questions per request). English translations.
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "max_tokens": 8192,
  "seed": 0,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · qwen3-14b_en_dontknow

Name
Qwen3-14B off · chat
Setup
Qwen3-14B (chat, reasoning off, on our GPU) · chat model · with the I don't know option F
Model
qwen3-14b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.5741 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.231 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.574. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run qwen3-14b_en_dontknow · first answer 2026-11-11 17:58:58 UTC · last answer 2026-11-11 18:12:48 UTC · span 830.0 s, pauses included (not charged) · charged 830.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.5741 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 27.6 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 43 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 14.8B dense · source https://huggingface.co/Qwen/Qwen3-14B · read 2026-11-11 · dense; the maker's stated figure (14.8B, the card) is shown, as for every model (turkce #4263); checkpoint count 14.77B
Input
text, English
Endpoint
http://127.0.0.1:19500/v1/chat/completions
Started
11 Nov 2026 17:58 UTC; last answer 11 Nov 2026 18:12 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Qwen3-14B as Reading's setup: Qwen/Qwen3-14B@40c069824f4251a91eefaf281ebe4c544efd3e18 BF16, served on our GPU host (one RTX PRO 6000) via docker vLLM 0.30.0 (vllm/vllm-openai:v0.30.0) from the HF cache as Reading's runs, max-model-len 16,384, gpu-memory-utilization 0.85; reasoning off as Reading's requests (chat_template_kwargs enable_thinking=false, thinking=false), temperature 0, seed 0; max_tokens 8192; one question per request (32 in flight), line prompt 'ANSWER: X', no system message and no JSON schema (Reading: system prompt, JSON schema, 16 questions per request). English translations.
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "max_tokens": 8192,
  "seed": 0,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · knowline-0.8b_en

Name
KnowLine-0.8B-Gen1
Setup
KnowLine-0.8B-Gen1 · decision model
Model
PelaAI/KnowLine-0.8B-Gen1 · provider knowline
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.002246 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0106 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.002 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run knowline-0.8b_en · first answer 2026-11-11 18:09:50 UTC · last answer 2026-11-11 18:10:28 UTC · span 38.0 s, pauses included (not charged) · charged 38.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0022 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 0.8B dense · source https://huggingface.co/PelaAI/KnowLine-0.8B-Gen1 · read 2026-11-11 · the card states a 0.8B decision model; checkpoint count 0.87B (counted from the checkpoint (873,438,784: language model 752,393,024, MTP layer 20,452,864 (model-extra-from-base.safetensors), vision tower 100,592,896); a LoRA fine-tune of Qwen3.5-0.8B merged into the weights; dense; the card: 0.8B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
11 Nov 2026 18:09 UTC; last answer 11 Nov 2026 18:10 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup KnowLine-0.8B-Gen1 (site run None); runner 5fb51788d; PelaAI/KnowLine-0.8B-Gen1@6b27831f7b783c03278bd946d4d8f4f7a98a3798 (Apache-2.0 weights and code; one-epoch LoRA SFT of Qwen3.5-0.8B merged into BF16, readout temperature 1.54 folded into the weights, score questions divided by temperatures.json) behind the card's serve_knowline.sh: SGLang 0.5.21 in BF16 (--enable-fp32-lm-head --mamba-radix-cache-strategy extra_buffer, MEM default 0.85) on port+1000 (20450) and knowline_server.py (read twice up to 3,000 tokens): cd ~/decision-models/KnowLine-0.8B-Gen1@6b27831f7b783c03278bd946d4d8f4f7a98a3798 && PATH=~/envs/knowline-sglang/bin:$PATH HOST=0.0.0.0 bash serve_knowline.sh ~/decision-models/KnowLine-0.8B-Gen1@6b27831f7b783c03278bd946d4d8f4f7a98a3798 0 19450, env ~/envs/knowline-sglang (sglang 0.5.21, torch 2.13.0+cu130, transformers 5.12.1); vram_mib is MEM 0.85 of the unified memory (SGLang sizes its KV cache to it), so it cannot run beside another large server; vision on (the server's default); bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'knowline'
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "PelaAI/KnowLine-0.8B-Gen1",
    "--provider",
    "knowline",
    "--timeout",
    "60",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "b55c22654bc72e59b7a430a92a37731267ddb6de",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "PelaAI/KnowLine-0.8B-Gen1",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "knowline",
  "questions": 1125,
  "request_seconds": 36.504834174585994,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "f565c99f2b880b8fb98e645e20d8ed9486e67eee27619eb97b5db8f7eb77e864",
  "runner": "benchmark_ollama.py",
  "runner_commit": "5fb51788d3992285ed8f0cf76fd6e8716353b0d9",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 836764,
    "output_tokens": 1125
  }
}
Prompt template
POST http://spark2:19450/v1/systemone
{
 "model": "PelaAI/KnowLine-0.8B-Gen1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · knowline-0.8b_en_dontknow

Name
KnowLine-0.8B-Gen1
Setup
KnowLine-0.8B-Gen1 · decision model · with the I don't know option F
Model
PelaAI/KnowLine-0.8B-Gen1 · provider knowline
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.002365 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0111 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.002 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run knowline-0.8b_en_dontknow · first answer 2026-11-11 18:11:36 UTC · last answer 2026-11-11 18:12:16 UTC · span 40.0 s, pauses included (not charged) · charged 40.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0024 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 0.8B dense · source https://huggingface.co/PelaAI/KnowLine-0.8B-Gen1 · read 2026-11-11 · the card states a 0.8B decision model; checkpoint count 0.87B (counted from the checkpoint (873,438,784: language model 752,393,024, MTP layer 20,452,864 (model-extra-from-base.safetensors), vision tower 100,592,896); a LoRA fine-tune of Qwen3.5-0.8B merged into the weights; dense; the card: 0.8B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
11 Nov 2026 18:11 UTC; last answer 11 Nov 2026 18:12 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup KnowLine-0.8B-Gen1 (site run None); runner 5fb51788d; PelaAI/KnowLine-0.8B-Gen1@6b27831f7b783c03278bd946d4d8f4f7a98a3798 (Apache-2.0 weights and code; one-epoch LoRA SFT of Qwen3.5-0.8B merged into BF16, readout temperature 1.54 folded into the weights, score questions divided by temperatures.json) behind the card's serve_knowline.sh: SGLang 0.5.21 in BF16 (--enable-fp32-lm-head --mamba-radix-cache-strategy extra_buffer, MEM default 0.85) on port+1000 (20450) and knowline_server.py (read twice up to 3,000 tokens): cd ~/decision-models/KnowLine-0.8B-Gen1@6b27831f7b783c03278bd946d4d8f4f7a98a3798 && PATH=~/envs/knowline-sglang/bin:$PATH HOST=0.0.0.0 bash serve_knowline.sh ~/decision-models/KnowLine-0.8B-Gen1@6b27831f7b783c03278bd946d4d8f4f7a98a3798 0 19450, env ~/envs/knowline-sglang (sglang 0.5.21, torch 2.13.0+cu130, transformers 5.12.1); vram_mib is MEM 0.85 of the unified memory (SGLang sizes its KV cache to it), so it cannot run beside another large server; vision on (the server's default); bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'knowline'
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "PelaAI/KnowLine-0.8B-Gen1",
    "--provider",
    "knowline",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "b55c22654bc72e59b7a430a92a37731267ddb6de",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "PelaAI/KnowLine-0.8B-Gen1",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "knowline",
  "questions": 1125,
  "request_seconds": 38.25984866899671,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "ff6ca1caac7c66eddfcad6a767434e5e0a8401bc37369ff8a232d958e68828aa",
  "runner": "benchmark_ollama.py",
  "runner_commit": "5fb51788d3992285ed8f0cf76fd6e8716353b0d9",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 904264,
    "output_tokens": 1125
  }
}
Prompt template
POST http://spark2:19450/v1/systemone
{
 "model": "PelaAI/KnowLine-0.8B-Gen1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · erk-14b_en

Name
Erk-14B off · chat
Setup
Erk-14B (chat, reasoning off, on our GPU) · chat model
Model
erk-14b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.4385 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.176 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.439. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run erk-14b_en · first answer 2026-11-11 18:52:54 UTC · last answer 2026-11-11 19:03:28 UTC · span 634.0 s, pauses included (not charged) · charged 634.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.4385 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 23.5 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 12 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 14.77B dense · source https://huggingface.co/ecloudtech/Erk-14B · read 2026-11-11 · dense; counted from the released checkpoint
Input
text, English
Endpoint
http://127.0.0.1:19501/v1/chat/completions
Started
11 Nov 2026 18:52 UTC; last answer 11 Nov 2026 19:03 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Erk-14B as Reading's setup: ecloudtech/Erk-14B@8584b14a829d5cc803bc227f44e9012e36e8ad04 BF16, served on our GPU host (one RTX PRO 6000) via docker vLLM 0.30.0 (vllm/vllm-openai:v0.30.0) from the HF cache as Reading's runs, max-model-len 16,384, gpu-memory-utilization 0.85; reasoning off as Reading's requests (chat_template_kwargs enable_thinking=false, thinking=false), temperature 0, seed 0; max_tokens 8192; one question per request (32 in flight), line prompt 'ANSWER: X', no system message and no JSON schema (Reading: system prompt, JSON schema, 16 questions per request); with no system message Erk-14B's chat template inserts its own default (Turkish identity) system prompt. English translations.
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "max_tokens": 8192,
  "seed": 0,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · erk-14b_en_dontknow

Name
Erk-14B off · chat
Setup
Erk-14B (chat, reasoning off, on our GPU) · chat model · with the I don't know option F
Model
erk-14b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.3355 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.135 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.335. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run erk-14b_en_dontknow · first answer 2026-11-11 19:03:32 UTC · last answer 2026-11-11 19:11:37 UTC · span 485.0 s, pauses included (not charged) · charged 485.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.3355 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 27.2 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 6 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 14.77B dense · source https://huggingface.co/ecloudtech/Erk-14B · read 2026-11-11 · dense; counted from the released checkpoint
Input
text, English
Endpoint
http://127.0.0.1:19501/v1/chat/completions
Started
11 Nov 2026 19:03 UTC; last answer 11 Nov 2026 19:11 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Erk-14B as Reading's setup: ecloudtech/Erk-14B@8584b14a829d5cc803bc227f44e9012e36e8ad04 BF16, served on our GPU host (one RTX PRO 6000) via docker vLLM 0.30.0 (vllm/vllm-openai:v0.30.0) from the HF cache as Reading's runs, max-model-len 16,384, gpu-memory-utilization 0.85; reasoning off as Reading's requests (chat_template_kwargs enable_thinking=false, thinking=false), temperature 0, seed 0; max_tokens 8192; one question per request (32 in flight), line prompt 'ANSWER: X', no system message and no JSON schema (Reading: system prompt, JSON schema, 16 questions per request); with no system message Erk-14B's chat template inserts its own default (Turkish identity) system prompt. English translations.
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "max_tokens": 8192,
  "seed": 0,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · gemini-decision-1_en

Name
Gemini Decision 1 20261111
Setup
Gemini Decision 1 · decision model
Model
google/gemini-decision-1 · provider openrouter
GPU
—
Cost
$0.04137 for the run · provider price × reported usage
Answer time
run gemini-decision-1_en · first answer 2026-11-11 19:14:57 UTC · last answer 2026-11-11 19:43:05 UTC · span 1688.0 s, pauses included (not charged) · charged 1688.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 0.3 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker (closed weights) · source https://blog.google/innovation-and-ai/technology/developers-tools/gemini-decisions-api/ · read 2026-11-11
Input
text, English
Endpoint
https://openrouter.ai/api
Started
11 Nov 2026 19:14 UTC; last answer 11 Nov 2026 19:43 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Gemini Decision 1 (site run None); runner 7b6fdc115; via OpenRouter's System One route (Google AI Studio serves it; replies name google/gemini-decision-1-20261111); latency includes OpenRouter's time; text only on Logic & Math. Launched 11 Nov 2026; Umit via Can #4966.
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "google/gemini-decision-1",
    "--provider",
    "openrouter",
    "--timeout",
    "60",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "b55c22654bc72e59b7a430a92a37731267ddb6de",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "google/gemini-decision-1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "OpenRouter's Gemini Decision 1 price (Google AI Studio)",
    "date": "2026-11-11",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.1,
      "cache_write_input_tokens": 0.1,
      "cached_input_tokens": 0.1,
      "input_tokens": 0.1,
      "output_tokens": 0.0
    },
    "source": "https://openrouter.ai/google/gemini-decision-1",
    "text": "$0.10 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "openrouter",
  "questions": 1125,
  "request_seconds": 512.6078630988777,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "746b4dd1bf468ad50543b55590d3ed60b226f524fb8b230068ba8a47edade523",
  "runner": "benchmark_ollama.py",
  "runner_commit": "7b6fdc11507bb13f2aa97d91cc180214a3031447",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 413714,
    "output_tokens": 0
  }
}
Prompt template
POST https://openrouter.ai/api/v1/systemone
{
 "model": "google/gemini-decision-1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · startlux-0.8b_en

Name
StartLux-Decision-v1.1-0.8B-Instant
Setup
StartLux-Decision-v1.1-0.8B-Instant · decision model
Model
startlux-models/StartLux-Decision-v1.1-0.8B-Instant · provider startlux
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.00201 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.00944 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.002 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run startlux-0.8b_en · first answer 2026-11-11 19:32:44 UTC · last answer 2026-11-11 19:33:18 UTC · span 34.0 s, pauses included (not charged) · charged 34.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0020 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 0.8B dense · source https://huggingface.co/startlux-models/StartLux-Decision-v1.1-0.8B-Instant · read 2026-11-11 · the card states a 0.8B decision model; checkpoint count 0.87B (counted from the checkpoint (873,438,784: language model incl. MTP layer 772,845,888, vision tower 100,592,896); from Qwen3.5-0.8B; dense; the card: 0.8B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
11 Nov 2026 19:32 UTC; last answer 11 Nov 2026 19:33 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup StartLux-Decision-v1.1-0.8B-Instant (site run None); runner 5fb51788d; startlux-models/StartLux-Decision-v1.1-0.8B-Instant@6496ae641cc71b149ce9dad69bd97f2a44e7bf11 (weights CC BY-NC 4.0, non-commercial: run for this non-commercial benchmark, as StartLux 4B and 2B; code Apache-2.0; Qwen3.5-0.8B, per-type temperatures from decision_config.json (choice 1.4364), prefix-bidirectional vLLM plugin) behind the card's server (github.com/StartLuxLabs/StartLux-Decision at a99977d): ~/envs/startlux/bin/startlux-decide serve --model ~/decision-models/StartLux-Decision-v1.1-0.8B-Instant@6496ae641cc71b149ce9dad69bd97f2a44e7bf11 --launch-vllm --gpus 0 --host 0.0.0.0 --port 19460 (env ~/envs/startlux: vllm==0.31.0, torch 2.13.0+cu130, the repository at a99977d with [vllm]; vLLM 0.31.0 because 0.19.1, the version of the card's published scores, has no aarch64 build; the card supports 0.31); vram_mib is vLLM's --gpu-memory-utilization default 0.88 of the unified memory, so it cannot run beside another large server; vision on (the server's default); bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'startlux' (System One at /v1/systemone, model ignored)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "startlux-models/StartLux-Decision-v1.1-0.8B-Instant",
    "--provider",
    "startlux",
    "--timeout",
    "60",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "b55c22654bc72e59b7a430a92a37731267ddb6de",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "startlux-models/StartLux-Decision-v1.1-0.8B-Instant",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "startlux",
  "questions": 1125,
  "request_seconds": 30.847514470195165,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "1f99ab009188745ea80379e05e073c26be9da55bcfca9f5f4e3968c616f31c5d",
  "runner": "benchmark_ollama.py",
  "runner_commit": "5fb51788d3992285ed8f0cf76fd6e8716353b0d9",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 394757,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19460/v1/systemone
{
 "model": "startlux-models/StartLux-Decision-v1.1-0.8B-Instant",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · startlux-0.8b_en_dontknow

Name
StartLux-Decision-v1.1-0.8B-Instant
Setup
StartLux-Decision-v1.1-0.8B-Instant · decision model · with the I don't know option F
Model
startlux-models/StartLux-Decision-v1.1-0.8B-Instant · provider startlux
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.00201 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.00944 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.002 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run startlux-0.8b_en_dontknow · first answer 2026-11-11 19:34:22 UTC · last answer 2026-11-11 19:34:56 UTC · span 34.0 s, pauses included (not charged) · charged 34.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0020 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 0.8B dense · source https://huggingface.co/startlux-models/StartLux-Decision-v1.1-0.8B-Instant · read 2026-11-11 · the card states a 0.8B decision model; checkpoint count 0.87B (counted from the checkpoint (873,438,784: language model incl. MTP layer 772,845,888, vision tower 100,592,896); from Qwen3.5-0.8B; dense; the card: 0.8B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
11 Nov 2026 19:34 UTC; last answer 11 Nov 2026 19:34 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup StartLux-Decision-v1.1-0.8B-Instant (site run None); runner 5fb51788d; startlux-models/StartLux-Decision-v1.1-0.8B-Instant@6496ae641cc71b149ce9dad69bd97f2a44e7bf11 (weights CC BY-NC 4.0, non-commercial: run for this non-commercial benchmark, as StartLux 4B and 2B; code Apache-2.0; Qwen3.5-0.8B, per-type temperatures from decision_config.json (choice 1.4364), prefix-bidirectional vLLM plugin) behind the card's server (github.com/StartLuxLabs/StartLux-Decision at a99977d): ~/envs/startlux/bin/startlux-decide serve --model ~/decision-models/StartLux-Decision-v1.1-0.8B-Instant@6496ae641cc71b149ce9dad69bd97f2a44e7bf11 --launch-vllm --gpus 0 --host 0.0.0.0 --port 19460 (env ~/envs/startlux: vllm==0.31.0, torch 2.13.0+cu130, the repository at a99977d with [vllm]; vLLM 0.31.0 because 0.19.1, the version of the card's published scores, has no aarch64 build; the card supports 0.31); vram_mib is vLLM's --gpu-memory-utilization default 0.88 of the unified memory, so it cannot run beside another large server; vision on (the server's default); bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'startlux' (System One at /v1/systemone, model ignored)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "startlux-models/StartLux-Decision-v1.1-0.8B-Instant",
    "--provider",
    "startlux",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "b55c22654bc72e59b7a430a92a37731267ddb6de",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "startlux-models/StartLux-Decision-v1.1-0.8B-Instant",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "startlux",
  "questions": 1125,
  "request_seconds": 31.834609649784397,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "b99dd6f71c75b4e50f73c0011df6f9786bdb2979a184c4b4b46196d58222067a",
  "runner": "benchmark_ollama.py",
  "runner_commit": "5fb51788d3992285ed8f0cf76fd6e8716353b0d9",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 426257,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19460/v1/systemone
{
 "model": "startlux-models/StartLux-Decision-v1.1-0.8B-Instant",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · gemini-decision-1_en_dontknow

Name
Gemini Decision 1 20261111
Setup
Gemini Decision 1 · decision model · with the I don't know option F
Model
google/gemini-decision-1 · provider openrouter
GPU
—
Cost
$0.04508 for the run · provider price × reported usage
Answer time
run gemini-decision-1_en_dontknow · first answer 2026-11-11 19:43:14 UTC · last answer 2026-11-11 19:58:15 UTC · span 901.0 s, pauses included (not charged) · charged 901.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 0.5 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker (closed weights) · source https://blog.google/innovation-and-ai/technology/developers-tools/gemini-decisions-api/ · read 2026-11-11
Input
text, English
Endpoint
https://openrouter.ai/api
Started
11 Nov 2026 19:43 UTC; last answer 11 Nov 2026 19:58 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Gemini Decision 1 (site run None); runner 7b6fdc115; via OpenRouter's System One route (Google AI Studio serves it; replies name google/gemini-decision-1-20261111); latency includes OpenRouter's time; text only on Logic & Math. Launched 11 Nov 2026; Umit via Can #4966.
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "google/gemini-decision-1",
    "--provider",
    "openrouter",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "b55c22654bc72e59b7a430a92a37731267ddb6de",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "google/gemini-decision-1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "OpenRouter's Gemini Decision 1 price (Google AI Studio)",
    "date": "2026-11-11",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.1,
      "cache_write_input_tokens": 0.1,
      "cached_input_tokens": 0.1,
      "input_tokens": 0.1,
      "output_tokens": 0.0
    },
    "source": "https://openrouter.ai/google/gemini-decision-1",
    "text": "$0.10 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "openrouter",
  "questions": 1125,
  "request_seconds": 443.5398940129089,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "73c082febcbd3453bd34c58d0ddcc6d9fa0c157ad863a7b2e0d0e1a2a49e00ae",
  "runner": "benchmark_ollama.py",
  "runner_commit": "7b6fdc11507bb13f2aa97d91cc180214a3031447",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 450839,
    "output_tokens": 0
  }
}
Prompt template
POST https://openrouter.ai/api/v1/systemone
{
 "model": "google/gemini-decision-1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · onejev1.6-0.8b_en

Name
OneJev1.6-0.8B
Setup
OneJev1.6-0.8B · decision model
Model
OmniJev/OneJev1.6-0.8B · provider onejev
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.001301 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.00611 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.001 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run onejev1.6-0.8b_en · first answer 2026-11-11 20:54:07 UTC · last answer 2026-11-11 20:54:29 UTC · span 22.0 s, pauses included (not charged) · charged 22.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0013 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.8 in flight (sum of request latencies / active time)
Model size
Parameters: 0.8B dense · source https://huggingface.co/OmniJev/OneJev1.6-0.8B · read 2026-11-10 · the official model name: OneJev1.6-0.8B (turkce #4263: the maker's stated total is shown); counted from the released files: 0.85B (counted from the checkpoint (852,985,920: language model 752,393,024, vision tower 100,592,896); an average of three Qwen3.5-0.8B fine-tunes; the card's chart: 0.9B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
11 Nov 2026 20:54 UTC; last answer 11 Nov 2026 20:54 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup OneJev1.6-0.8B (site run None); runner 7b6fdc115; OmniJev/OneJev1.6-0.8B@bcd88a382f314b26cb7b1b7f369c1ed3502ee7a0 (Apache-2.0; average of three Qwen3.5-0.8B fine-tunes; no calibration.json shipped, so temperature 1); main moved from c89933c2 to bcd88a38 on 10 Nov (README only: 'Link the Flide paper'); all other files byte-identical, verified 2026-11-11. Behind the card's server, Flide (flide pinned at 0.1.11, the current PyPI release; the card names no version): ~/envs/onejev16/bin/flide serve --model ~/decision-models/OneJev1.6-0.8B@bcd88a382f314b26cb7b1b7f369c1ed3502ee7a0 --name OmniJev/OneJev1.6-0.8B --host 0.0.0.0 --port 19420 (env: glide-family's stack + flide==0.1.11, accelerate, torchvision 0.29.1+cu130); vision on (the server's default); bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider onejev as is (model field = --name; probabilities rounded to 6 places)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "OmniJev/OneJev1.6-0.8B",
    "--provider",
    "onejev",
    "--timeout",
    "30",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "b49f0758f6a5aa13377063fd0c34ff02dd9deaa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "OmniJev/OneJev1.6-0.8B",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "onejev",
  "questions": 1125,
  "request_seconds": 18.095550994214136,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "a67ce3bc49b5fb1d3dc9245f350f9c80693065134de7e7e0f02bc81d2a527bbf",
  "runner": "benchmark_ollama.py",
  "runner_commit": "7b6fdc11507bb13f2aa97d91cc180214a3031447",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 431882,
    "output_tokens": 1125
  }
}
Prompt template
POST http://spark2:19420/v1/systemone
{
 "model": "OmniJev/OneJev1.6-0.8B",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · onejev1.6-0.8b_en_dontknow

Name
OneJev1.6-0.8B
Setup
OneJev1.6-0.8B · decision model · with the I don't know option F
Model
OmniJev/OneJev1.6-0.8B · provider onejev
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.001478 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.00694 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.001 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run onejev1.6-0.8b_en_dontknow · first answer 2026-11-11 20:55:14 UTC · last answer 2026-11-11 20:55:39 UTC · span 25.0 s, pauses included (not charged) · charged 25.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0015 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.8 in flight (sum of request latencies / active time)
Model size
Parameters: 0.8B dense · source https://huggingface.co/OmniJev/OneJev1.6-0.8B · read 2026-11-10 · the official model name: OneJev1.6-0.8B (turkce #4263: the maker's stated total is shown); counted from the released files: 0.85B (counted from the checkpoint (852,985,920: language model 752,393,024, vision tower 100,592,896); an average of three Qwen3.5-0.8B fine-tunes; the card's chart: 0.9B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
11 Nov 2026 20:55 UTC; last answer 11 Nov 2026 20:55 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup OneJev1.6-0.8B (site run None); runner 7b6fdc115; OmniJev/OneJev1.6-0.8B@bcd88a382f314b26cb7b1b7f369c1ed3502ee7a0 (Apache-2.0; average of three Qwen3.5-0.8B fine-tunes; no calibration.json shipped, so temperature 1); main moved from c89933c2 to bcd88a38 on 10 Nov (README only: 'Link the Flide paper'); all other files byte-identical, verified 2026-11-11. Behind the card's server, Flide (flide pinned at 0.1.11, the current PyPI release; the card names no version): ~/envs/onejev16/bin/flide serve --model ~/decision-models/OneJev1.6-0.8B@bcd88a382f314b26cb7b1b7f369c1ed3502ee7a0 --name OmniJev/OneJev1.6-0.8B --host 0.0.0.0 --port 19420 (env: glide-family's stack + flide==0.1.11, accelerate, torchvision 0.29.1+cu130); vision on (the server's default); bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider onejev as is (model field = --name; probabilities rounded to 6 places)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "OmniJev/OneJev1.6-0.8B",
    "--provider",
    "onejev",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "b49f0758f6a5aa13377063fd0c34ff02dd9deaa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "OmniJev/OneJev1.6-0.8B",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "onejev",
  "questions": 1125,
  "request_seconds": 20.524674505839357,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "d1d5d032ecffe648414da84a7e53399776b9fa945b0804032d5c5fa163cadae0",
  "runner": "benchmark_ollama.py",
  "runner_commit": "7b6fdc11507bb13f2aa97d91cc180214a3031447",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 467882,
    "output_tokens": 1125
  }
}
Prompt template
POST http://spark2:19420/v1/systemone
{
 "model": "OmniJev/OneJev1.6-0.8B",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · lfm2.5-2.6b_en

Name
LFM2.5-2.6B on · chat
Setup
LFM2.5-2.6B (chat, thinking, on our GPU) · chat model
Model
lfm2.5-2.6b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$2.789 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 1.12 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $2.789. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run lfm2.5-2.6b_en · first answer 2026-11-12 01:11:31 UTC · last answer 2026-11-12 02:18:44 UTC · span 4033.0 s, pauses included (not charged) · charged 4033.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $2.7895 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 30.0 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 36 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 2.69B dense · source https://huggingface.co/LiquidAI/LFM2.5-2.6B · read 2026-11-12 · the card states 2.69B total parameters (the maker's figure is shown); checkpoint count 2.70B (2,697,198,592 incl. tied embeddings, text only; dense)
Input
text, English
Endpoint
http://127.0.0.1:18613/v1/chat/completions
Started
12 Nov 2026 01:11 UTC; last answer 12 Nov 2026 02:18 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
LFM2.5-2.6B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from LiquidAI/LFM2.5-2.6B@7141b56cd4650af95c7c7b86983443aed050c96a, vLLM reasoning parser qwen3, one RTX PRO 6000 on our GPU host alone; thinking always on (the template opens <think> for every answer; no switch, no levels); sampling temperature 0.1, top_k 50, repetition_penalty 1.1, from generation_config.json; seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 2.70B / 2.70B active / total (dense; checkpoint 2,697,198,592 incl. tied embeddings, text only; card: 2.69B). English translations (DeepSeek + review). No thinking-off setup (bus #6516/#6924/#8187/#8202): the current served chat template exposes no supported switch for a separate thinking-off run; no separate off variant has been run or scored. (Checked 15 Nov in the pinned template: its only option, preserve_thinking, keeps or drops earlier turns' thinking.)
Parameters
{
  "max_tokens": 32768,
  "repetition_penalty": 1.1,
  "seed": 0,
  "temperature": 0.1,
  "top_k": 50
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · claude-opus-5.5-sub_en

Name
Claude Opus 5.5 chat
Setup
Claude Opus 5.5 · chat model
Model
claude-opus-5-5 · provider claude-subscription
GPU
—
Cost
$21.98 for the run, API list-price equivalent · Claude subscription
Answer time
run claude-opus-5.5-sub_en · first answer 2026-11-12 01:24:30 UTC · last answer 2026-11-12 12:14:44 UTC · span 39014.0 s, pauses included (not charged) · charged 8088.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker (closed weights) · source https://platform.claude.com/docs/en/about-claude/models · read 2026-11-13
Input
text, English
Endpoint
claude-code-subscription
Started
12 Nov 2026 01:24 UTC; last answer 12 Nov 2026 12:14 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Claude Opus 5.5 on the Claude subscription through the Claude Code CLI (`claude --print`, as turkce's runs 968/971), model claude-opus-5-5, --effort high (the highest of low/medium/high/xhigh/max; adaptive thinking always on); CLAUDE_CODE_MAX_OUTPUT_TOKENS 32768; no temperature sent (Anthropic's default 1.0); Claude Code's agent prompt replaced by a neutral one-line system prompt, tools, settings, MCP servers and session saving off, API-key variables removed; one question per request, one request at a time; cost $0 (Claude subscription); api_equivalent_usd is Claude Code's API-list-price estimate, not a cost. English translations.
Parameters
{
  "effort": "high",
  "max_output_tokens": 32768,
  "route": "claude-code-cli --print",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message."
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · lfm2.5-2.6b_en_dontknow

Name
LFM2.5-2.6B on · chat
Setup
LFM2.5-2.6B (chat, thinking, on our GPU) · chat model · with the I don't know option F
Model
lfm2.5-2.6b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$2.514 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 1.01 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $2.514. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run lfm2.5-2.6b_en_dontknow · first answer 2026-11-12 02:37:44 UTC · last answer 2026-11-12 03:38:18 UTC · span 3634.0 s, pauses included (not charged) · charged 3634.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $2.5135 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.4 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 20 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 2.69B dense · source https://huggingface.co/LiquidAI/LFM2.5-2.6B · read 2026-11-12 · the card states 2.69B total parameters (the maker's figure is shown); checkpoint count 2.70B (2,697,198,592 incl. tied embeddings, text only; dense)
Input
text, English
Endpoint
http://127.0.0.1:18613/v1/chat/completions
Started
12 Nov 2026 02:37 UTC; last answer 12 Nov 2026 03:38 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
LFM2.5-2.6B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from LiquidAI/LFM2.5-2.6B@7141b56cd4650af95c7c7b86983443aed050c96a, vLLM reasoning parser qwen3, one RTX PRO 6000 on our GPU host alone; thinking always on (the template opens <think> for every answer; no switch, no levels); sampling temperature 0.1, top_k 50, repetition_penalty 1.1, from generation_config.json; seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 2.70B / 2.70B active / total (dense; checkpoint 2,697,198,592 incl. tied embeddings, text only; card: 2.69B). English translations (DeepSeek + review). No thinking-off setup (bus #6516/#6924/#8187/#8202): the current served chat template exposes no supported switch for a separate thinking-off run; no separate off variant has been run or scored. (Checked 15 Nov in the pinned template: its only option, preserve_thinking, keeps or drops earlier turns' thinking.)
Parameters
{
  "max_tokens": 32768,
  "repetition_penalty": 1.1,
  "seed": 0,
  "temperature": 0.1,
  "top_k": 50
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · claude-fable-5.2-sub_en

Name
Claude Fable 5.2 chat
Setup
Claude Fable 5.2 · chat model
Model
claude-fable-5-2 · provider claude-subscription
GPU
—
Cost
$0 for the run · Claude subscription
Answer time
run claude-fable-5.2-sub_en · first answer 2026-11-12 04:05:52 UTC · last answer 2026-11-12 11:32:41 UTC · span 26809.0 s, pauses included (not charged) · charged 4980.0 s in 9 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 4.9 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker (closed weights) · source https://platform.claude.com/docs/en/about-claude/models · read 2026-11-13
Input
text, English
Endpoint
claude-code-subscription
Started
12 Nov 2026 04:05 UTC; last answer 12 Nov 2026 11:32 UTC
Progress
147 answered of 1,125 asked; 0 request errors; in progress
Notes
Claude Fable 5.2 on the Claude subscription through the Claude Code CLI (`claude --print`, as turkce's runs 968/971), model claude-fable-5-2, --effort max (the highest of low/medium/high/xhigh/max; adaptive thinking always on); CLAUDE_CODE_MAX_OUTPUT_TOKENS 32768; no temperature sent (Anthropic's default 1.0); Claude Code's agent prompt replaced by a neutral one-line system prompt, tools, settings, MCP servers and session saving off, API-key variables removed; one question per request, one request at a time; cost $0 (Claude subscription); api_equivalent_usd is Claude Code's API-list-price estimate, not a cost. English translations. 12 Nov 2026: 6 Logic replies stored as request errors (logic benchmark.db rows 238666, 238694, 241498, 242157, 242300, 243470) re-scored in place as the model's outcome, no letter, counted wrong, the CLI's stored record kept as the reply: 5 API refusals (finish_reason 'refusal') and 1 reply cut at the 32,768-token cap (finish_reason 'length'); run_eval.py now stores such Claude Code replies the same way and never retries or re-asks them (zeki #5263).
Parameters
{
  "effort": "max",
  "max_output_tokens": 32768,
  "route": "claude-code-cli --print",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message."
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · gemma-4-31b-it_en_t0_dontknow

Name
Gemma 4 31B chat
Setup
Gemma 4 31B (chat, on our GPU) · chat model · with the I don't know option F
Model
gemma-4-31B-it · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.081 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.434 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.081. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gemma-4-31b-it_en_t0_dontknow · first answer 2026-11-12 08:31:51 UTC · last answer 2026-11-12 08:57:54 UTC · span 1563.0 s, pauses included (not charged) · charged 1563.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.0811 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 15.7 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 2 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 30.7B dense · source https://huggingface.co/google/gemma-4-31B-it · read 2026-11-10 · the card states Total Parameters 30.7B (plus a vision encoder of about 550M) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 31.27B (counted from the checkpoint (31,273,088,876, the vision encoder included); the card gives 30.7B plus a vision encoder of about 550M)
Input
text, English
Endpoint
http://127.0.0.1:8021/v1/chat/completions
Started
12 Nov 2026 08:31 UTC; last answer 12 Nov 2026 08:57 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Gemma 4 31B-it FP8 weights + FP8 KV (vLLM v0.30.0, GPU 0 of our GPU host; same flags as the shared Spark host's server), thinking off (server default), temperature 0, English translations (DeepSeek + review); dont_know run (sixth option F "I don't know"), otherwise the same server and request settings as gemma-4-31b-it_en_t0
Parameters
{
  "max_tokens": 8192,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · gliclass-v4-edge_en

Name
GLiClass v4 Edge
Setup
GLiClass v4 Edge · decision model
Model
knowledgator/gliclass-v4-edge · provider gliclass
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.001478 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.00694 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.001 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gliclass-v4-edge_en · first answer 2026-11-12 08:32:32 UTC · last answer 2026-11-12 08:32:57 UTC · span 25.0 s, pauses included (not charged) · charged 25.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0015 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 0.4B dense · source https://huggingface.co/knowledgator/gliclass-v4-edge · read 2026-11-11 · the card states a 0.4B-parameter encoder (435M in its comparison table); checkpoint count 0.44B (counted from the checkpoint (439,790,596: DeBERTa-v3-large encoder 433,913,856, text and class projectors 4,198,400, scorer 557,569, calibrator 1,120,770); dense, text only; the card: 0.4B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
12 Nov 2026 08:32 UTC; last answer 12 Nov 2026 08:32 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup GLiClass v4 Edge (site run None); runner df9ae2734; knowledgator/gliclass-v4-edge@885af1af8a07edf4ed34df83ca0d995ed8201283 (Apache-2.0; a 0.44B GLiClass uni-encoder on DeBERTa-v3-large with a calibrator; text only) behind turkce's runner/servers/gliclass_server.py (sha256 aa39e5f4…, copied unchanged to ~/decision-src/gliclass_server.py on a shared Spark host), which applies the card's own System One mapping (its jev_api.py: one gliclass pass per choice question, the state as text, the instructions as the prompt, each option's description as a label; probabilities the softmax over the calibrated label logits) with gliclass 0.1.22 in BF16 and refuses an input over 16,384 tokens with 422, never cut: HF_HUB_OFFLINE=1 ~/envs/gliclass/bin/python ~/decision-src/gliclass_server.py --model ~/decision-models/gliclass-v4-edge@885af1af8a07edf4ed34df83ca0d995ed8201283 --device cuda:0 --host 0.0.0.0 --port 19480 (env ~/envs/gliclass: gliclass 0.1.22 (the card's pip install gliclass -U on 11 November 2026), torch 2.14.0+cu130, transformers 5.22.2); the server has no health route (health null: check with a request); model field absent or knowledgator/gliclass-v4-edge; vram_mib a weights-plus-activations estimate; bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'gliclass' (tb-next)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "knowledgator/gliclass-v4-edge",
    "--provider",
    "gliclass",
    "--timeout",
    "60",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "6ad0eac51a4abb950bde74b24687209aec628aa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "knowledgator/gliclass-v4-edge",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "gliclass",
  "questions": 1125,
  "request_seconds": 22.764959992084187,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "d03ccf04653ff2f42f522865d3f9613ba604386efe26c056df68d1c90bcdfaec",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 311677,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19480/v1/systemone
{
 "model": "knowledgator/gliclass-v4-edge",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · gliclass-v4-edge_en_dontknow

Name
GLiClass v4 Edge
Setup
GLiClass v4 Edge · decision model · with the I don't know option F
Model
knowledgator/gliclass-v4-edge · provider gliclass
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.001596 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0075 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.002 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gliclass-v4-edge_en_dontknow · first answer 2026-11-12 08:33:34 UTC · last answer 2026-11-12 08:34:01 UTC · span 27.0 s, pauses included (not charged) · charged 27.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0016 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 0.4B dense · source https://huggingface.co/knowledgator/gliclass-v4-edge · read 2026-11-11 · the card states a 0.4B-parameter encoder (435M in its comparison table); checkpoint count 0.44B (counted from the checkpoint (439,790,596: DeBERTa-v3-large encoder 433,913,856, text and class projectors 4,198,400, scorer 557,569, calibrator 1,120,770); dense, text only; the card: 0.4B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
12 Nov 2026 08:33 UTC; last answer 12 Nov 2026 08:34 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup GLiClass v4 Edge (site run None); runner df9ae2734; knowledgator/gliclass-v4-edge@885af1af8a07edf4ed34df83ca0d995ed8201283 (Apache-2.0; a 0.44B GLiClass uni-encoder on DeBERTa-v3-large with a calibrator; text only) behind turkce's runner/servers/gliclass_server.py (sha256 aa39e5f4…, copied unchanged to ~/decision-src/gliclass_server.py on a shared Spark host), which applies the card's own System One mapping (its jev_api.py: one gliclass pass per choice question, the state as text, the instructions as the prompt, each option's description as a label; probabilities the softmax over the calibrated label logits) with gliclass 0.1.22 in BF16 and refuses an input over 16,384 tokens with 422, never cut: HF_HUB_OFFLINE=1 ~/envs/gliclass/bin/python ~/decision-src/gliclass_server.py --model ~/decision-models/gliclass-v4-edge@885af1af8a07edf4ed34df83ca0d995ed8201283 --device cuda:0 --host 0.0.0.0 --port 19480 (env ~/envs/gliclass: gliclass 0.1.22 (the card's pip install gliclass -U on 11 November 2026), torch 2.14.0+cu130, transformers 5.22.2); the server has no health route (health null: check with a request); model field absent or knowledgator/gliclass-v4-edge; vram_mib a weights-plus-activations estimate; bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'gliclass' (tb-next)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "knowledgator/gliclass-v4-edge",
    "--provider",
    "gliclass",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "6ad0eac51a4abb950bde74b24687209aec628aa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "knowledgator/gliclass-v4-edge",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "gliclass",
  "questions": 1125,
  "request_seconds": 24.660366751719266,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "63cb3c5a92766571c1be807efbd49182a5842f5c929050b0e5ddb330f898a430",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 343177,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19480/v1/systemone
{
 "model": "knowledgator/gliclass-v4-edge",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · qwen3.8-27b-bf16_en_low_dontknow

Name
Qwen3.8-27B low · chat
Setup
Qwen3.8-27B (chat, reasoning low, on our GPU) · chat model · with the I don't know option F
Model
qwen38-27b-bf16 · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$2.656 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 1.07 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $2.656. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-12 10:32:03–10:44:19 UTC
GPU time
run qwen3.8-27b-bf16_en_low_dontknow · first answer 2026-11-12 09:28:03 UTC · last answer 2026-11-12 10:44:19 UTC · span 4576.0 s, pauses included (not charged) · charged 3840.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $2.6560 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 15.5 in flight (sum of request latencies / active time)
Output limit
60,000 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 27B dense · source https://huggingface.co/Qwen/Qwen3.8-27B · read 2026-11-10 · the card states Number of Parameters: 27B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 27.78B (counted from the checkpoint (27,781,427,952, the vision encoder included); the card says 27B)
Input
text, English
Endpoint
http://127.0.0.1:8022/v1/chat/completions
Started
12 Nov 2026 09:27 UTC; last answer 12 Nov 2026 10:44 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Qwen3.8-27B BF16 (vLLM v0.30.0, MTP, GPU 0 of our GPU host), thinking on at reasoning_effort low with the model's generation config sampling (1.0/0.95/20, seed 0) and a JSON answer schema, as turkce's run 318; English translations (DeepSeek + review); dont_know run in the line answer format: the standard run used the JSON answer schema (A–E only), which the runner doesn't define for option F; otherwise the same server and request settings
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "reasoning_effort": "low"
  },
  "max_tokens": 60000,
  "seed": 0,
  "temperature": 1.0,
  "top_k": 20,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · minicpm5-2b_en

Name
MiniCPM5-2B on · chat
Setup
MiniCPM5-2B (chat, thinking, on our GPU) · chat model
Model
minicpm5-2b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$3.828 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 1.54 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $3.828. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run minicpm5-2b_en · first answer 2026-11-12 10:01:09 UTC · last answer 2026-11-12 11:33:24 UTC · span 5535.0 s, pauses included (not charged) · charged 5535.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $3.8284 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 29.7 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 61 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 2.52B dense · from the run's notes (dense; checkpoint 2,516,756,480 incl. untied lm_head, text only; card: 2,516,756,480); also https://huggingface.co/openbmb/MiniCPM5-2B · read 2026-11-12 · counted from the checkpoint (2,516,756,480 incl. untied lm_head, text only; dense); the card: 2,516,756,480
Input
text, English
Endpoint
http://127.0.0.1:18614/v1/chat/completions
Started
12 Nov 2026 10:01 UTC; last answer 12 Nov 2026 11:33 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
MiniCPM5-2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from openbmb/MiniCPM5-2B@42433e90dad95123ab87729fd7011e7110629556, vLLM reasoning parser qwen3, one RTX PRO 6000 on our GPU host alone; thinking on (enable_thinking true; on/off only, no levels); sampling temperature 1, top_p 0.95, from generation_config.json; seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 2.52B / 2.52B active / total (dense; checkpoint 2,516,756,480 incl. untied lm_head, text only; card: 2,516,756,480). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "seed": 0,
  "temperature": 1.0,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · claude-sonnet-5.5-sub_en

Name
Claude Sonnet 5.5 chat
Setup
Claude Sonnet 5.5 · chat model
Model
claude-sonnet-5-5 · provider claude-subscription
GPU
—
Cost
$12.91 for the run, API list-price equivalent · Claude subscription
Answer time
run claude-sonnet-5.5-sub_en · first answer 2026-11-12 10:47:05 UTC · last answer 2026-11-12 13:31:20 UTC · span 9855.0 s, pauses included (not charged) · charged 9855.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker (closed weights) · source https://platform.claude.com/docs/en/about-claude/models · read 2026-11-13
Input
text, English
Endpoint
claude-code-subscription
Started
12 Nov 2026 10:47 UTC; last answer 12 Nov 2026 13:31 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Claude Sonnet 5.5 on the Claude subscription through the Claude Code CLI (`claude --print`, as turkce's runs 968/971), model claude-sonnet-5-5, --effort high (the highest of low/medium/high/xhigh/max; adaptive thinking always on); CLAUDE_CODE_MAX_OUTPUT_TOKENS 32768; no temperature sent (Anthropic's default 1.0); Claude Code's agent prompt replaced by a neutral one-line system prompt, tools, settings, MCP servers and session saving off, API-key variables removed; one question per request, one request at a time; cost $0 (Claude subscription); api_equivalent_usd is Claude Code's API-list-price estimate, not a cost. English translations.
Parameters
{
  "effort": "high",
  "max_output_tokens": 32768,
  "route": "claude-code-cli --print",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message."
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · grok-4.7-sub_en

Name
Grok 4.7 chat
Setup
Grok 4.7 · chat model
Model
grok-4.7 · provider xai-subscription
GPU
—
Cost
$46.56 for the run, API list-price equivalent · xAI subscription
Answer time
run grok-4.7-sub_en · first answer 2026-11-12 11:13:39 UTC · last answer 2026-11-15 11:36:03 UTC · span 260544.0 s, pauses included (not charged) · charged 49832.0 s in 36 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 3.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker (closed weights) · source https://x.ai · read 2026-11-11
Input
text, English
Endpoint
https://api.x.ai/v1/responses
Started
12 Nov 2026 11:12 UTC; last answer 15 Nov 2026 11:36 UTC
Progress
576 answered of 1,125 asked; 0 request errors; paused for now at 576 of 1,125 (Ümit #8603, zeki #8637, 2026-11-15T11:37Z)
Notes
Grok 4.7 on the SuperGrok subscription through xAI's Responses API with OMP's xai-oauth login (as turkce's ask_xai and jevgammon's xai-subscription route), model grok-4.7, reasoning effort high (Reading's setting); max_output_tokens 32768; no temperature sent (xAI's default) and no tools; a neutral one-line system prompt; one question per request, one request at a time; a usage-limit reply stops the run without retrying it (resumable); cost $0 (xAI subscription); api_equivalent_usd is the tokens at xAI's API list price {"input": 2.0, "cached_input": 0.5, "output": 6.0} per 1M tokens, not a cost. English translations. PAUSED (Umit #8603, zeki #8637, 2026-11-15T11:37Z): skipped for now, stopped at 576 answers; large models are left for now (in-flight requests at the stop were not stored); resumable from the stored answers at the same settings if asked.
Parameters
{
  "effort": "high",
  "list_price_per_mtok": {
    "cached_input": 0.5,
    "input": 2.0,
    "output": 6.0
  },
  "max_output_tokens": 32768,
  "route": "xai responses api (OMP xai-oauth)",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message.",
  "tools": "none"
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · minicpm5-2b_en_dontknow

Name
MiniCPM5-2B on · chat
Setup
MiniCPM5-2B (chat, thinking, on our GPU) · chat model · with the I don't know option F
Model
minicpm5-2b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$2.879 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 1.16 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $2.879. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run minicpm5-2b_en_dontknow · first answer 2026-11-12 11:54:07 UTC · last answer 2026-11-12 13:03:29 UTC · span 4162.0 s, pauses included (not charged) · charged 4162.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $2.8787 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 30.7 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 20 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 2.52B dense · from the run's notes (dense; checkpoint 2,516,756,480 incl. untied lm_head, text only; card: 2,516,756,480); also https://huggingface.co/openbmb/MiniCPM5-2B · read 2026-11-12 · counted from the checkpoint (2,516,756,480 incl. untied lm_head, text only; dense); the card: 2,516,756,480
Input
text, English
Endpoint
http://127.0.0.1:18614/v1/chat/completions
Started
12 Nov 2026 11:54 UTC; last answer 12 Nov 2026 13:03 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
MiniCPM5-2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from openbmb/MiniCPM5-2B@42433e90dad95123ab87729fd7011e7110629556, vLLM reasoning parser qwen3, one RTX PRO 6000 on our GPU host alone; thinking on (enable_thinking true; on/off only, no levels); sampling temperature 1, top_p 0.95, from generation_config.json; seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 2.52B / 2.52B active / total (dense; checkpoint 2,516,756,480 incl. untied lm_head, text only; card: 2,516,756,480). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "seed": 0,
  "temperature": 1.0,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · claude-opus-5.5-sub_en_dontknow

Name
Claude Opus 5.5 chat
Setup
Claude Opus 5.5 · chat model · with the I don't know option F
Model
claude-opus-5-5 · provider claude-subscription
GPU
—
Cost
$22.15 for the run, API list-price equivalent · Claude subscription
Answer time
run claude-opus-5.5-sub_en_dontknow · first answer 2026-11-12 12:14:49 UTC · last answer 2026-11-12 14:26:31 UTC · span 7902.0 s, pauses included (not charged) · charged 7902.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker (closed weights) · source https://platform.claude.com/docs/en/about-claude/models · read 2026-11-13
Input
text, English
Endpoint
claude-code-subscription
Started
12 Nov 2026 12:14 UTC; last answer 12 Nov 2026 14:26 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Claude Opus 5.5 on the Claude subscription through the Claude Code CLI (`claude --print`, as turkce's runs 968/971), model claude-opus-5-5, --effort high (the highest of low/medium/high/xhigh/max; adaptive thinking always on); CLAUDE_CODE_MAX_OUTPUT_TOKENS 32768; no temperature sent (Anthropic's default 1.0); Claude Code's agent prompt replaced by a neutral one-line system prompt, tools, settings, MCP servers and session saving off, API-key variables removed; one question per request, one request at a time; cost $0 (Claude subscription); api_equivalent_usd is Claude Code's API-list-price estimate, not a cost. English translations.
Parameters
{
  "effort": "high",
  "max_output_tokens": 32768,
  "route": "claude-code-cli --print",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message."
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · claude-sonnet-5.5-sub_en_dontknow

Name
Claude Sonnet 5.5 chat
Setup
Claude Sonnet 5.5 · chat model · with the I don't know option F
Model
claude-sonnet-5-5 · provider claude-subscription
GPU
—
Cost
$12.78 for the run, API list-price equivalent · Claude subscription
Answer time
run claude-sonnet-5.5-sub_en_dontknow · first answer 2026-11-12 13:31:26 UTC · last answer 2026-11-12 16:03:52 UTC · span 9146.0 s, pauses included (not charged) · charged 9146.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker (closed weights) · source https://platform.claude.com/docs/en/about-claude/models · read 2026-11-13
Input
text, English
Endpoint
claude-code-subscription
Started
12 Nov 2026 13:31 UTC; last answer 12 Nov 2026 16:03 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Claude Sonnet 5.5 on the Claude subscription through the Claude Code CLI (`claude --print`, as turkce's runs 968/971), model claude-sonnet-5-5, --effort high (the highest of low/medium/high/xhigh/max; adaptive thinking always on); CLAUDE_CODE_MAX_OUTPUT_TOKENS 32768; no temperature sent (Anthropic's default 1.0); Claude Code's agent prompt replaced by a neutral one-line system prompt, tools, settings, MCP servers and session saving off, API-key variables removed; one question per request, one request at a time; cost $0 (Claude subscription); api_equivalent_usd is Claude Code's API-list-price estimate, not a cost. English translations.
Parameters
{
  "effort": "high",
  "max_output_tokens": 32768,
  "route": "claude-code-cli --print",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message."
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · gemini-4-carbon-sub_en

Name
Gemini 4 Carbon chat
Setup
Gemini 4 Carbon (Antigravity subscription) · chat model
Model
gemini-4-carbon · provider Antigravity (Gemini subscription)
GPU
—
Cost
not measured
Answer time
run gemini-4-carbon-sub_en · first answer 2026-11-12 14:48:53 UTC · last answer 2026-11-14 12:53:40 UTC · span 165887.0 s, pauses included (not charged) · charged 69630.0 s in 24 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.7 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 1 answer cut off at it with no letter (counted wrong) · 8 empty replies asked once more
Model size
Parameters: not disclosed by maker (closed weights) · source https://ai.google.dev/gemini-api/docs/models · read 2026-11-11
Input
text, English
Endpoint
http://lan-host:8899/v1/chat/completions
Started
12 Nov 2026 14:47 UTC; last answer 14 Nov 2026 12:53 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Gemini 4 Carbon (gemini-4-carbon = gemini-4-carbon-high) through Can's Antigravity subscription (antigravity-benchmark #4762; Umit via zeki #5133): tools denied, ~10.7k tokens of Antigravity's own context per request, one request at a time; temperature 1.0 and max_tokens 32,768 sent but ignored by Antigravity (native sampling, output cap 262,144). A separate setup from the paid API run gemini-4-carbon_en. 2026-11-14T01:20:09Z: the Antigravity (Gemini subscription) bridge now answers up to 4 requests at once (antigravity-benchmark #6946); this run still sends one at a time; Carbon, Hydrogen and turkce's Reading arms share the bridge, so a request time can include the bridge's queue. 2026-11-14T06:49:26Z: two requests at a time from here on (turkce freed one of her bridge lanes, #7088); before, one at a time. Accuracy unaffected; request times not comparable across the switch. Bridge address changed 14 Nov 17:44 UTC from lan-host to lan-host (the Mac moved to its wired adapter after a network change; same Antigravity (Gemini subscription) service, same settings; antigravity-benchmark #7808); answers after that came over the new address.
Parameters
{
  "max_tokens": 32768,
  "temperature": 1.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · claude-fable-5.2-sub-r2_en

Name
Claude Fable 5.2 chat
Setup
Claude Fable 5.2 · chat model
Model
claude-fable-5-2 · provider claude-subscription
GPU
—
Cost
$0 for the run · Claude subscription
Answer time
run claude-fable-5.2-sub-r2_en · first answer 2026-11-13 15:56:44 UTC · last answer 2026-11-14 01:46:56 UTC · span 35412.0 s, pauses included (not charged) · charged 6451.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker (closed weights) · source https://platform.claude.com/docs/en/about-claude/models · read 2026-11-13
Input
text, English
Endpoint
claude-code-subscription
Started
12 Nov 2026 15:52 UTC; last answer 14 Nov 2026 01:46 UTC
Progress
127 answered of 1,125 asked; 2 request errors; in progress
Notes
Run 2 of Claude Fable 5.2 (max), gated (fable52_integration v2 tls, zeki #5397-#5455): Claude Code CLI 2.1.312 (`claude --print`) on the Claude subscription, model claude-fable-5-2, --effort max, CLAUDE_CODE_MAX_OUTPUT_TOKENS 32768, the neutral one-line system prompt, tools/settings/MCP/session saving off, API-key variables removed; each question through a local gate: one model reply per question, its FIRST reply stored (refusal or cap = no letter, never re-asked), served model must be exactly claude-fable-5-2 (else policy failure, run stops), at most 3 transport attempts per invocation (pending, asked again on resume), HTTPS only to https://api.anthropic.com with verified TLS and no redirects; one question at a time; cost $0 (Claude subscription); api_equivalent_usd is Claude Code's estimate, not a cost. Run 1 (claude-fable-5.2-sub_en) is kept apart. English translations.
Parameters
{
  "effort": "max",
  "gate": {
    "admitted": [
      "claude-fable-5-2"
    ],
    "first_outcome": "StrictStore",
    "max_transport_attempts": 3,
    "upstream": "https://api.anthropic.com",
    "version": "fable52_integration v2"
  },
  "max_output_tokens": 32768,
  "route": "claude-code-cli --print",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message."
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · lfm2.5-24b-a2b_en

Name
LFM2.5-24B-A2B on · chat
Setup
LFM2.5-24B-A2B (chat, thinking, on our GPU) · chat model
Model
lfm2.5-24b-a2b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$8.507 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 3.42 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $8.507. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run lfm2.5-24b-a2b_en · first answer 2026-11-12 16:14:21 UTC · last answer 2026-11-12 19:39:20 UTC · span 12299.0 s, pauses included (not charged) · charged 12299.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $8.5068 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 30.3 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 96 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 24.0B MoE · 2.45B active · source https://huggingface.co/LiquidAI/LFM2.5-24B-A2B · read 2026-11-12 · the card states 24.0B total / 2.45B active (the maker's figure is shown); checkpoint count 23.97B / 2.45B active (MoE, 4 of 64 experts; checkpoint 23,971,587,712, active 2,454,808,192 incl. tied embeddings)
Input
text, English
Endpoint
http://127.0.0.1:18618/v1/chat/completions
Started
12 Nov 2026 16:14 UTC; last answer 12 Nov 2026 19:39 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
LFM2.5-24B-A2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from LiquidAI/LFM2.5-24B-A2B@1769516ac588c4c558089c94f27d96b0dc67b183, vLLM reasoning parser qwen3 (the revision backgammon served (jevgammon tools/v3/servers/lfm25-24b.sh), not today's main 3dc4cd09; parser qwen3 as our other LFM2.5 runs, where backgammon's server used deepseek_r1), one RTX PRO 6000 on our GPU host alone; thinking always on (the template opens <think> for every answer; no switch, no levels); sampling temperature 0.1, top_k 50, repetition_penalty 1.05, from generation_config.json; seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 2.45B / 23.97B active / total (MoE, 4 of 64 experts; checkpoint 23,971,587,712, active 2,454,808,192 incl. tied embeddings; card: 24.0B total, 2.45B active). English translations (DeepSeek + review). No thinking-off setup (bus #6516/#6924/#8187/#8202): the current served chat template exposes no supported switch for a separate thinking-off run; no separate off variant has been run or scored. (Checked 15 Nov in the pinned template: its only option, preserve_thinking, keeps or drops earlier turns' thinking.)
Parameters
{
  "max_tokens": 32768,
  "repetition_penalty": 1.05,
  "seed": 0,
  "temperature": 0.1,
  "top_k": 50
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · knowline-2b_en

Name
KnowLine-2B-Gen1
Setup
KnowLine-2B-Gen1 · decision model
Model
PelaAI/KnowLine-2B-Gen1 · provider knowline
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.004256 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.02 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.004 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run knowline-2b_en · first answer 2026-11-12 16:33:55 UTC · last answer 2026-11-12 16:35:07 UTC · span 72.0 s, pauses included (not charged) · charged 72.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0043 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 2B dense · source https://huggingface.co/PelaAI/KnowLine-2B-Gen1 · read 2026-11-11 · the card states a 2B decision model; checkpoint count 2.27B (counted from the checkpoint (2,274,069,824: language model 1,881,825,088, MTP layer 60,828,160 (model-extra-from-base.safetensors), vision tower 331,416,576); a LoRA fine-tune of Qwen3.5-2B merged into the weights; dense; the card: 2B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
12 Nov 2026 16:33 UTC; last answer 12 Nov 2026 16:35 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup KnowLine-2B-Gen1 (site run None); runner df9ae2734; PelaAI/KnowLine-2B-Gen1@53245d470a4ce316e32fd0d18c24e142c697f40b (Apache-2.0 weights and code; one-epoch LoRA SFT of Qwen3.5-2B merged into BF16, readout temperature 1.32 folded into the weights, score questions divided by temperatures.json) behind the card's serve_knowline.sh: SGLang 0.5.21 in BF16 (--enable-fp32-lm-head --mamba-radix-cache-strategy extra_buffer, MEM default 0.85) on port+1000 (20490) and knowline_server.py (read twice up to 3,000 tokens): cd ~/decision-models/KnowLine-2B-Gen1@53245d470a4ce316e32fd0d18c24e142c697f40b && PATH=~/envs/knowline-sglang/bin:$PATH HOST=0.0.0.0 bash serve_knowline.sh ~/decision-models/KnowLine-2B-Gen1@53245d470a4ce316e32fd0d18c24e142c697f40b 0 19490, env ~/envs/knowline-sglang (sglang 0.5.21, torch 2.13.0+cu130, transformers 5.12.1); vram_mib is MEM 0.85 of the unified memory (SGLang sizes its KV cache to it), so it cannot run beside another large server; vision on (the server's default); bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'knowline'
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "PelaAI/KnowLine-2B-Gen1",
    "--provider",
    "knowline",
    "--timeout",
    "60",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "6ad0eac51a4abb950bde74b24687209aec628aa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "PelaAI/KnowLine-2B-Gen1",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "knowline",
  "questions": 1125,
  "request_seconds": 67.47958214126993,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "3c768c86942c3442dfffdb2f6d60dffed026c672e5fe8237b8ecb212671a5fbe",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 836764,
    "output_tokens": 1125
  }
}
Prompt template
POST http://spark2:19490/v1/systemone
{
 "model": "PelaAI/KnowLine-2B-Gen1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · knowline-2b_en_dontknow

Name
KnowLine-2B-Gen1
Setup
KnowLine-2B-Gen1 · decision model · with the I don't know option F
Model
PelaAI/KnowLine-2B-Gen1 · provider knowline
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.004375 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0206 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.004 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run knowline-2b_en_dontknow · first answer 2026-11-12 16:37:12 UTC · last answer 2026-11-12 16:38:26 UTC · span 74.0 s, pauses included (not charged) · charged 74.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0044 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 2B dense · source https://huggingface.co/PelaAI/KnowLine-2B-Gen1 · read 2026-11-11 · the card states a 2B decision model; checkpoint count 2.27B (counted from the checkpoint (2,274,069,824: language model 1,881,825,088, MTP layer 60,828,160 (model-extra-from-base.safetensors), vision tower 331,416,576); a LoRA fine-tune of Qwen3.5-2B merged into the weights; dense; the card: 2B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
12 Nov 2026 16:37 UTC; last answer 12 Nov 2026 16:38 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup KnowLine-2B-Gen1 (site run None); runner df9ae2734; PelaAI/KnowLine-2B-Gen1@53245d470a4ce316e32fd0d18c24e142c697f40b (Apache-2.0 weights and code; one-epoch LoRA SFT of Qwen3.5-2B merged into BF16, readout temperature 1.32 folded into the weights, score questions divided by temperatures.json) behind the card's serve_knowline.sh: SGLang 0.5.21 in BF16 (--enable-fp32-lm-head --mamba-radix-cache-strategy extra_buffer, MEM default 0.85) on port+1000 (20490) and knowline_server.py (read twice up to 3,000 tokens): cd ~/decision-models/KnowLine-2B-Gen1@53245d470a4ce316e32fd0d18c24e142c697f40b && PATH=~/envs/knowline-sglang/bin:$PATH HOST=0.0.0.0 bash serve_knowline.sh ~/decision-models/KnowLine-2B-Gen1@53245d470a4ce316e32fd0d18c24e142c697f40b 0 19490, env ~/envs/knowline-sglang (sglang 0.5.21, torch 2.13.0+cu130, transformers 5.12.1); vram_mib is MEM 0.85 of the unified memory (SGLang sizes its KV cache to it), so it cannot run beside another large server; vision on (the server's default); bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'knowline'
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "PelaAI/KnowLine-2B-Gen1",
    "--provider",
    "knowline",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "6ad0eac51a4abb950bde74b24687209aec628aa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "PelaAI/KnowLine-2B-Gen1",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "knowline",
  "questions": 1125,
  "request_seconds": 69.47055103478488,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "850f68bd7117fba0e4f99507d53e4f83b6831b6f60f117dd9292103c5b3542d5",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 904264,
    "output_tokens": 1125
  }
}
Prompt template
POST http://spark2:19490/v1/systemone
{
 "model": "PelaAI/KnowLine-2B-Gen1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · sieve-31b_en

Name
Sieve 31B v3
Setup
Sieve 31B v3 · decision model
Model
sthanika-ai/Sieve-31B-v3 · provider sieve
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.07082 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.333 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.071 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run sieve-31b_en · first answer 2026-11-12 17:27:55 UTC · last answer 2026-11-12 17:47:53 UTC · span 1198.0 s, pauses included (not charged) · charged 1198.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0708 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 31B dense · source https://huggingface.co/sthanika-ai/Sieve-31B-v3 · read 2026-11-10 · the official model name: Sieve 31B v3 (turkce #4263: the maker's stated total is shown); counted from the released files: 31.27B (counted from the checkpoint (31,273,088,876: language model 30,697,345,340, vision tower 575,743,536); a full fine-tune of Gemma 4 31B IT; the card: 29.29B trained parameters (the decoder projections))
Input
text, English
Endpoint
internal address on a shared Spark host
Started
12 Nov 2026 17:27 UTC; last answer 12 Nov 2026 17:47 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Sieve 31B v3 (site run None); runner df9ae2734; sthanika-ai/Sieve-31B-v3@8fcee858f7f20a50bfa3c8f984c22adb3cbb46a2 (Apache-2.0; Gemma 4 31B IT full fine-tune, letters readout from head.pt, each question read in 2 option orders) behind the card's sieve-serve (github.com/sthanika-ai/Sieve main at 1e64657, v0.5.1; the card asks for sieve >= 0.4.0): ~/envs/sieve/bin/sieve-serve --model ~/decision-models/Sieve-31B-v3@8fcee858f7f20a50bfa3c8f984c22adb3cbb46a2 --host 0.0.0.0 --port 19370 (env ~/envs/sieve: the repository's requirements.txt exactly (transformers 5.17.0, peft 0.21.0, accelerate 1.15.0, fastapi 0.141.1, uvicorn 0.53.0, pydantic 2.13.5, ...) except torch 2.14.0+cu130 instead of the tested 2.8.0, plus sieve-decisions 0.5.1 from the checkout at 1e64657); text only; bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'sieve' (System One body at /v1/decide, no model field; probabilities rounded to 4 places, no usage.input_tokens)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "sthanika-ai/Sieve-31B-v3",
    "--provider",
    "sieve",
    "--timeout",
    "120",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "6ad0eac51a4abb950bde74b24687209aec628aa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "sthanika-ai/Sieve-31B-v3",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "sieve",
  "questions": 1125,
  "request_seconds": 1190.228150093346,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "b119e96348258df67a933827c476d985388ced6cda584a2db18af9ee41c620af",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 0,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19370/v1/decide
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · lfm3-1.7b_en_dontknow

Name
LFM3-1.7B on · chat
Setup
LFM3-1.7B (chat, thinking, on our GPU) · chat model · with the I don't know option F
Model
lfm3-1.7b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.104 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.443 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.104. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run lfm3-1.7b_en_dontknow · first answer 2026-11-12 18:12:16 UTC · last answer 2026-11-12 18:38:52 UTC · span 1596.0 s, pauses included (not charged) · charged 1596.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.1039 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.2 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 10 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 1.73B dense · from the run's notes (dense; BF16 safetensors 1,728,801,536, text only; the GGUF of the llama.cpp answers: 1,728,801,600 incl. a 64-value rope_freqs table; card: 1.7B); also https://huggingface.co/LiquidAI/LFM3-1.7B-GGUF · read 2026-11-09 · the released checkpoint's tensors without registered buffers: 1,728,801,536 (the served BF16 GGUF also stores a 64-element RoPE buffer, 1,728,801,600); text only; card: 1.7B
Input
text, English
Endpoint
http://127.0.0.1:18605/v1/chat/completions
Started
12 Nov 2026 18:12 UTC; last answer 12 Nov 2026 18:38 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
LFM3-1.7B, served with vLLM 0.34.0rc1.dev302+gbe47189f3 (nightly, commit be47189f37f6, CUDA 13.5 image; docker vllm/vllm-openai@sha256:7f0618da95bac2f1cc97fcd1791380d7c1a06d24a4998da45aeb7fbe6c49179e), BF16 from LiquidAI/LFM3-1.7B@0dc4268b561499c648c7ba7c9466bf3d8d01517a, vLLM reasoning parser qwen3 (vLLM nightly because no vLLM release has Lfm3ForCausalLM; the same weights as the llama.cpp setup's official BF16 GGUF LiquidAI/LFM3-1.7B-GGUF@cd0c74c5 (tensors byte-identical), the same chat template and prompt token ids; the dont_know runs ran entirely on this engine, the standard runs lfm3-1.7b_en began on llama.cpp b12020 and switched part-way (ENGINE BOUNDARY in their notes); qualification work/lfm3_vllm_qual.md), one RTX PRO 6000 on our GPU host alone; thinking on (enable_thinking true, the template default; on/off only); sampling temperature 0.1, top_k 50, repetition_penalty 1.1, top_p 1, min_p 0, from generation_config.json, passed 1:1 as vLLM sampling params (repetition_penalty over prompt and output, as the llama.cpp setup's repeat_penalty over the whole context); seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 1.73B / 1.73B active / total (dense; BF16 safetensors 1,728,801,536, text only; the GGUF of the llama.cpp answers: 1,728,801,600 incl. a 64-value rope_freqs table; card: 1.7B). English translations (DeepSeek + review). All answers: 4 llama-server instances × 8 slots (same build, GGUF, flags, sampling, per-slot context 69,632, 32 requests in flight as in lfm3-1.7b_en) behind a forwarder on the same port, for throughput only (zeki #5247/#5265).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "repetition_penalty": 1.1,
  "seed": 0,
  "temperature": 0.1,
  "top_k": 50,
  "top_p": 1.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · sieve-31b_en_dontknow

Name
Sieve 31B v3
Setup
Sieve 31B v3 · decision model · with the I don't know option F
Model
sthanika-ai/Sieve-31B-v3 · provider sieve
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.07969 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.374 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.080 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run sieve-31b_en_dontknow · first answer 2026-11-12 18:12:31 UTC · last answer 2026-11-12 18:34:59 UTC · span 1348.0 s, pauses included (not charged) · charged 1348.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0797 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 31B dense · source https://huggingface.co/sthanika-ai/Sieve-31B-v3 · read 2026-11-10 · the official model name: Sieve 31B v3 (turkce #4263: the maker's stated total is shown); counted from the released files: 31.27B (counted from the checkpoint (31,273,088,876: language model 30,697,345,340, vision tower 575,743,536); a full fine-tune of Gemma 4 31B IT; the card: 29.29B trained parameters (the decoder projections))
Input
text, English
Endpoint
internal address on a shared Spark host
Started
12 Nov 2026 18:12 UTC; last answer 12 Nov 2026 18:34 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Sieve 31B v3 (site run None); runner df9ae2734; sthanika-ai/Sieve-31B-v3@8fcee858f7f20a50bfa3c8f984c22adb3cbb46a2 (Apache-2.0; Gemma 4 31B IT full fine-tune, letters readout from head.pt, each question read in 2 option orders) behind the card's sieve-serve (github.com/sthanika-ai/Sieve main at 1e64657, v0.5.1; the card asks for sieve >= 0.4.0): ~/envs/sieve/bin/sieve-serve --model ~/decision-models/Sieve-31B-v3@8fcee858f7f20a50bfa3c8f984c22adb3cbb46a2 --host 0.0.0.0 --port 19370 (env ~/envs/sieve: the repository's requirements.txt exactly (transformers 5.17.0, peft 0.21.0, accelerate 1.15.0, fastapi 0.141.1, uvicorn 0.53.0, pydantic 2.13.5, ...) except torch 2.14.0+cu130 instead of the tested 2.8.0, plus sieve-decisions 0.5.1 from the checkout at 1e64657); text only; bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'sieve' (System One body at /v1/decide, no model field; probabilities rounded to 4 places, no usage.input_tokens)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "sthanika-ai/Sieve-31B-v3",
    "--provider",
    "sieve",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "6ad0eac51a4abb950bde74b24687209aec628aa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "sthanika-ai/Sieve-31B-v3",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "sieve",
  "questions": 1125,
  "request_seconds": 1341.8881828036392,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "0101179db36cbd11443d42407e28a20508aaca57728722882e84adcd2302299c",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 0,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19370/v1/decide
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · lfm2.5-24b-a2b_en_dontknow

Name
LFM2.5-24B-A2B on · chat
Setup
LFM2.5-24B-A2B (chat, thinking, on our GPU) · chat model · with the I don't know option F
Model
lfm2.5-24b-a2b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$7.65 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 3.07 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $7.650. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run lfm2.5-24b-a2b_en_dontknow · first answer 2026-11-12 20:32:25 UTC · last answer 2026-11-12 23:36:45 UTC · span 11060.0 s, pauses included (not charged) · charged 11060.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $7.6498 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.6 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 50 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 24.0B MoE · 2.45B active · source https://huggingface.co/LiquidAI/LFM2.5-24B-A2B · read 2026-11-12 · the card states 24.0B total / 2.45B active (the maker's figure is shown); checkpoint count 23.97B / 2.45B active (MoE, 4 of 64 experts; checkpoint 23,971,587,712, active 2,454,808,192 incl. tied embeddings)
Input
text, English
Endpoint
http://127.0.0.1:18618/v1/chat/completions
Started
12 Nov 2026 20:31 UTC; last answer 12 Nov 2026 23:36 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
LFM2.5-24B-A2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from LiquidAI/LFM2.5-24B-A2B@1769516ac588c4c558089c94f27d96b0dc67b183, vLLM reasoning parser qwen3 (the revision backgammon served (jevgammon tools/v3/servers/lfm25-24b.sh), not today's main 3dc4cd09; parser qwen3 as our other LFM2.5 runs, where backgammon's server used deepseek_r1), one RTX PRO 6000 on our GPU host alone; thinking always on (the template opens <think> for every answer; no switch, no levels); sampling temperature 0.1, top_k 50, repetition_penalty 1.05, from generation_config.json; seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 2.45B / 23.97B active / total (MoE, 4 of 64 experts; checkpoint 23,971,587,712, active 2,454,808,192 incl. tied embeddings; card: 24.0B total, 2.45B active). English translations (DeepSeek + review). No thinking-off setup (bus #6516/#6924/#8187/#8202): the current served chat template exposes no supported switch for a separate thinking-off run; no separate off variant has been run or scored. (Checked 15 Nov in the pinned template: its only option, preserve_thinking, keeps or drops earlier turns' thinking.)
Parameters
{
  "max_tokens": 32768,
  "repetition_penalty": 1.05,
  "seed": 0,
  "temperature": 0.1,
  "top_k": 50
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · gemma-4-e2b-it-think_en

Name
Gemma 4 E2B on · chat
Setup
Gemma 4 E2B (chat, thinking on, on our GPU) · chat model
Model
gemma-4-e2b-it-think · provider vllm
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.4087 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 1.92 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.409 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gemma-4-e2b-it-think_en · first answer 2026-11-12 20:51:08 UTC · last answer 2026-11-12 22:46:21 UTC · span 6913.0 s, pauses included (not charged) · charged 6913.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.4087 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 29.6 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 50 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 5.1B dense · 2.3B effective · source https://huggingface.co/google/gemma-4-E2B-it · read 2026-11-10 · the card states 2.3B effective (5.1B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 5.12B (counted from the checkpoint (5,123,178,979: language model 4.65B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 5.1B with embeddings, 2.3B effective; not a mixture of experts)
Input
text, English
Endpoint
http://lan-host:18624/v1/chat/completions
Started
12 Nov 2026 20:51 UTC; last answer 12 Nov 2026 22:46 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Gemma 4 E2B, served with vLLM v0.30.0 (docker vllm/vllm-openai@sha256:8a69ffad015f138d7170c4ddc429e230a3bc1c1719f67e14324749df200a4b90, the v0.30.0 release's index digest, linux/arm64 build sha256:4864d46625cb…; vLLM commit ced6857afa0e), BF16 from google/gemma-4-E2B-it@4adf15748de9e1a0097b6274cee9c15836c41c37, vLLM reasoning parser gemma4, one DGX Spark (a shared Spark host, NVIDIA GB10, 121 GB unified memory) alone; thinking on (chat_template_kwargs enable_thinking true, the template's own switch: it opens a system turn with <|think|>; the template default is off; on/off only, no levels); sampling temperature 0, as the Gemma 4 31B chat run; no seed (as the Gemma 4 31B run); max_tokens 8,192; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 5.12B / 5.12B active / total (dense; checkpoint 5,123,178,979 incl. vision 0.17B and audio 0.31B towers; card: 5.1B with per-layer embeddings, 2.3B effective). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 8192,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · gemma-4-e2b-it-think_en_dontknow

Name
Gemma 4 E2B on · chat
Setup
Gemma 4 E2B (chat, thinking on, on our GPU) · chat model · with the I don't know option F
Model
gemma-4-e2b-it-think · provider vllm
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.3861 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 1.81 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.386 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gemma-4-e2b-it-think_en_dontknow · first answer 2026-11-13 00:12:37 UTC · last answer 2026-11-13 02:01:28 UTC · span 6531.0 s, pauses included (not charged) · charged 6531.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.3861 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 31.3 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 39 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 5.1B dense · 2.3B effective · source https://huggingface.co/google/gemma-4-E2B-it · read 2026-11-10 · the card states 2.3B effective (5.1B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 5.12B (counted from the checkpoint (5,123,178,979: language model 4.65B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 5.1B with embeddings, 2.3B effective; not a mixture of experts)
Input
text, English
Endpoint
http://lan-host:18624/v1/chat/completions
Started
13 Nov 2026 00:12 UTC; last answer 13 Nov 2026 02:01 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Gemma 4 E2B, served with vLLM v0.30.0 (docker vllm/vllm-openai@sha256:8a69ffad015f138d7170c4ddc429e230a3bc1c1719f67e14324749df200a4b90, the v0.30.0 release's index digest, linux/arm64 build sha256:4864d46625cb…; vLLM commit ced6857afa0e), BF16 from google/gemma-4-E2B-it@4adf15748de9e1a0097b6274cee9c15836c41c37, vLLM reasoning parser gemma4, one DGX Spark (a shared Spark host, NVIDIA GB10, 121 GB unified memory) alone; thinking on (chat_template_kwargs enable_thinking true, the template's own switch: it opens a system turn with <|think|>; the template default is off; on/off only, no levels); sampling temperature 0, as the Gemma 4 31B chat run; no seed (as the Gemma 4 31B run); max_tokens 8,192; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 5.12B / 5.12B active / total (dense; checkpoint 5,123,178,979 incl. vision 0.17B and audio 0.31B towers; card: 5.1B with per-layer embeddings, 2.3B effective). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 8192,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · spark-x2.6-4b_en

Name
Spark-X2.6-4B high · chat
Setup
Spark-X2.6-4B (chat, thinking, on our GPU) · chat model
Model
spark-x2.6-4b · provider llama.cpp
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.231 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.494 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.231. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-13 01:51:02–02:13:02 UTC
GPU time
run spark-x2.6-4b_en · first answer 2026-11-13 01:26:31 UTC · last answer 2026-11-13 02:18:11 UTC · span 3100.0 s, pauses included (not charged) · charged 1780.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.2312 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 8.1 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 4.11B dense · from the run's notes (dense; checkpoint 4,112,079,360 incl. tied embeddings (the BF16 GGUF holds the same 4,112,079,360); also https://huggingface.co/XHToken/Spark-X2.6-4B · read 2026-11-12 · counted from the checkpoint (4,112,079,360 incl. tied embeddings, text only; dense); the card states no count beyond the name's 4B
Input
text, English
Endpoint
http://127.0.0.1:18615/v1/chat/completions
Started
13 Nov 2026 01:25 UTC; last answer 13 Nov 2026 02:18 UTC
Progress
20 answered of 1,125 asked; 0 request errors; in progress
Notes
Spark-X2.6-4B, served with llama.cpp server b12020 (commit 0ec4027, CUDA build), official BF16 GGUF XHToken/Spark-X2.6-4B-GGUF@9d716770ac7477776de5f891f70cdf98b003a5cd Spark-X2.6-4B.gguf (vLLM v0.30.0 has no Spark2_5ForCausalLM), chat template from XHToken/Spark-X2.6-4B@64ca8f2a295b8ceb2757b9d4749a6558c4759e5b, llama.cpp --reasoning-format deepseek, one RTX PRO 6000 on our GPU host alone; thinking on at its highest level (enable_thinking true, reasoning_effort high; levels low and high); sampling temperature 1, top_p 0.95, top_k 0, min_p 0, repeat_penalty 1, presence_penalty 0, frequency_penalty 0, from generation_config.json; its top_k -1 (off) as llama.cpp top_k 0, llama.cpp's own min_p default switched off; seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 4.11B / 4.11B active / total (dense; checkpoint 4,112,079,360 incl. tied embeddings (the BF16 GGUF holds the same 4,112,079,360), text only; card: 4B). English translations (DeepSeek + review). PAUSED (Main, 13 Nov 02:18Z): smoke cut at 20 of 25 Logic answers (0 Math), moved to the end of the queue; 5 in-flight Logic requests dropped (not stored); pace on this setup: llama.cpp main thread CPU-bound (99% of one core, GPU ~180 W), about 5-7 tokens/s per slot, the 20 stored answers took 107,490 tokens in 52 min; kept, not ranked.
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "reasoning_effort": "high"
  },
  "frequency_penalty": 0.0,
  "max_tokens": 32768,
  "min_p": 0.0,
  "presence_penalty": 0.0,
  "repeat_penalty": 1.0,
  "seed": 0,
  "temperature": 1.0,
  "top_k": 0,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · qwen3.5-4b_en

Name
Qwen3.5-4B on · chat
Setup
Qwen3.5-4B (chat, thinking, on our GPU) · chat model
Model
qwen3.5-4b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$3.965 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 1.59 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $3.965. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run qwen3.5-4b_en · first answer 2026-11-13 02:21:39 UTC · last answer 2026-11-13 03:57:12 UTC · span 5733.0 s, pauses included (not charged) · charged 5733.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $3.9653 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 30.0 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 2 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 4.66B dense · from the run's notes (dense; checkpoint 4,659,865,088 incl. vision 0.33B; card: 4B language model); also https://huggingface.co/Qwen/Qwen3.5-4B · read 2026-11-09 · counted from the checkpoint (4,659,865,088: language model 4.33B, vision encoder 0.33B); the card: 4B language model
Input
text, English
Endpoint
http://127.0.0.1:18604/v1/chat/completions
Started
13 Nov 2026 02:21 UTC; last answer 13 Nov 2026 03:57 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Qwen3.5-4B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from Qwen/Qwen3.5-4B@851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a, vLLM reasoning parser qwen3 (vision tower loaded, for Reading's Page images; text-only prompts here), one RTX PRO 6000 on our GPU host alone; thinking on (enable_thinking true; on/off only, no levels); sampling temperature 1, top_p 0.95, top_k 20, min_p 0, presence_penalty 1.5, repetition_penalty 1, sampling from the model card, generation_config has none (thinking mode, general tasks; the repo has no generation_config.json); seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 4.66B / 4.66B active / total (dense; checkpoint 4,659,865,088 incl. vision 0.33B; card: 4B language model). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "presence_penalty": 1.5,
  "repetition_penalty": 1.0,
  "seed": 0,
  "temperature": 1.0,
  "top_k": 20,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · gemma-4-e4b-it-think_en

Name
Gemma 4 E4B on · chat
Setup
Gemma 4 E4B (chat, thinking on, on our GPU) · chat model
Model
gemma-4-e4b-it-think · provider vllm
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.7307 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 3.43 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.731 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gemma-4-e4b-it-think_en · first answer 2026-11-13 03:31:41 UTC · last answer 2026-11-13 06:57:42 UTC · span 12361.0 s, pauses included (not charged) · charged 12361.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.7307 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 28.6 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 61 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 8B dense · 4.5B effective · source https://huggingface.co/google/gemma-4-E4B-it · read 2026-11-10 · the card states 4.5B effective (8B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 8.00B (counted from the checkpoint (7,996,157,418: language model 7.52B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 8B with embeddings, 4.5B effective; not a mixture of experts)
Input
text, English
Endpoint
http://lan-host:18625/v1/chat/completions
Started
13 Nov 2026 03:31 UTC; last answer 13 Nov 2026 06:57 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Gemma 4 E4B, served with vLLM v0.30.0 (docker vllm/vllm-openai@sha256:8a69ffad015f138d7170c4ddc429e230a3bc1c1719f67e14324749df200a4b90, the v0.30.0 release's index digest, linux/arm64 build sha256:4864d46625cb…; vLLM commit ced6857afa0e), BF16 from google/gemma-4-E4B-it@4ecc2ba63bcac23ba4a767fc87025b1450f84c9c, vLLM reasoning parser gemma4, one DGX Spark (a shared Spark host, NVIDIA GB10, 121 GB unified memory) alone; thinking on (chat_template_kwargs enable_thinking true, the template's own switch: it opens a system turn with <|think|>; the template default is off; on/off only, no levels); sampling temperature 0, as the Gemma 4 31B chat run; no seed (as the Gemma 4 31B run); max_tokens 8,192; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 8.00B / 8.00B active / total (dense; checkpoint 7,996,157,418 incl. vision 0.17B and audio 0.31B towers; card: 8B with per-layer embeddings, 4.5B effective). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 8192,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · qwen3.5-4b_en_dontknow

Name
Qwen3.5-4B on · chat
Setup
Qwen3.5-4B (chat, thinking, on our GPU) · chat model · with the I don't know option F
Model
qwen3.5-4b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$3.588 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 1.44 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $3.588. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run qwen3.5-4b_en_dontknow · first answer 2026-11-13 05:06:09 UTC · last answer 2026-11-13 06:32:36 UTC · span 5187.0 s, pauses included (not charged) · charged 5187.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $3.5877 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.5 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 4.66B dense · from the run's notes (dense; checkpoint 4,659,865,088 incl. vision 0.33B; card: 4B language model); also https://huggingface.co/Qwen/Qwen3.5-4B · read 2026-11-09 · counted from the checkpoint (4,659,865,088: language model 4.33B, vision encoder 0.33B); the card: 4B language model
Input
text, English
Endpoint
http://127.0.0.1:18604/v1/chat/completions
Started
13 Nov 2026 05:05 UTC; last answer 13 Nov 2026 06:32 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Qwen3.5-4B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from Qwen/Qwen3.5-4B@851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a, vLLM reasoning parser qwen3 (vision tower loaded, for Reading's Page images; text-only prompts here), one RTX PRO 6000 on our GPU host alone; thinking on (enable_thinking true; on/off only, no levels); sampling temperature 1, top_p 0.95, top_k 20, min_p 0, presence_penalty 1.5, repetition_penalty 1, sampling from the model card, generation_config has none (thinking mode, general tasks; the repo has no generation_config.json); seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 4.66B / 4.66B active / total (dense; checkpoint 4,659,865,088 incl. vision 0.33B; card: 4B language model). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "presence_penalty": 1.5,
  "repetition_penalty": 1.0,
  "seed": 0,
  "temperature": 1.0,
  "top_k": 20,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · gemma-4-e4b-it-think_en_dontknow

Name
Gemma 4 E4B on · chat
Setup
Gemma 4 E4B (chat, thinking on, on our GPU) · chat model · with the I don't know option F
Model
gemma-4-e4b-it-think · provider vllm
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.6426 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 3.02 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.643 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gemma-4-e4b-it-think_en_dontknow · first answer 2026-11-13 09:19:59 UTC · last answer 2026-11-13 12:21:09 UTC · span 10870.0 s, pauses included (not charged) · charged 10870.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.6426 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 31.5 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 37 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 8B dense · 4.5B effective · source https://huggingface.co/google/gemma-4-E4B-it · read 2026-11-10 · the card states 4.5B effective (8B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 8.00B (counted from the checkpoint (7,996,157,418: language model 7.52B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 8B with embeddings, 4.5B effective; not a mixture of experts)
Input
text, English
Endpoint
http://lan-host:18625/v1/chat/completions
Started
13 Nov 2026 09:18 UTC; last answer 13 Nov 2026 12:21 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Gemma 4 E4B, served with vLLM v0.30.0 (docker vllm/vllm-openai@sha256:8a69ffad015f138d7170c4ddc429e230a3bc1c1719f67e14324749df200a4b90, the v0.30.0 release's index digest, linux/arm64 build sha256:4864d46625cb…; vLLM commit ced6857afa0e), BF16 from google/gemma-4-E4B-it@4ecc2ba63bcac23ba4a767fc87025b1450f84c9c, vLLM reasoning parser gemma4, one DGX Spark (a shared Spark host, NVIDIA GB10, 121 GB unified memory) alone; thinking on (chat_template_kwargs enable_thinking true, the template's own switch: it opens a system turn with <|think|>; the template default is off; on/off only, no levels); sampling temperature 0, as the Gemma 4 31B chat run; no seed (as the Gemma 4 31B run); max_tokens 8,192; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 8.00B / 8.00B active / total (dense; checkpoint 7,996,157,418 incl. vision 0.17B and audio 0.31B towers; card: 8B with per-layer embeddings, 4.5B effective). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 8192,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · granite-4.2-3b_en

Name
Granite-4.2-3B high · chat
Setup
Granite-4.2-3B (chat, thinking, on our GPU) · chat model
Model
granite-4.2-3b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.3721 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.149 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.372. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run granite-4.2-3b_en · first answer 2026-11-13 09:34:21 UTC · last answer 2026-11-13 09:43:19 UTC · span 538.0 s, pauses included (not charged) · charged 538.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.3721 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 13.5 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 8 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 3B dense · source https://huggingface.co/ibm-granite/granite-4.2-3b · read 2026-11-12 · the card states 3B (the maker's figure is shown); checkpoint count 3.66B (3,659,737,600 incl. untied lm_head, text only; dense)
Input
text, English
Endpoint
http://127.0.0.1:18616/v1/chat/completions
Started
13 Nov 2026 09:34 UTC; last answer 13 Nov 2026 09:43 UTC
Progress
25 answered of 1,125 asked; 0 request errors; smoke test at 32,768 tokens only; the full run uses a higher limit (run granite-4.2-3b-64k_en)
Notes
Granite-4.2-3B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from ibm-granite/granite-4.2-3b@4e9c275e232f10211ec88644d106a1f6a1b60e06, vLLM reasoning parser granite_thinking_parser, one RTX PRO 6000 on our GPU host alone; thinking on at full effort (enable_thinking true, reasoning_effort high; the template's levels: full or low); sampling temperature 1, top_p 0.95, from generation_config.json; seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 3.66B / 3.66B active / total (dense; checkpoint 3,659,737,600 incl. untied lm_head, text only; card: 3B). English translations (DeepSeek + review). SUPERSEDED by granite-4.2-3b-64k_en (cut-offs only; one retry at 65,536).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "reasoning_effort": "high"
  },
  "max_tokens": 32768,
  "seed": 0,
  "temperature": 1.0,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · granite-4.2-3b-64k_en

Name
Granite-4.2-3B high · chat
Setup
Granite-4.2-3B (chat, thinking, on our GPU) · chat model
Model
granite-4.2-3b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$12.25 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 4.92 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $12.254. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run granite-4.2-3b-64k_en · first answer 2026-11-13 09:45:13 UTC · last answer 2026-11-13 14:40:30 UTC · span 17717.0 s, pauses included (not charged) · charged 17717.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $12.2543 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 28.1 in flight (sum of request latencies / active time)
Output limit
65,536 tokens · 14 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 3B dense · source https://huggingface.co/ibm-granite/granite-4.2-3b · read 2026-11-12 · the card states 3B (the maker's figure is shown); checkpoint count 3.66B (3,659,737,600 incl. untied lm_head, text only; dense)
Input
text, English
Endpoint
http://127.0.0.1:18616/v1/chat/completions
Started
13 Nov 2026 09:45 UTC; last answer 13 Nov 2026 14:40 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Granite-4.2-3B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from ibm-granite/granite-4.2-3b@4e9c275e232f10211ec88644d106a1f6a1b60e06, vLLM reasoning parser granite_thinking_parser, one RTX PRO 6000 on our GPU host alone; thinking on at full effort (enable_thinking true, reasoning_effort high; the template's levels: full or low); sampling temperature 1, top_p 0.95, from generation_config.json; seed 0; max_tokens 65,536 (raised from 32,768 once: the smoke failed on cut-offs only); one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 3.66B / 3.66B active / total (dense; checkpoint 3,659,737,600 incl. untied lm_head, text only; card: 3B). English translations (DeepSeek + review). Full runs: 13 Nov 10:05-10:30Z at 48 requests in flight (vLLM --max-num-seqs 48; the KV cache filled, requests queued), from ~10:32Z at 32 in flight (--max-num-seqs 32), Main; the 25 smoke answers ran at 32 in flight with --max-num-seqs 16.
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "reasoning_effort": "high"
  },
  "max_tokens": 65536,
  "seed": 0,
  "temperature": 1.0,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · turion-jev-31b_en

Name
Turion Jev 31B
Setup
Turion Jev 31B · decision model
Model
turion-ai/turion-jev-31b · provider turion-jev
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.02802 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.132 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.028 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run turion-jev-31b_en · first answer 2026-11-13 11:34:44 UTC · last answer 2026-11-13 11:42:38 UTC · span 474.0 s, pauses included (not charged) · charged 474.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0280 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 31B dense · source https://huggingface.co/turion-ai/turion-jev-31b · read 2026-11-10 · the card states 31B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 31.27B (counted from the checkpoint (31,273,088,876: language model 30,697,345,340, vision tower 575,743,536, unused by its server); a fine-tune of Gemma 4 31B IT; the card: 31B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
13 Nov 2026 11:34 UTC; last answer 13 Nov 2026 11:42 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Turion Jev 31B (site run None); runner df9ae2734; turion-ai/turion-jev-31b@866c0fc6e040cba23fcc72113a80e5c99d5ebab1 (Apache-2.0; Gemma 4 31B IT LoRA merge; calibration.json temperatures) behind the card's server: ~/envs/turion-jev/bin/python -m turion_jev.serve --model ~/decision-models/turion-jev-31b@866c0fc6e040cba23fcc72113a80e5c99d5ebab1 --backend vllm --quantization fp8 --no-recheck --host 0.0.0.0 --port 19380; re-scoring off, as in the card's benchmark run; the server default is on; env ~/envs/turion-jev (vllm==0.32.0, torch 2.13.0+cu130, transformers 5.19.0, the repository's code/ with [serve]); text only; bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'turion-jev' (System One at /v1/systemone, model ignored, no confidence field, probabilities rounded to 6 places)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "turion-ai/turion-jev-31b",
    "--provider",
    "turion-jev",
    "--timeout",
    "120",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "6ad0eac51a4abb950bde74b24687209aec628aa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "turion-ai/turion-jev-31b",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "turion-jev",
  "questions": 1125,
  "request_seconds": 464.7950645760284,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "073991b319813d79992ca844e00af083f91078da8cb4123a8e9420a0740b2b31",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 868768,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19380/v1/systemone
{
 "model": "turion-ai/turion-jev-31b",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · claude-haiku-5.5-sub_en

Name
Claude Haiku 5.5 chat
Setup
Claude Haiku 5.5 · chat model
Model
claude-haiku-5-5 · provider claude-subscription
GPU
—
Cost
$0 for the run · Claude subscription
Answer time
run claude-haiku-5.5-sub_en · first answer 2026-11-13 11:52:47 UTC · last answer 2026-11-14 10:40:58 UTC · span 82091.0 s, pauses included (not charged) · charged 52531.0 s in 3 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker (closed weights) · source https://openrouter.ai/anthropic/claude-haiku-5.5 · read 2026-11-10
Input
text, English
Endpoint
claude-code-subscription
Started
13 Nov 2026 11:52 UTC; last answer 14 Nov 2026 10:40 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Claude Haiku 5.5 on the Claude subscription through the Claude Code CLI (`claude --print`, as turkce's runs 968/971), model claude-haiku-5-5, --effort max (the highest of low/medium/high/xhigh/max; adaptive thinking always on); CLAUDE_CODE_MAX_OUTPUT_TOKENS 32768; no temperature sent (Anthropic's default 1.0); Claude Code's agent prompt replaced by a neutral one-line system prompt, tools, settings, MCP servers and session saving off, API-key variables removed; one question per request, one request at a time; cost $0 (Claude subscription); api_equivalent_usd is Claude Code's API-list-price estimate, not a cost. English translations.
Parameters
{
  "effort": "max",
  "max_output_tokens": 32768,
  "route": "claude-code-cli --print",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message."
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · turion-jev-31b_en_dontknow

Name
Turion Jev 31B
Setup
Turion Jev 31B · decision model · with the I don't know option F
Model
turion-ai/turion-jev-31b · provider turion-jev
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.03139 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.147 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.031 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run turion-jev-31b_en_dontknow · first answer 2026-11-13 11:53:43 UTC · last answer 2026-11-13 12:02:34 UTC · span 531.0 s, pauses included (not charged) · charged 531.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0314 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 31B dense · source https://huggingface.co/turion-ai/turion-jev-31b · read 2026-11-10 · the card states 31B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 31.27B (counted from the checkpoint (31,273,088,876: language model 30,697,345,340, vision tower 575,743,536, unused by its server); a fine-tune of Gemma 4 31B IT; the card: 31B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
13 Nov 2026 11:53 UTC; last answer 13 Nov 2026 12:02 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Turion Jev 31B (site run None); runner df9ae2734; turion-ai/turion-jev-31b@866c0fc6e040cba23fcc72113a80e5c99d5ebab1 (Apache-2.0; Gemma 4 31B IT LoRA merge; calibration.json temperatures) behind the card's server: ~/envs/turion-jev/bin/python -m turion_jev.serve --model ~/decision-models/turion-jev-31b@866c0fc6e040cba23fcc72113a80e5c99d5ebab1 --backend vllm --quantization fp8 --no-recheck --host 0.0.0.0 --port 19380; re-scoring off, as in the card's benchmark run; the server default is on; env ~/envs/turion-jev (vllm==0.32.0, torch 2.13.0+cu130, transformers 5.19.0, the repository's code/ with [serve]); text only; bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'turion-jev' (System One at /v1/systemone, model ignored, no confidence field, probabilities rounded to 6 places)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "turion-ai/turion-jev-31b",
    "--provider",
    "turion-jev",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "6ad0eac51a4abb950bde74b24687209aec628aa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "turion-ai/turion-jev-31b",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "turion-jev",
  "questions": 1125,
  "request_seconds": 525.6723356174189,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "82602bbcdcbfb7a3cd1ee7f6d91c2d81907b870a2365adb81a4baffd2cd23593",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 976768,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19380/v1/systemone
{
 "model": "turion-ai/turion-jev-31b",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · claude-fable-5.1-sub_en

Name
Claude Fable 5.1 chat
Setup
Claude Fable 5.1 · chat model
Model
claude-fable-5-1 · provider claude-subscription
GPU
—
Cost
$0 for the run · Claude subscription
Answer time
run claude-fable-5.1-sub_en · first answer 2026-11-13 11:55:25 UTC · last answer 2026-11-13 14:11:07 UTC · span 8142.0 s, pauses included (not charged) · charged 8142.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker (closed weights) · source https://platform.claude.com/docs/en/about-claude/models · read 2026-11-13
Input
text, English
Endpoint
claude-code-subscription
Started
13 Nov 2026 11:55 UTC; last answer 13 Nov 2026 14:11 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Claude Fable 5.1 on the Claude subscription through the Claude Code CLI (`claude --print`, as turkce's runs 968/971), model claude-fable-5-1, --effort high (the highest of low/medium/high/xhigh/max; adaptive thinking always on); CLAUDE_CODE_MAX_OUTPUT_TOKENS 32768; no temperature sent (Anthropic's default 1.0); Claude Code's agent prompt replaced by a neutral one-line system prompt, tools, settings, MCP servers and session saving off, API-key variables removed; one question per request, one request at a time; cost $0 (Claude subscription); api_equivalent_usd is Claude Code's API-list-price estimate, not a cost. English translations.
Parameters
{
  "effort": "high",
  "max_output_tokens": 32768,
  "route": "claude-code-cli --print",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message."
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · claude-fable-5.1-sub_en_dontknow

Name
Claude Fable 5.1 chat
Setup
Claude Fable 5.1 · chat model · with the I don't know option F
Model
claude-fable-5-1 · provider claude-subscription
GPU
—
Cost
$0 for the run · Claude subscription
Answer time
run claude-fable-5.1-sub_en_dontknow · first answer 2026-11-13 14:11:13 UTC · last answer 2026-11-13 16:23:05 UTC · span 7912.0 s, pauses included (not charged) · charged 7912.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker (closed weights) · source https://platform.claude.com/docs/en/about-claude/models · read 2026-11-13
Input
text, English
Endpoint
claude-code-subscription
Started
13 Nov 2026 14:11 UTC; last answer 13 Nov 2026 16:23 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Claude Fable 5.1 on the Claude subscription through the Claude Code CLI (`claude --print`, as turkce's runs 968/971), model claude-fable-5-1, --effort high (the highest of low/medium/high/xhigh/max; adaptive thinking always on); CLAUDE_CODE_MAX_OUTPUT_TOKENS 32768; no temperature sent (Anthropic's default 1.0); Claude Code's agent prompt replaced by a neutral one-line system prompt, tools, settings, MCP servers and session saving off, API-key variables removed; one question per request, one request at a time; cost $0 (Claude subscription); api_equivalent_usd is Claude Code's API-list-price estimate, not a cost. English translations.
Parameters
{
  "effort": "high",
  "max_output_tokens": 32768,
  "route": "claude-code-cli --print",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message."
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · lfm3-1.7b-vllm-nothink_en

Name
LFM3-1.7B off · chat
Setup
LFM3-1.7B (chat, thinking off, on our GPU) · chat model
Model
lfm3-1.7b-vllm-nothink · provider vllm
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.5476 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 2.57 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.548 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run lfm3-1.7b-vllm-nothink_en · first answer 2026-11-13 14:46:30 UTC · last answer 2026-11-13 17:20:53 UTC · span 9263.0 s, pauses included (not charged) · charged 9263.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.5476 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 31.3 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 9 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 1.73B dense · from the run's notes (dense; BF16 safetensors 1,728,801,536, text only; card: 1.7B); also https://huggingface.co/LiquidAI/LFM3-1.7B-GGUF · read 2026-11-09 · the released checkpoint's tensors without registered buffers: 1,728,801,536 (the served BF16 GGUF also stores a 64-element RoPE buffer, 1,728,801,600); text only; card: 1.7B
Input
text, English
Endpoint
http://lan-host:18626/v1/chat/completions
Started
13 Nov 2026 14:45 UTC; last answer 13 Nov 2026 17:20 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
LFM3-1.7B, served with vLLM 0.34.0rc1.dev302+gbe47189f3 (nightly, commit be47189f37f6, CUDA 13.5 image; docker vllm/vllm-openai@sha256:7f0618da95bac2f1cc97fcd1791380d7c1a06d24a4998da45aeb7fbe6c49179e, linux/arm64 build sha256:c41b69fa5f34…), BF16 from LiquidAI/LFM3-1.7B@0dc4268b561499c648c7ba7c9466bf3d8d01517a, vLLM reasoning parser qwen3 (thinking off via chat_template_kwargs enable_thinking false, the template's own switch (the assistant turn opens with an empty <think></think>), not a prompt instruction; otherwise the settings of the thinking-on vLLM runs lfm3-1.7b_en (Umit #6101); vLLM nightly because no vLLM release has Lfm3ForCausalLM; qualification work/lfm3_vllm_qual.md), one DGX Spark (a shared Spark host, NVIDIA GB10, 121 GB unified memory) alone; thinking off (chat_template_kwargs enable_thinking false; the template default true is overridden); sampling temperature 0.1, top_k 50, repetition_penalty 1.1, top_p 1, min_p 0, from generation_config.json, passed 1:1 as vLLM sampling params; seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 1.73B / 1.73B active / total (dense; BF16 safetensors 1,728,801,536, text only; card: 1.7B). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "repetition_penalty": 1.1,
  "seed": 0,
  "temperature": 0.1,
  "top_k": 50,
  "top_p": 1.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · gpt-6.1-terra-sub_en

Name
GPT-6.1 Terra chat
Setup
GPT-6.1 Terra · chat model
Model
gpt-6.1-terra · provider chatgpt-subscription
GPU
—
Cost
$0 for the run · ChatGPT subscription
Answer time
run gpt-6.1-terra-sub_en · first answer 2026-11-13 19:32:05 UTC · last answer 2026-11-13 21:37:36 UTC · span 7531.0 s, pauses included (not charged) · charged 7531.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not on record yet
Input
text, English
Endpoint
codex-chatgpt-subscription
Started
13 Nov 2026 15:15 UTC; last answer 13 Nov 2026 21:37 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
GPT-6.1 Terra on the ChatGPT subscription through the Codex CLI (`codex exec`, ChatGPT sign-in, as turkce's ask_openai), model gpt-6.1-terra, model_reasoning_effort high (Reading's setting); no output limit sent (Codex has no setting for it) and no temperature; Codex's agent instructions replaced by a neutral one-line system prompt, web search and tool features off, read-only sandbox, no user config, rules or saved session, API-key variables removed; one question per request, one request at a time; a usage-limit reply stops the run without retrying it (resumable); cost $0 (ChatGPT subscription); api_equivalent_usd is the tokens at OpenAI's API list price {"input": 1.0, "cached_input": 0.05, "cache_write": 1.25, "output": 5.0} per 1M tokens, not a cost. Codex reports no served model id. English translations.
Parameters
{
  "disabled_features": [
    "shell_tool",
    "unified_exec",
    "goals",
    "apps",
    "plugins",
    "remote_plugin",
    "tool_suggest",
    "skill_search",
    "view_image",
    "image_generation",
    "sleep_tool",
    "browser_use",
    "browser_use_external",
    "computer_use",
    "in_app_browser",
    "multi_agent",
    "unbounded_connection_retries"
  ],
  "effort": "high",
  "list_price_per_mtok": {
    "cache_write": 1.25,
    "cached_input": 0.05,
    "input": 1.0,
    "output": 5.0
  },
  "max_output_tokens": null,
  "route": "codex exec (ChatGPT sign-in)",
  "sandbox": "read-only",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message.",
  "web_search": "disabled"
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · granite-4.2-3b-64k_en_dontknow

Name
Granite-4.2-3B high · chat
Setup
Granite-4.2-3B (chat, thinking, on our GPU) · chat model · with the I don't know option F
Model
granite-4.2-3b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$10.38 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 4.17 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $10.377. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-13 16:42:49–00:32:44 UTC; 2026-11-14 01:05:30–11:12:20 UTC; 2026-11-14 13:24:17–13:57:57 UTC
GPU time
run granite-4.2-3b-64k_en_dontknow · first answer 2026-11-13 15:29:19 UTC · last answer 2026-11-14 14:09:47 UTC · span 81628.0 s, pauses included (not charged) · charged 15003.0 s in 4 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $10.3771 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 28.6 in flight (sum of request latencies / active time)
Output limit
65,536 tokens · 4 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 3B dense · source https://huggingface.co/ibm-granite/granite-4.2-3b · read 2026-11-12 · the card states 3B (the maker's figure is shown); checkpoint count 3.66B (3,659,737,600 incl. untied lm_head, text only; dense)
Input
text, English
Endpoint
http://127.0.0.1:18616/v1/chat/completions
Started
13 Nov 2026 15:28 UTC; last answer 14 Nov 2026 14:09 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Granite-4.2-3B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from ibm-granite/granite-4.2-3b@4e9c275e232f10211ec88644d106a1f6a1b60e06, vLLM reasoning parser granite_thinking_parser, one RTX PRO 6000 on our GPU host alone; thinking on at full effort (enable_thinking true, reasoning_effort high; the template's levels: full or low); sampling temperature 1, top_p 0.95, from generation_config.json; seed 0; max_tokens 65,536 (raised from 32,768 once: the smoke failed on cut-offs only); one question per request (32 requests in flight), standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 3.66B / 3.66B active / total (dense; checkpoint 3,659,737,600 incl. untied lm_head, text only; card: 3B). English translations (DeepSeek + review). Interrupted 13 Nov 16:44Z (Main, Umit #6471: GPU 1 lent to the LFM3-1.7B vLLM rerun for a public post) at 162 stored answers; the requests in flight were dropped (not stored); resumes from the stored answers with the same server flags (32 in flight). Paused 2026-11-14T01:07:23Z for one hour so turkce runs Granite's Reading arms alone on the same server (#6931); in-flight requests dropped unstored, re-asked on resume.
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "reasoning_effort": "high"
  },
  "max_tokens": 65536,
  "seed": 0,
  "temperature": 1.0,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · qwen3.8-flash_en_dontknow

Name
Qwen3.8 Flash off · chat
Setup
Qwen3.8 Flash · chat model · with the I don't know option F
Model
qwen/qwen3.8-flash · provider openrouter
GPU
—
Cost
$1.495 for the run · provider price × reported usage
Answer time
run qwen3.8-flash_en_dontknow · first answer 2026-11-13 15:29:36 UTC · last answer 2026-11-13 16:30:22 UTC · span 3646.0 s, pauses included (not charged) · charged 3646.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 8.0 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: not disclosed by maker (closed weights) · source https://openrouter.ai/qwen/qwen3.8-flash · read 2026-11-13
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
13 Nov 2026 15:29 UTC; last answer 13 Nov 2026 16:30 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Qwen3.8 Flash through OpenRouter (qwen/qwen3.8-flash), provider pinned to alibaba, fallbacks off; reasoning {"enabled": false}; temperature 1.0; max_tokens 32768; one question per request; parameters not disclosed (closed weights). English translations.
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "alibaba"
    ]
  },
  "reasoning": {
    "enabled": false
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · startlux-4b_en

Name
StartLux-Decision-v1.1-4B-Instant
Setup
StartLux-Decision v1.1 4B · decision model
Model
startlux-models/StartLux-Decision-v1.1-4B-Instant · provider startlux
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.005202 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0244 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.005 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run startlux-4b_en · first answer 2026-11-13 16:00:15 UTC · last answer 2026-11-13 16:01:43 UTC · span 88.0 s, pauses included (not charged) · charged 88.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0052 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 4B dense · source https://huggingface.co/startlux-models/StartLux-Decision-v1.1-4B-Instant · read 2026-11-10 · the card states 4B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 4.66B (counted from the checkpoint (4,659,865,088: language model incl. MTP layer 4,326,350,848, vision tower 333,514,240); from Qwen3.5-4B; the card: 4B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
13 Nov 2026 16:00 UTC; last answer 13 Nov 2026 16:01 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup StartLux-Decision v1.1 4B (site run None); runner df9ae2734; startlux-models/StartLux-Decision-v1.1-4B-Instant@416088f77cfce09633906a3675c7cd2e5dabcf10 (weights CC BY-NC 4.0, non-commercial: run for this non-commercial benchmark; code Apache-2.0; Qwen3.5-4B, prefix-bidirectional vLLM plugin) behind the card's server (github.com/StartLuxLabs/StartLux-Decision at a99977d): ~/envs/startlux/bin/startlux-decide serve --model ~/decision-models/StartLux-Decision-v1.1-4B-Instant@416088f77cfce09633906a3675c7cd2e5dabcf10 --launch-vllm --gpus 0 --host 0.0.0.0 --port 19410 (env ~/envs/startlux: vllm==0.31.0, torch 2.13.0+cu130, the repository at a99977d with [vllm]; vLLM 0.31.0 because 0.19.1, the version of the card's published scores, has no aarch64 build; the plugin is validated on both); vision on (the server's default); bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'startlux' (System One at /v1/systemone, model ignored)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "startlux-models/StartLux-Decision-v1.1-4B-Instant",
    "--provider",
    "startlux",
    "--timeout",
    "60",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "6ad0eac51a4abb950bde74b24687209aec628aa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "startlux-models/StartLux-Decision-v1.1-4B-Instant",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "startlux",
  "questions": 1125,
  "request_seconds": 85.45292209362378,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "0c0e9f6d8f2b82ea8d03ccaa356055d8e6bd7ac3851eb96b97d0a5da3f63494c",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 394757,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19410/v1/systemone
{
 "model": "startlux-models/StartLux-Decision-v1.1-4B-Instant",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · startlux-4b_en_dontknow

Name
StartLux-Decision-v1.1-4B-Instant
Setup
StartLux-Decision v1.1 4B · decision model · with the I don't know option F
Model
startlux-models/StartLux-Decision-v1.1-4B-Instant · provider startlux
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.00538 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0253 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.005 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run startlux-4b_en_dontknow · first answer 2026-11-13 16:04:23 UTC · last answer 2026-11-13 16:05:54 UTC · span 91.0 s, pauses included (not charged) · charged 91.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0054 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 4B dense · source https://huggingface.co/startlux-models/StartLux-Decision-v1.1-4B-Instant · read 2026-11-10 · the card states 4B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 4.66B (counted from the checkpoint (4,659,865,088: language model incl. MTP layer 4,326,350,848, vision tower 333,514,240); from Qwen3.5-4B; the card: 4B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
13 Nov 2026 16:04 UTC; last answer 13 Nov 2026 16:05 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup StartLux-Decision v1.1 4B (site run None); runner df9ae2734; startlux-models/StartLux-Decision-v1.1-4B-Instant@416088f77cfce09633906a3675c7cd2e5dabcf10 (weights CC BY-NC 4.0, non-commercial: run for this non-commercial benchmark; code Apache-2.0; Qwen3.5-4B, prefix-bidirectional vLLM plugin) behind the card's server (github.com/StartLuxLabs/StartLux-Decision at a99977d): ~/envs/startlux/bin/startlux-decide serve --model ~/decision-models/StartLux-Decision-v1.1-4B-Instant@416088f77cfce09633906a3675c7cd2e5dabcf10 --launch-vllm --gpus 0 --host 0.0.0.0 --port 19410 (env ~/envs/startlux: vllm==0.31.0, torch 2.13.0+cu130, the repository at a99977d with [vllm]; vLLM 0.31.0 because 0.19.1, the version of the card's published scores, has no aarch64 build; the plugin is validated on both); vision on (the server's default); bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'startlux' (System One at /v1/systemone, model ignored)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "startlux-models/StartLux-Decision-v1.1-4B-Instant",
    "--provider",
    "startlux",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "6ad0eac51a4abb950bde74b24687209aec628aa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "startlux-models/StartLux-Decision-v1.1-4B-Instant",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "startlux",
  "questions": 1125,
  "request_seconds": 88.0209794419934,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "835f6f9a283286e0299a4df7bf011a8d09d135ec76117d62ab873a826477d181",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 426257,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19410/v1/systemone
{
 "model": "startlux-models/StartLux-Decision-v1.1-4B-Instant",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · cerno-9b_en

Name
Cerno 9B v1.2
Setup
Cerno 9B v1.2 · decision model
Model
apertixai/cerno-9b-v1.2 · provider cerno
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.01502 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0706 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.015 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run cerno-9b_en · first answer 2026-11-13 16:31:55 UTC · last answer 2026-11-13 16:36:09 UTC · span 254.0 s, pauses included (not charged) · charged 254.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0150 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/apertixai/cerno-9b-v1.2 · read 2026-11-10 · the card states 9B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 8.95B (counted from the checkpoint (8,953,803,264: backbone 7,936,684,544 + untied lm_head 1,017,118,720; text only); a full fine-tune of Qwen3.5-9B; the card: 9B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
13 Nov 2026 16:31 UTC; last answer 13 Nov 2026 16:36 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Cerno 9B v1.2 (site run None); runner df9ae2734; apertixai/cerno-9b-v1.2@f02a632fbe3a4698cd1ff5c0b929ea240e2b8eda (Apache-2.0; Qwen3.5-9B full fine-tune, block attention, read twice up to 3,000 tokens, temperature 1.256 from decider_config.json); served by our thin wrapper around the card's CernoEngine (decision/servers/cerno_server.py: request/response mapping only; vendored decider 1c15a48, decision-index kit 9eb2dbe): PYTHONPATH=~/decision-models/cerno-9b-v1.2@f02a632fbe3a4698cd1ff5c0b929ea240e2b8eda/inference HF_HUB_OFFLINE=1 ~/envs/cerno/bin/python ~/decision-src/cerno_server.py --model ~/decision-models/cerno-9b-v1.2@f02a632fbe3a4698cd1ff5c0b929ea240e2b8eda --name apertixai/cerno-9b-v1.2 --host 0.0.0.0 --port 19390 (env ~/envs/cerno exactly as the card's inference/requirements-v1.2.txt: torch 2.14.1+cu130, transformers 5.17.0, flash-linear-attention 0.5.2, decision-index kit 9eb2dbe; usage.input_tokens is the engine's own count); text only; bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'cerno'; probabilities rounded to 4 places
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "apertixai/cerno-9b-v1.2",
    "--provider",
    "cerno",
    "--timeout",
    "120",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "6ad0eac51a4abb950bde74b24687209aec628aa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "apertixai/cerno-9b-v1.2",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "cerno",
  "questions": 1125,
  "request_seconds": 249.62783049052814,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "ba7527bb3cc1eb0d99ffd5b0f4f2ebca9fa9b0efa9b36721d875d82235c2df35",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 715454,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19390/v1/systemone
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · cerno-9b_en_dontknow

Name
Cerno 9B v1.2
Setup
Cerno 9B v1.2 · decision model · with the I don't know option F
Model
apertixai/cerno-9b-v1.2 · provider cerno
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.01632 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0767 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.016 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run cerno-9b_en_dontknow · first answer 2026-11-13 16:42:11 UTC · last answer 2026-11-13 16:46:47 UTC · span 276.0 s, pauses included (not charged) · charged 276.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0163 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/apertixai/cerno-9b-v1.2 · read 2026-11-10 · the card states 9B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 8.95B (counted from the checkpoint (8,953,803,264: backbone 7,936,684,544 + untied lm_head 1,017,118,720; text only); a full fine-tune of Qwen3.5-9B; the card: 9B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
13 Nov 2026 16:42 UTC; last answer 13 Nov 2026 16:46 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Cerno 9B v1.2 (site run None); runner df9ae2734; apertixai/cerno-9b-v1.2@f02a632fbe3a4698cd1ff5c0b929ea240e2b8eda (Apache-2.0; Qwen3.5-9B full fine-tune, block attention, read twice up to 3,000 tokens, temperature 1.256 from decider_config.json); served by our thin wrapper around the card's CernoEngine (decision/servers/cerno_server.py: request/response mapping only; vendored decider 1c15a48, decision-index kit 9eb2dbe): PYTHONPATH=~/decision-models/cerno-9b-v1.2@f02a632fbe3a4698cd1ff5c0b929ea240e2b8eda/inference HF_HUB_OFFLINE=1 ~/envs/cerno/bin/python ~/decision-src/cerno_server.py --model ~/decision-models/cerno-9b-v1.2@f02a632fbe3a4698cd1ff5c0b929ea240e2b8eda --name apertixai/cerno-9b-v1.2 --host 0.0.0.0 --port 19390 (env ~/envs/cerno exactly as the card's inference/requirements-v1.2.txt: torch 2.14.1+cu130, transformers 5.17.0, flash-linear-attention 0.5.2, decision-index kit 9eb2dbe; usage.input_tokens is the engine's own count); text only; bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'cerno'; probabilities rounded to 4 places
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "apertixai/cerno-9b-v1.2",
    "--provider",
    "cerno",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "6ad0eac51a4abb950bde74b24687209aec628aa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "apertixai/cerno-9b-v1.2",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "cerno",
  "questions": 1125,
  "request_seconds": 263.9490013771574,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "71c9af70158a04d39f8eed1c1d602ea3e01d89ea2b493e42d78cfa0f43750ae2",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 780470,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19390/v1/systemone
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · lfm3-1.7b-vllm_en

Name
LFM3-1.7B on · chat
Setup
LFM3-1.7B (chat, thinking, on our GPU) · chat model
Model
lfm3-1.7b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.127 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.453 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.127. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run lfm3-1.7b-vllm_en · first answer 2026-11-13 16:47:10 UTC · last answer 2026-11-13 17:14:20 UTC · span 1630.0 s, pauses included (not charged) · charged 1630.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.1274 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.3 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 10 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 1.73B dense · from the run's notes (dense; BF16 safetensors 1,728,801,536, text only; card: 1.7B); also https://huggingface.co/LiquidAI/LFM3-1.7B-GGUF · read 2026-11-09 · the released checkpoint's tensors without registered buffers: 1,728,801,536 (the served BF16 GGUF also stores a 64-element RoPE buffer, 1,728,801,600); text only; card: 1.7B
Input
text, English
Endpoint
http://127.0.0.1:18634/v1/chat/completions
Started
13 Nov 2026 16:46 UTC; last answer 13 Nov 2026 17:14 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
LFM3-1.7B, served with vLLM 0.34.0rc1.dev302+gbe47189f3 (nightly, commit be47189f37f6, CUDA 13.5 image; docker vllm/vllm-openai@sha256:7f0618da95bac2f1cc97fcd1791380d7c1a06d24a4998da45aeb7fbe6c49179e), BF16 from LiquidAI/LFM3-1.7B@0dc4268b561499c648c7ba7c9466bf3d8d01517a, vLLM reasoning parser qwen3 (vLLM nightly because no vLLM release has Lfm3ForCausalLM; a clean rerun of the standard runs entirely on this engine with the dont_know runs' server and settings (Umit #6250), replacing lfm3-1.7b_en, whose answers mix llama.cpp b12020 and vLLM; the same weights as the llama.cpp setup's official BF16 GGUF (tensors byte-identical); qualification work/lfm3_vllm_qual.md), one RTX PRO 6000 on our GPU host alone; thinking on (enable_thinking true, the template default; on/off only); sampling temperature 0.1, top_k 50, repetition_penalty 1.1, top_p 1, min_p 0, from generation_config.json, passed 1:1 as vLLM sampling params; seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 1.73B / 1.73B active / total (dense; BF16 safetensors 1,728,801,536, text only; card: 1.7B). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "repetition_penalty": 1.1,
  "seed": 0,
  "temperature": 0.1,
  "top_k": 50,
  "top_p": 1.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · lfm3-1.7b-vllm-nothink_en_dontknow

Name
LFM3-1.7B off · chat
Setup
LFM3-1.7B (chat, thinking off, on our GPU) · chat model · with the I don't know option F
Model
lfm3-1.7b-vllm-nothink · provider vllm
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.5002 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 2.35 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.500 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run lfm3-1.7b-vllm-nothink_en_dontknow · first answer 2026-11-13 18:03:28 UTC · last answer 2026-11-13 20:24:29 UTC · span 8461.0 s, pauses included (not charged) · charged 8461.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.5002 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 31.2 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 3 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 1.73B dense · from the run's notes (dense; BF16 safetensors 1,728,801,536, text only; card: 1.7B); also https://huggingface.co/LiquidAI/LFM3-1.7B-GGUF · read 2026-11-09 · the released checkpoint's tensors without registered buffers: 1,728,801,536 (the served BF16 GGUF also stores a 64-element RoPE buffer, 1,728,801,600); text only; card: 1.7B
Input
text, English
Endpoint
http://lan-host:18626/v1/chat/completions
Started
13 Nov 2026 18:02 UTC; last answer 13 Nov 2026 20:24 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
LFM3-1.7B, served with vLLM 0.34.0rc1.dev302+gbe47189f3 (nightly, commit be47189f37f6, CUDA 13.5 image; docker vllm/vllm-openai@sha256:7f0618da95bac2f1cc97fcd1791380d7c1a06d24a4998da45aeb7fbe6c49179e, linux/arm64 build sha256:c41b69fa5f34…), BF16 from LiquidAI/LFM3-1.7B@0dc4268b561499c648c7ba7c9466bf3d8d01517a, vLLM reasoning parser qwen3 (thinking off via chat_template_kwargs enable_thinking false, the template's own switch (the assistant turn opens with an empty <think></think>), not a prompt instruction; otherwise the settings of the thinking-on vLLM runs lfm3-1.7b_en (Umit #6101); vLLM nightly because no vLLM release has Lfm3ForCausalLM; qualification work/lfm3_vllm_qual.md), one DGX Spark (a shared Spark host, NVIDIA GB10, 121 GB unified memory) alone; thinking off (chat_template_kwargs enable_thinking false; the template default true is overridden); sampling temperature 0.1, top_k 50, repetition_penalty 1.1, top_p 1, min_p 0, from generation_config.json, passed 1:1 as vLLM sampling params; seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 1.73B / 1.73B active / total (dense; BF16 safetensors 1,728,801,536, text only; card: 1.7B). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "repetition_penalty": 1.1,
  "seed": 0,
  "temperature": 0.1,
  "top_k": 50,
  "top_p": 1.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · minicpm5-2b-nothink_en

Name
MiniCPM5-2B off · chat
Setup
MiniCPM5-2B (chat, thinking off, on our GPU) · chat model
Model
minicpm5-2b-nothink · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.594 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.64 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.594. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run minicpm5-2b-nothink_en · first answer 2026-11-13 18:14:36 UTC · last answer 2026-11-13 18:53:01 UTC · span 2305.0 s, pauses included (not charged) · charged 2305.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.5943 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 28.6 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 2 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 2.52B dense · from the run's notes (dense; checkpoint 2,516,756,480 incl. untied lm_head, text only; card: 2,516,756,480); also https://huggingface.co/openbmb/MiniCPM5-2B · read 2026-11-12 · counted from the checkpoint (2,516,756,480 incl. untied lm_head, text only; dense); the card: 2,516,756,480
Input
text, English
Endpoint
http://127.0.0.1:18615/v1/chat/completions
Started
13 Nov 2026 18:14 UTC; last answer 13 Nov 2026 18:53 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
MiniCPM5-2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from openbmb/MiniCPM5-2B@42433e90dad95123ab87729fd7011e7110629556, vLLM reasoning parser qwen3, one RTX PRO 6000 on our GPU host alone; thinking off (enable_thinking false, the template's own switch); sampling temperature 1, top_p 0.95, from generation_config.json; seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 2.52B / 2.52B active / total (dense; checkpoint 2,516,756,480 incl. untied lm_head, text only; card: 2,516,756,480). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "max_tokens": 32768,
  "seed": 0,
  "temperature": 1.0,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · minicpm5-2b-nothink_en_dontknow

Name
MiniCPM5-2B off · chat
Setup
MiniCPM5-2B (chat, thinking off, on our GPU) · chat model · with the I don't know option F
Model
minicpm5-2b-nothink · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.107 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.445 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.107. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run minicpm5-2b-nothink_en_dontknow · first answer 2026-11-13 19:14:18 UTC · last answer 2026-11-13 19:40:59 UTC · span 1601.0 s, pauses included (not charged) · charged 1601.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.1074 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.1 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 2 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 2.52B dense · from the run's notes (dense; checkpoint 2,516,756,480 incl. untied lm_head, text only; card: 2,516,756,480); also https://huggingface.co/openbmb/MiniCPM5-2B · read 2026-11-12 · counted from the checkpoint (2,516,756,480 incl. untied lm_head, text only; dense); the card: 2,516,756,480
Input
text, English
Endpoint
http://127.0.0.1:18615/v1/chat/completions
Started
13 Nov 2026 19:14 UTC; last answer 13 Nov 2026 19:40 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
MiniCPM5-2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from openbmb/MiniCPM5-2B@42433e90dad95123ab87729fd7011e7110629556, vLLM reasoning parser qwen3, one RTX PRO 6000 on our GPU host alone; thinking off (enable_thinking false, the template's own switch); sampling temperature 1, top_p 0.95, from generation_config.json; seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 2.52B / 2.52B active / total (dense; checkpoint 2,516,756,480 incl. untied lm_head, text only; card: 2,516,756,480). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "max_tokens": 32768,
  "seed": 0,
  "temperature": 1.0,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · gpt-6.1-terra-sub_en_dontknow

Name
GPT-6.1 Terra chat
Setup
GPT-6.1 Terra · chat model · with the I don't know option F
Model
gpt-6.1-terra · provider chatgpt-subscription
GPU
—
Cost
$0 for the run · ChatGPT subscription
Answer time
run gpt-6.1-terra-sub_en_dontknow · first answer 2026-11-13 21:37:42 UTC · last answer 2026-11-13 23:24:42 UTC · span 6420.0 s, pauses included (not charged) · charged 6420.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not on record yet
Input
text, English
Endpoint
codex-chatgpt-subscription
Started
13 Nov 2026 21:37 UTC; last answer 13 Nov 2026 23:24 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
GPT-6.1 Terra on the ChatGPT subscription through the Codex CLI (`codex exec`, ChatGPT sign-in from OMP's openai-codex login in a private CODEX_HOME, no refresh token, since 13 Nov 19:35 UTC; as turkce's ask_openai), model gpt-6.1-terra, model_reasoning_effort high (Reading's setting); no output limit sent (Codex has no setting for it) and no temperature; Codex's agent instructions replaced by a neutral one-line system prompt, web search and tool features off, read-only sandbox, no user config, rules or saved session, API-key variables removed; one question per request, one request at a time; a usage-limit reply stops the run without retrying it (resumable); cost $0 (ChatGPT subscription); api_equivalent_usd is the tokens at OpenAI's API list price {"input": 1.0, "cached_input": 0.05, "cache_write": 1.25, "output": 5.0} per 1M tokens, not a cost. Codex reports no served model id. English translations.
Parameters
{
  "disabled_features": [
    "shell_tool",
    "unified_exec",
    "goals",
    "apps",
    "plugins",
    "remote_plugin",
    "tool_suggest",
    "skill_search",
    "view_image",
    "image_generation",
    "sleep_tool",
    "browser_use",
    "browser_use_external",
    "computer_use",
    "in_app_browser",
    "multi_agent",
    "unbounded_connection_retries"
  ],
  "effort": "high",
  "list_price_per_mtok": {
    "cache_write": 1.25,
    "cached_input": 0.05,
    "input": 1.0,
    "output": 5.0
  },
  "max_output_tokens": null,
  "route": "codex exec (ChatGPT sign-in)",
  "sandbox": "read-only",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message.",
  "web_search": "disabled"
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · gemini-4-hydrogen-sub_en

Name
Gemini 4 Hydrogen chat
Setup
Gemini 4 Hydrogen (Antigravity subscription) · chat model
Model
gemini-4-hydrogen · provider Antigravity (Gemini subscription)
GPU
—
Cost
not measured
Answer time
run gemini-4-hydrogen-sub_en · first answer 2026-11-14 00:55:51 UTC · last answer 2026-11-14 06:25:06 UTC · span 19755.0 s, pauses included (not charged) · charged 15176.0 s in 5 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.3 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 2 empty replies asked once more
Model size
Parameters: not on record yet
Input
text, English
Endpoint
http://lan-host:8899/v1/chat/completions
Started
14 Nov 2026 00:48 UTC; last answer 14 Nov 2026 06:25 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
[Turkish wording, not shown]
Parameters
{
  "max_tokens": 32768,
  "temperature": 1.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · lfm3-1.7b-vllm-seed1_en

Name
LFM3-1.7B on · chat
Setup
LFM3-1.7B (chat, thinking, on our GPU) · chat model
Model
lfm3-1.7b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.144 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.459 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.144. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run lfm3-1.7b-vllm-seed1_en · first answer 2026-11-14 02:53:44 UTC · last answer 2026-11-14 03:21:18 UTC · span 1654.0 s, pauses included (not charged) · charged 1654.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.1440 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.3 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 8 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 1.73B dense · from the run's notes (dense; BF16 safetensors 1,728,801,536, text only; card: 1.7B); also https://huggingface.co/LiquidAI/LFM3-1.7B-GGUF · read 2026-11-09 · the released checkpoint's tensors without registered buffers: 1,728,801,536 (the served BF16 GGUF also stores a 64-element RoPE buffer, 1,728,801,600); text only; card: 1.7B
Input
text, English
Endpoint
http://127.0.0.1:18634/v1/chat/completions
Started
14 Nov 2026 02:53 UTC; last answer 14 Nov 2026 03:21 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
LFM3-1.7B, served with vLLM 0.34.0rc1.dev302+gbe47189f3 (nightly, commit be47189f37f6, CUDA 13.5 image; docker vllm/vllm-openai@sha256:7f0618da95bac2f1cc97fcd1791380d7c1a06d24a4998da45aeb7fbe6c49179e), BF16 from LiquidAI/LFM3-1.7B@0dc4268b561499c648c7ba7c9466bf3d8d01517a, vLLM reasoning parser qwen3 (seed 1 of three (Umit via zeki #6969: seeds 0, 1, 2; seed 0 is lfm3-1.7b-vllm_en); vLLM nightly because no vLLM release has Lfm3ForCausalLM; the same server and settings as seed 0), one RTX PRO 6000 on our GPU host alone; thinking on (enable_thinking true, the template default; on/off only); sampling temperature 0.1, top_k 50, repetition_penalty 1.1, top_p 1, min_p 0, from generation_config.json, passed 1:1 as vLLM sampling params; seed 1; max_tokens 32,768; one question per request (32 requests in flight), standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 1.73B / 1.73B active / total (dense; BF16 safetensors 1,728,801,536, text only; card: 1.7B). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "repetition_penalty": 1.1,
  "seed": 1,
  "temperature": 0.1,
  "top_k": 50,
  "top_p": 1.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · lfm3-1.7b-vllm-seed2_en

Name
LFM3-1.7B on · chat
Setup
LFM3-1.7B (chat, thinking, on our GPU) · chat model
Model
lfm3-1.7b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.162 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.467 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.162. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run lfm3-1.7b-vllm-seed2_en · first answer 2026-11-14 03:34:13 UTC · last answer 2026-11-14 04:02:13 UTC · span 1680.0 s, pauses included (not charged) · charged 1680.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.1620 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.2 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 13 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 1.73B dense · from the run's notes (dense; BF16 safetensors 1,728,801,536, text only; card: 1.7B); also https://huggingface.co/LiquidAI/LFM3-1.7B-GGUF · read 2026-11-09 · the released checkpoint's tensors without registered buffers: 1,728,801,536 (the served BF16 GGUF also stores a 64-element RoPE buffer, 1,728,801,600); text only; card: 1.7B
Input
text, English
Endpoint
http://127.0.0.1:18634/v1/chat/completions
Started
14 Nov 2026 03:34 UTC; last answer 14 Nov 2026 04:02 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
LFM3-1.7B, served with vLLM 0.34.0rc1.dev302+gbe47189f3 (nightly, commit be47189f37f6, CUDA 13.5 image; docker vllm/vllm-openai@sha256:7f0618da95bac2f1cc97fcd1791380d7c1a06d24a4998da45aeb7fbe6c49179e), BF16 from LiquidAI/LFM3-1.7B@0dc4268b561499c648c7ba7c9466bf3d8d01517a, vLLM reasoning parser qwen3 (seed 2 of three (Umit via zeki #6969: seeds 0, 1, 2; seed 0 is lfm3-1.7b-vllm_en); vLLM nightly because no vLLM release has Lfm3ForCausalLM; the same server and settings as seed 0), one RTX PRO 6000 on our GPU host alone; thinking on (enable_thinking true, the template default; on/off only); sampling temperature 0.1, top_k 50, repetition_penalty 1.1, top_p 1, min_p 0, from generation_config.json, passed 1:1 as vLLM sampling params; seed 2; max_tokens 32,768; one question per request (32 requests in flight), standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 1.73B / 1.73B active / total (dense; BF16 safetensors 1,728,801,536, text only; card: 1.7B). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "repetition_penalty": 1.1,
  "seed": 2,
  "temperature": 0.1,
  "top_k": 50,
  "top_p": 1.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · gpt-6-astra-sub_en

Name
GPT-6 Astra chat
Setup
GPT-6 Astra · chat model
Model
gpt-6-astra · provider chatgpt-subscription
GPU
—
Cost
$0 for the run · ChatGPT subscription
Answer time
run gpt-6-astra-sub_en · first answer 2026-11-14 04:32:53 UTC · last answer 2026-11-14 07:17:32 UTC · span 9879.0 s, pauses included (not charged) · charged 9879.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not on record yet
Input
text, English
Endpoint
codex-chatgpt-subscription
Started
14 Nov 2026 04:32 UTC; last answer 14 Nov 2026 07:17 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
GPT-6 Astra on the ChatGPT subscription through the Codex CLI (`codex exec`, ChatGPT sign-in from OMP's openai-codex login in a private CODEX_HOME, no refresh token, since 13 Nov 19:35 UTC; as turkce's ask_openai), model gpt-6-astra, model_reasoning_effort high (Reading's setting); no output limit sent (Codex has no setting for it) and no temperature; Codex's agent instructions replaced by a neutral one-line system prompt, web search and tool features off, read-only sandbox, no user config, rules or saved session, API-key variables removed; one question per request, one request at a time; a usage-limit reply stops the run without retrying it (resumable); cost $0 (ChatGPT subscription); api_equivalent_usd is the tokens at OpenAI's API list price {"input": 10.0, "cached_input": 1.0, "cache_write": 12.5, "output": 50.0} per 1M tokens, not a cost. Codex reports no served model id. English translations.
Parameters
{
  "disabled_features": [
    "shell_tool",
    "unified_exec",
    "goals",
    "apps",
    "plugins",
    "remote_plugin",
    "tool_suggest",
    "skill_search",
    "view_image",
    "image_generation",
    "sleep_tool",
    "browser_use",
    "browser_use_external",
    "computer_use",
    "in_app_browser",
    "multi_agent",
    "unbounded_connection_retries"
  ],
  "effort": "high",
  "list_price_per_mtok": {
    "cache_write": 12.5,
    "cached_input": 1.0,
    "input": 10.0,
    "output": 50.0
  },
  "max_output_tokens": null,
  "route": "codex exec (ChatGPT sign-in)",
  "sandbox": "read-only",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message.",
  "web_search": "disabled"
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · minicpm5-2b-seed1_en

Name
MiniCPM5-2B on · chat
Setup
MiniCPM5-2B (chat, thinking, on our GPU) · chat model
Model
minicpm5-2b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$3.768 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 1.51 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $3.768. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run minicpm5-2b-seed1_en · first answer 2026-11-14 05:08:20 UTC · last answer 2026-11-14 06:39:07 UTC · span 5447.0 s, pauses included (not charged) · charged 5447.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $3.7675 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.3 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 59 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 2.52B dense · from the run's notes (dense; checkpoint 2,516,756,480 incl. untied lm_head, text only; card: 2,516,756,480); also https://huggingface.co/openbmb/MiniCPM5-2B · read 2026-11-12 · counted from the checkpoint (2,516,756,480 incl. untied lm_head, text only; dense); the card: 2,516,756,480
Input
text, English
Endpoint
http://127.0.0.1:18614/v1/chat/completions
Started
14 Nov 2026 05:08 UTC; last answer 14 Nov 2026 06:39 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
MiniCPM5-2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from openbmb/MiniCPM5-2B@42433e90dad95123ab87729fd7011e7110629556, vLLM reasoning parser qwen3 (seed 1 of three (Umit via zeki #6979: seeds 0, 1, 2; seed 0 is minicpm5-2b_en); the same checkpoint, engine, prompt and sampling as seed 0; baseline server capacity not retained; repeats use max-num-seqs 16 (zeki #7001)), one RTX PRO 6000 on our GPU host alone; thinking on (enable_thinking true; on/off only, no levels); sampling temperature 1, top_p 0.95, from generation_config.json; seed 1; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 2.52B / 2.52B active / total (dense; checkpoint 2,516,756,480 incl. untied lm_head, text only; card: 2,516,756,480). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "seed": 1,
  "temperature": 1.0,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · gemini-4-hydrogen-sub_en_dontknow

Name
Gemini 4 Hydrogen chat
Setup
Gemini 4 Hydrogen (Antigravity subscription) · chat model · with the I don't know option F
Model
gemini-4-hydrogen · provider Antigravity (Gemini subscription)
GPU
—
Cost
not measured
Answer time
run gemini-4-hydrogen-sub_en_dontknow · first answer 2026-11-14 06:27:55 UTC · last answer 2026-11-14 10:40:21 UTC · span 15146.0 s, pauses included (not charged) · charged 15146.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: not on record yet
Input
text, English
Endpoint
http://lan-host:8899/v1/chat/completions
Started
14 Nov 2026 06:27 UTC; last answer 14 Nov 2026 10:40 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
[Turkish wording, not shown]
Parameters
{
  "max_tokens": 32768,
  "temperature": 1.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · minicpm5-2b-seed2_en

Name
MiniCPM5-2B on · chat
Setup
MiniCPM5-2B (chat, thinking, on our GPU) · chat model
Model
minicpm5-2b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$3.658 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 1.47 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $3.658. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run minicpm5-2b-seed2_en · first answer 2026-11-14 07:05:53 UTC · last answer 2026-11-14 08:34:02 UTC · span 5289.0 s, pauses included (not charged) · charged 5289.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $3.6582 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.2 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 48 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 2.52B dense · from the run's notes (dense; checkpoint 2,516,756,480 incl. untied lm_head, text only; card: 2,516,756,480); also https://huggingface.co/openbmb/MiniCPM5-2B · read 2026-11-12 · counted from the checkpoint (2,516,756,480 incl. untied lm_head, text only; dense); the card: 2,516,756,480
Input
text, English
Endpoint
http://127.0.0.1:18614/v1/chat/completions
Started
14 Nov 2026 07:05 UTC; last answer 14 Nov 2026 08:34 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
MiniCPM5-2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from openbmb/MiniCPM5-2B@42433e90dad95123ab87729fd7011e7110629556, vLLM reasoning parser qwen3 (seed 2 of three (Umit via zeki #6979: seeds 0, 1, 2; seed 0 is minicpm5-2b_en); the same checkpoint, engine, prompt and sampling as seed 0; baseline server capacity not retained; repeats use max-num-seqs 16 (zeki #7001)), one RTX PRO 6000 on our GPU host alone; thinking on (enable_thinking true; on/off only, no levels); sampling temperature 1, top_p 0.95, from generation_config.json; seed 2; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 2.52B / 2.52B active / total (dense; checkpoint 2,516,756,480 incl. untied lm_head, text only; card: 2,516,756,480). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "seed": 2,
  "temperature": 1.0,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · gpt-6-astra-sub_en_dontknow

Name
GPT-6 Astra chat
Setup
GPT-6 Astra · chat model · with the I don't know option F
Model
gpt-6-astra · provider chatgpt-subscription
GPU
—
Cost
$0 for the run · ChatGPT subscription
Answer time
run gpt-6-astra-sub_en_dontknow · first answer 2026-11-14 07:17:39 UTC · last answer 2026-11-14 09:58:46 UTC · span 9667.0 s, pauses included (not charged) · charged 9667.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not on record yet
Input
text, English
Endpoint
codex-chatgpt-subscription
Started
14 Nov 2026 07:17 UTC; last answer 14 Nov 2026 09:58 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
GPT-6 Astra on the ChatGPT subscription through the Codex CLI (`codex exec`, ChatGPT sign-in from OMP's openai-codex login in a private CODEX_HOME, no refresh token, since 13 Nov 19:35 UTC; as turkce's ask_openai), model gpt-6-astra, model_reasoning_effort high (Reading's setting); no output limit sent (Codex has no setting for it) and no temperature; Codex's agent instructions replaced by a neutral one-line system prompt, web search and tool features off, read-only sandbox, no user config, rules or saved session, API-key variables removed; one question per request, one request at a time; a usage-limit reply stops the run without retrying it (resumable); cost $0 (ChatGPT subscription); api_equivalent_usd is the tokens at OpenAI's API list price {"input": 10.0, "cached_input": 1.0, "cache_write": 12.5, "output": 50.0} per 1M tokens, not a cost. Codex reports no served model id. English translations.
Parameters
{
  "disabled_features": [
    "shell_tool",
    "unified_exec",
    "goals",
    "apps",
    "plugins",
    "remote_plugin",
    "tool_suggest",
    "skill_search",
    "view_image",
    "image_generation",
    "sleep_tool",
    "browser_use",
    "browser_use_external",
    "computer_use",
    "in_app_browser",
    "multi_agent",
    "unbounded_connection_retries"
  ],
  "effort": "high",
  "list_price_per_mtok": {
    "cache_write": 12.5,
    "cached_input": 1.0,
    "input": 10.0,
    "output": 50.0
  },
  "max_output_tokens": null,
  "route": "codex exec (ChatGPT sign-in)",
  "sandbox": "read-only",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message.",
  "web_search": "disabled"
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · claude-haiku-5.5-sub_en_dontknow

Name
Claude Haiku 5.5 chat
Setup
Claude Haiku 5.5 · chat model · with the I don't know option F
Model
claude-haiku-5-5 · provider claude-subscription
GPU
—
Cost
$0 for the run · Claude subscription
Answer time
run claude-haiku-5.5-sub_en_dontknow · first answer 2026-11-14 10:41:05 UTC · last answer 2026-11-14 12:51:23 UTC · span 7818.0 s, pauses included (not charged) · charged 7818.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker (closed weights) · source https://openrouter.ai/anthropic/claude-haiku-5.5 · read 2026-11-10
Input
text, English
Endpoint
claude-code-subscription
Started
14 Nov 2026 10:40 UTC; last answer 14 Nov 2026 12:51 UTC
Progress
100 answered of 1,125 asked; 0 request errors; in progress
Notes
Claude Haiku 5.5 on the Claude subscription through the Claude Code CLI (`claude --print`, as turkce's runs 968/971), model claude-haiku-5-5, --effort max (the highest of low/medium/high/xhigh/max; adaptive thinking always on); CLAUDE_CODE_MAX_OUTPUT_TOKENS 32768; no temperature sent (Anthropic's default 1.0); Claude Code's agent prompt replaced by a neutral one-line system prompt, tools, settings, MCP servers and session saving off, API-key variables removed; one question per request, one request at a time; cost $0 (Claude subscription); api_equivalent_usd is Claude Code's API-list-price estimate, not a cost. English translations.
Parameters
{
  "effort": "max",
  "max_output_tokens": 32768,
  "route": "claude-code-cli --print",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message."
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · gemini-4-carbon-sub_en_dontknow

Name
Gemini 4 Carbon chat
Setup
Gemini 4 Carbon (Antigravity subscription) · chat model · with the I don't know option F
Model
gemini-4-carbon · provider Antigravity (Gemini subscription)
GPU
—
Cost
not measured
Answer time
run gemini-4-carbon-sub_en_dontknow · first answer 2026-11-14 12:57:27 UTC · last answer 2026-11-15 03:53:04 UTC · span 53737.0 s, pauses included (not charged) · charged 50068.0 s in 4 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 2.0 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 14 empty replies asked once more
Model size
Parameters: not disclosed by maker (closed weights) · source https://ai.google.dev/gemini-api/docs/models · read 2026-11-11
Input
text, English
Endpoint
http://lan-host:8899/v1/chat/completions
Started
14 Nov 2026 12:56 UTC; last answer 15 Nov 2026 03:53 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Gemini 4 Carbon (gemini-4-carbon = gemini-4-carbon-high) through Can's Antigravity subscription (antigravity-benchmark #4762; Umit via zeki #5133): tools denied, ~10.7k tokens of Antigravity's own context per request, one request at a time; temperature 1.0 and max_tokens 32,768 sent but ignored by Antigravity (native sampling, output cap 262,144). A separate setup from the paid API run gemini-4-carbon_en. Bridge address changed 14 Nov 17:44 UTC from lan-host to lan-host (the Mac moved to its wired adapter after a network change; same Antigravity (Gemini subscription) service, same settings; antigravity-benchmark #7808); answers after that came over the new address.
Parameters
{
  "max_tokens": 32768,
  "temperature": 1.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · decider-flash-v1.3_en

Name
Perplexity Decider Flash v1.3
Setup
Perplexity Decider Flash v1.3 · decision model
Model
pplx-decider-flash-v1.3 · provider pplx-decider
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.02974 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0119 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.030. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run decider-flash-v1.3_en · first answer 2026-11-14 15:48:40 UTC · last answer 2026-11-14 15:49:23 UTC · span 43.0 s, pauses included (not charged) · charged 43.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0297 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 4B dense · source https://huggingface.co/perplexity-ai/pplx-decider-flash-v1.3 · read 2026-11-14 · 4B as v1.2 (built on Qwen3.5-4B); counted from the released files: 4.54B for decisions (backbone + readout 4,539,918,336, the same as v1.2); the Locate adapter and head (131,508,901) are not used for decisions
Input
text, English
Endpoint
http://127.0.0.1:19185
Started
14 Nov 2026 15:48 UTC; last answer 14 Nov 2026 15:49 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Perplexity Decider Flash v1.3 (site run None); runner df9ae2734; published on a DGX Spark (GB10); gpu-lanes runs it on an RTX PRO 6000
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "pplx-decider-flash-v1.3",
    "--provider",
    "pplx-decider",
    "--timeout",
    "30",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "6ad0eac51a4abb950bde74b24687209aec628aa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "pplx-decider-flash-v1.3",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "pplx-decider",
  "questions": 1125,
  "request_seconds": 38.4861737200099,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "a251fc598df3035a8eea9b659c0a5a7e9a3969d731603973429cdc40c8769529",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 404882,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19185/v1/systemone
{
 "model": "pplx-decider-flash-v1.3",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · decider-flash-v1.3_en_dontknow

Name
Perplexity Decider Flash v1.3
Setup
Perplexity Decider Flash v1.3 · decision model · with the I don't know option F
Model
pplx-decider-flash-v1.3 · provider pplx-decider
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.02905 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0117 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.029. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run decider-flash-v1.3_en_dontknow · first answer 2026-11-14 15:50:39 UTC · last answer 2026-11-14 15:51:21 UTC · span 42.0 s, pauses included (not charged) · charged 42.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0291 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 4B dense · source https://huggingface.co/perplexity-ai/pplx-decider-flash-v1.3 · read 2026-11-14 · 4B as v1.2 (built on Qwen3.5-4B); counted from the released files: 4.54B for decisions (backbone + readout 4,539,918,336, the same as v1.2); the Locate adapter and head (131,508,901) are not used for decisions
Input
text, English
Endpoint
http://127.0.0.1:19185
Started
14 Nov 2026 15:50 UTC; last answer 14 Nov 2026 15:51 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Perplexity Decider Flash v1.3 (site run None); runner df9ae2734; published on a DGX Spark (GB10); gpu-lanes runs it on an RTX PRO 6000
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "pplx-decider-flash-v1.3",
    "--provider",
    "pplx-decider",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "6ad0eac51a4abb950bde74b24687209aec628aa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "pplx-decider-flash-v1.3",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "pplx-decider",
  "questions": 1125,
  "request_seconds": 37.79201372103307,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "bd3ed8708dc7e524adc2bef02932c73dd21163a7116155f61ca440ff0cd0e6c2",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 438632,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19185/v1/systemone
{
 "model": "pplx-decider-flash-v1.3",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · decider-v1.3-local_en

Name
Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell
Setup
Perplexity Decider 27B v1.3 (local GPU) · decision model
Model
pplx-decider-v1.3 · provider pplx-decider
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.0913 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0367 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.091. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run decider-v1.3-local_en · first answer 2026-11-14 16:20:35 UTC · last answer 2026-11-14 16:22:47 UTC · span 132.0 s, pauses included (not charged) · charged 132.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0913 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/perplexity-ai/pplx-decider-v1.3 · read 2026-11-14 · the official model name: 27B; counted from the released files: 26.09B for decisions (backbone + readout 26,086,635,760, the same as v1.2); the Locate adapter and head (470,209,189) are not used for decisions
Input
text, English
Endpoint
http://127.0.0.1:19175
Started
14 Nov 2026 16:20 UTC; last answer 14 Nov 2026 16:22 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Perplexity Decider 27B v1.3 (local GPU) (site run None); runner df9ae2734; published on a DGX Spark (GB10); gpu-lanes runs it on an RTX PRO 6000
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "pplx-decider-v1.3",
    "--provider",
    "pplx-decider",
    "--timeout",
    "30",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "6ad0eac51a4abb950bde74b24687209aec628aa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "pplx-decider-v1.3",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "pplx-decider",
  "questions": 1125,
  "request_seconds": 128.29566527798306,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "30fc2e5cc11c7a2e6e1a2d7796abc8bf538134ee959358b003e91ea5b38aa942",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 404882,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19175/v1/systemone
{
 "model": "pplx-decider-v1.3",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · decider-v1.3-local_en_dontknow

Name
Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell
Setup
Perplexity Decider 27B v1.3 (local GPU) · decision model · with the I don't know option F
Model
pplx-decider-v1.3 · provider pplx-decider
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.09545 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0383 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.095. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run decider-v1.3-local_en_dontknow · first answer 2026-11-14 16:26:23 UTC · last answer 2026-11-14 16:28:41 UTC · span 138.0 s, pauses included (not charged) · charged 138.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0955 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/perplexity-ai/pplx-decider-v1.3 · read 2026-11-14 · the official model name: 27B; counted from the released files: 26.09B for decisions (backbone + readout 26,086,635,760, the same as v1.2); the Locate adapter and head (470,209,189) are not used for decisions
Input
text, English
Endpoint
http://127.0.0.1:19175
Started
14 Nov 2026 16:26 UTC; last answer 14 Nov 2026 16:28 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Perplexity Decider 27B v1.3 (local GPU) (site run None); runner df9ae2734; published on a DGX Spark (GB10); gpu-lanes runs it on an RTX PRO 6000
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "pplx-decider-v1.3",
    "--provider",
    "pplx-decider",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "6ad0eac51a4abb950bde74b24687209aec628aa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "pplx-decider-v1.3",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "pplx-decider",
  "questions": 1125,
  "request_seconds": 134.9070179420214,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "2a3849c86a8db70213c3db1738896d260c314d6f650eb4394892299269ff084a",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 438632,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19175/v1/systemone
{
 "model": "pplx-decider-v1.3",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · qwen3.5-4b-nothink_en

Name
Qwen3.5-4B off · chat
Setup
Qwen3.5-4B (chat, thinking off, on our GPU) · chat model
Model
qwen3.5-4b-nothink · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.598 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.642 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.598. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-14 17:15:09–19:55:00 UTC
GPU time
run qwen3.5-4b-nothink_en · first answer 2026-11-14 17:10:51 UTC · last answer 2026-11-14 20:29:12 UTC · span 11901.0 s, pauses included (not charged) · charged 2310.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.5978 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 29.6 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 4.66B dense · from the run's notes (dense; checkpoint 4,659,865,088 incl. vision 0.33B; card: 4B language model); also https://huggingface.co/Qwen/Qwen3.5-4B · read 2026-11-09 · counted from the checkpoint (4,659,865,088: language model 4.33B, vision encoder 0.33B); the card: 4B language model
Input
text, English
Endpoint
http://127.0.0.1:18642/v1/chat/completions
Started
14 Nov 2026 17:10 UTC; last answer 14 Nov 2026 20:29 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Qwen3.5-4B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from Qwen/Qwen3.5-4B@851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a, vLLM reasoning parser qwen3 (vision tower loaded, for Reading's Page images; text-only prompts here), one RTX PRO 6000 on our GPU host alone; thinking off (enable_thinking false; on/off only, no levels); sampling temperature 1, top_p 0.95, top_k 20, min_p 0, presence_penalty 1.5, repetition_penalty 1, sampling from the model card, generation_config has none (non-thinking mode for reasoning tasks; the repo has no generation_config.json); seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 4.66B / 4.66B active / total (dense; checkpoint 4,659,865,088 incl. vision 0.33B; card: 4B language model). English translations (DeepSeek + review). Paused 17:15 UTC 14 Nov so Gemma 4 E4B/E2B thinking on at 32k run first (#7721); resumed from stored answers afterwards (in-flight requests re-asked).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "presence_penalty": 1.5,
  "repetition_penalty": 1.0,
  "seed": 0,
  "temperature": 1.0,
  "top_k": 20,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · gemma-4-e4b-it-think-32k_en

Name
Gemma 4 E4B on · chat · 32k cap
Setup
Gemma 4 E4B (chat, thinking on, 32k, on our GPU) · chat model
Model
gemma-4-e4b-it-think-32k · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.69 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.679 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.690. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gemma-4-e4b-it-think-32k_en · first answer 2026-11-14 17:23:53 UTC · last answer 2026-11-14 18:04:37 UTC · span 2444.0 s, pauses included (not charged) · charged 2444.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.6904 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 29.6 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 8B dense · 4.5B effective · source https://huggingface.co/google/gemma-4-E4B-it · read 2026-11-10 · the card states 4.5B effective (8B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 8.00B (counted from the checkpoint (7,996,157,418: language model 7.52B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 8B with embeddings, 4.5B effective; not a mixture of experts)
Input
text, English
Endpoint
http://127.0.0.1:18639/v1/chat/completions
Started
14 Nov 2026 17:23 UTC; last answer 14 Nov 2026 18:04 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Gemma 4 E4B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from google/gemma-4-E4B-it@4ecc2ba63bcac23ba4a767fc87025b1450f84c9c, vLLM reasoning parser gemma4, one RTX PRO 6000 on our GPU host alone; thinking on (chat_template_kwargs enable_thinking true, the template's own switch: it opens a system turn with <|think|>; the template default is off; on/off only, no levels); sampling temperature 0, as the Gemma 4 31B chat run; no seed (as the Gemma 4 31B run); max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 8.00B / 8.00B active / total (dense; checkpoint 7,996,157,418 incl. vision 0.17B and audio 0.31B towers; card: 8B with per-layer embeddings, 4.5B effective). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · gpt-6.1-sol-sub_en

Name
GPT-6.1 Sol chat
Setup
GPT-6.1 Sol · chat model
Model
gpt-6.1-sol · provider chatgpt-subscription
GPU
—
Cost
$0 for the run · ChatGPT subscription
Answer time
run gpt-6.1-sol-sub_en · first answer 2026-11-14 17:27:04 UTC · last answer 2026-11-14 20:32:02 UTC · span 11098.0 s, pauses included (not charged) · charged 11098.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not on record yet
Input
text, English
Endpoint
codex-chatgpt-subscription
Started
14 Nov 2026 17:26 UTC; last answer 14 Nov 2026 20:32 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
GPT-6.1 Sol on the ChatGPT subscription through the Codex CLI (`codex exec`, ChatGPT sign-in from OMP's openai-codex login in a private CODEX_HOME, no refresh token, since 13 Nov 19:35 UTC; as turkce's ask_openai), model gpt-6.1-sol, model_reasoning_effort high (Reading's setting); no output limit sent (Codex has no setting for it) and no temperature; Codex's agent instructions replaced by a neutral one-line system prompt, web search and tool features off, read-only sandbox, no user config, rules or saved session, API-key variables removed; one question per request, one request at a time; a usage-limit reply stops the run without retrying it (resumable); cost $0 (ChatGPT subscription); api_equivalent_usd is the tokens at OpenAI's API list price {"input": 2.0, "cached_input": 0.1, "cache_write": 2.5, "output": 10.0} per 1M tokens, not a cost. Codex reports no served model id. English translations.
Parameters
{
  "disabled_features": [
    "shell_tool",
    "unified_exec",
    "goals",
    "apps",
    "plugins",
    "remote_plugin",
    "tool_suggest",
    "skill_search",
    "view_image",
    "image_generation",
    "sleep_tool",
    "browser_use",
    "browser_use_external",
    "computer_use",
    "in_app_browser",
    "multi_agent",
    "unbounded_connection_retries"
  ],
  "effort": "high",
  "list_price_per_mtok": {
    "cache_write": 2.5,
    "cached_input": 0.1,
    "input": 2.0,
    "output": 10.0
  },
  "max_output_tokens": null,
  "route": "codex exec (ChatGPT sign-in)",
  "sandbox": "read-only",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message.",
  "web_search": "disabled"
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · gemma-4-e2b-it-think-32k_en

Name
Gemma 4 E2B on · chat · 32k cap
Setup
Gemma 4 E2B (chat, thinking on, 32k, on our GPU) · chat model
Model
gemma-4-e2b-it-think-32k · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.342 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.539 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.342. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gemma-4-e2b-it-think-32k_en · first answer 2026-11-14 18:51:02 UTC · last answer 2026-11-14 19:23:22 UTC · span 1940.0 s, pauses included (not charged) · charged 1940.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.3418 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 26.1 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 9 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 5.1B dense · 2.3B effective · source https://huggingface.co/google/gemma-4-E2B-it · read 2026-11-10 · the card states 2.3B effective (5.1B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 5.12B (counted from the checkpoint (5,123,178,979: language model 4.65B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 5.1B with embeddings, 2.3B effective; not a mixture of experts)
Input
text, English
Endpoint
http://127.0.0.1:18638/v1/chat/completions
Started
14 Nov 2026 18:51 UTC; last answer 14 Nov 2026 19:23 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Gemma 4 E2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from google/gemma-4-E2B-it@4adf15748de9e1a0097b6274cee9c15836c41c37, vLLM reasoning parser gemma4, one RTX PRO 6000 on our GPU host alone; thinking on (chat_template_kwargs enable_thinking true, the template's own switch: it opens a system turn with <|think|>; the template default is off; on/off only, no levels); sampling temperature 0, as the Gemma 4 31B chat run; no seed (as the Gemma 4 31B run); max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 5.12B / 5.12B active / total (dense; checkpoint 5,123,178,979 incl. vision 0.17B and audio 0.31B towers; card: 5.1B with per-layer embeddings, 2.3B effective). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · gpt-6.1-sol-sub_en_dontknow

Name
GPT-6.1 Sol chat
Setup
GPT-6.1 Sol · chat model · with the I don't know option F
Model
gpt-6.1-sol · provider chatgpt-subscription
GPU
—
Cost
$0 for the run · ChatGPT subscription
Answer time
run gpt-6.1-sol-sub_en_dontknow · first answer 2026-11-14 20:32:09 UTC · last answer 2026-11-14 23:16:53 UTC · span 9884.0 s, pauses included (not charged) · charged 9884.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not on record yet
Input
text, English
Endpoint
codex-chatgpt-subscription
Started
14 Nov 2026 20:32 UTC; last answer 14 Nov 2026 23:16 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
GPT-6.1 Sol on the ChatGPT subscription through the Codex CLI (`codex exec`, ChatGPT sign-in from OMP's openai-codex login in a private CODEX_HOME, no refresh token, since 13 Nov 19:35 UTC; as turkce's ask_openai), model gpt-6.1-sol, model_reasoning_effort high (Reading's setting); no output limit sent (Codex has no setting for it) and no temperature; Codex's agent instructions replaced by a neutral one-line system prompt, web search and tool features off, read-only sandbox, no user config, rules or saved session, API-key variables removed; one question per request, one request at a time; a usage-limit reply stops the run without retrying it (resumable); cost $0 (ChatGPT subscription); api_equivalent_usd is the tokens at OpenAI's API list price {"input": 2.0, "cached_input": 0.1, "cache_write": 2.5, "output": 10.0} per 1M tokens, not a cost. Codex reports no served model id. English translations.
Parameters
{
  "disabled_features": [
    "shell_tool",
    "unified_exec",
    "goals",
    "apps",
    "plugins",
    "remote_plugin",
    "tool_suggest",
    "skill_search",
    "view_image",
    "image_generation",
    "sleep_tool",
    "browser_use",
    "browser_use_external",
    "computer_use",
    "in_app_browser",
    "multi_agent",
    "unbounded_connection_retries"
  ],
  "effort": "high",
  "list_price_per_mtok": {
    "cache_write": 2.5,
    "cached_input": 0.1,
    "input": 2.0,
    "output": 10.0
  },
  "max_output_tokens": null,
  "route": "codex exec (ChatGPT sign-in)",
  "sandbox": "read-only",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message.",
  "web_search": "disabled"
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · qwen3.5-4b-nothink_en_dontknow

Name
Qwen3.5-4B off · chat
Setup
Qwen3.5-4B (chat, thinking off, on our GPU) · chat model · with the I don't know option F
Model
qwen3.5-4b-nothink · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.491 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.599 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.491. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run qwen3.5-4b-nothink_en_dontknow · first answer 2026-11-14 20:51:04 UTC · last answer 2026-11-14 21:26:59 UTC · span 2155.0 s, pauses included (not charged) · charged 2155.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.4905 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.0 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 4.66B dense · from the run's notes (dense; checkpoint 4,659,865,088 incl. vision 0.33B; card: 4B language model); also https://huggingface.co/Qwen/Qwen3.5-4B · read 2026-11-09 · counted from the checkpoint (4,659,865,088: language model 4.33B, vision encoder 0.33B); the card: 4B language model
Input
text, English
Endpoint
http://127.0.0.1:18642/v1/chat/completions
Started
14 Nov 2026 20:50 UTC; last answer 14 Nov 2026 21:26 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Qwen3.5-4B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from Qwen/Qwen3.5-4B@851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a, vLLM reasoning parser qwen3 (vision tower loaded, for Reading's Page images; text-only prompts here), one RTX PRO 6000 on our GPU host alone; thinking off (enable_thinking false; on/off only, no levels); sampling temperature 1, top_p 0.95, top_k 20, min_p 0, presence_penalty 1.5, repetition_penalty 1, sampling from the model card, generation_config has none (non-thinking mode for reasoning tasks; the repo has no generation_config.json); seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 4.66B / 4.66B active / total (dense; checkpoint 4,659,865,088 incl. vision 0.33B; card: 4B language model). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "presence_penalty": 1.5,
  "repetition_penalty": 1.0,
  "seed": 0,
  "temperature": 1.0,
  "top_k": 20,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · raon-1-nano-nothink_en

Name
Raon-1 nano off · chat
Setup
Raon-1 nano (chat, thinking off, KRAFTON vLLM fork, on our GPU) · chat model
Model
raon-1-nano-nothink · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$10.43 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 4.19 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $10.429. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-15 02:04:30–12:10:30 UTC†
GPU time
run raon-1-nano-nothink_en · first answer 2026-11-15 01:53:45 UTC · last answer 2026-11-16 16:11:03 UTC · span 137838.0 s, pauses included (not charged) · charged 15078.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $10.4290 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)†
Requests in flight
≈ 29.5 in flight (sum of request latencies / active time)†
Output limit
32,768 tokens · 121 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 3.1B dense · 1.5B effective · source https://huggingface.co/KRAFTON/Raon-1-nano · read 2026-11-13 · the card states 1.5B effective parameters (3.1B total, including a 1.5B per-layer embedding table) (the maker's figure is shown); checkpoint count 3.13B (3,130,797,328: per-layer embedding table 1.60B, token embeddings 0.40B tied, layers 1.12B; plus a materialised copy of the tied lm_head 0.40B and the MTP module 0.05B in separate files; dense)
Input
text, English
Endpoint
http://127.0.0.1:18637/v1/chat/completions
Started
15 Nov 2026 01:53 UTC; last answer 16 Nov 2026 16:11 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Raon-1 nano, served with KRAFTON's vLLM fork krafton-ai/vllm@13a5dd87 (branch raon-1-nano, precompiled; not a vLLM release) from KRAFTON/Raon-1-nano@fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee, vLLM reasoning parser raon_1 (the card lists Korean and English only; cost and speed are on another engine than the stock-vLLM rows), one RTX PRO 6000 on our GPU host alone; thinking off (enable_thinking false; the card: 'designed for thinking mode; disabling it is not recommended'); sampling temperature 0.1, presence_penalty 1.5, sampling from the model card, generation_config has none ('General' row); seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 1.5B / 3.1B active / total (card: 1.5B effective, 3.1B total incl. a 1.5B per-layer embedding table; MTP module 55M not loaded). English translations (DeepSeek + review). SMOKE FAILED (gate, 2026-11-15T02:06Z): 7 of 50 bad: 3 cut off, 0 empty, 4 no letter, 0 failed; full runs asked anyway at the same settings (bus #6515: both modes at 32,768, published whichever way they come out; zeki #8211); the smoke answers are kept as these runs' first answers. The full runs wait in the GPU 1 queue (zeki #8340: shown as not running, not as failed). SMOKE GATE FAILED (2026-11-16T12:10Z): 7 of 50 bad: 3 cut off, 0 empty, 4 no letter, 0 failed; full runs asked anyway at the same settings (bus #6515: both modes at 32,768, published whichever way they come out; zeki #8211); the smoke answers are kept as these runs' first answers.
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "truncate_history_thinking": false
  },
  "max_tokens": 32768,
  "presence_penalty": 1.5,
  "seed": 0,
  "temperature": 0.1
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · raon-1-nano_en

Name
Raon-1 nano on · chat
Setup
Raon-1 nano (chat, thinking on, KRAFTON vLLM fork, on our GPU) · chat model
Model
raon-1-nano · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$17.25 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 6.93 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $17.247. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-15 04:56:16–09:50:02 UTC†
GPU time
run raon-1-nano_en · first answer 2026-11-15 02:33:42 UTC · last answer 2026-11-15 14:23:03 UTC · span 42561.0 s, pauses included (not charged) · charged 24935.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $17.2467 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)†
Requests in flight
≈ 29.5 in flight (sum of request latencies / active time)†
Output limit
32,768 tokens · 242 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 3.1B dense · 1.5B effective · source https://huggingface.co/KRAFTON/Raon-1-nano · read 2026-11-13 · the card states 1.5B effective parameters (3.1B total, including a 1.5B per-layer embedding table) (the maker's figure is shown); checkpoint count 3.13B (3,130,797,328: per-layer embedding table 1.60B, token embeddings 0.40B tied, layers 1.12B; plus a materialised copy of the tied lm_head 0.40B and the MTP module 0.05B in separate files; dense)
Input
text, English
Endpoint
http://127.0.0.1:18636/v1/chat/completions
Started
15 Nov 2026 02:33 UTC; last answer 15 Nov 2026 14:23 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; paused at 1,125 of 1,125 since 2026-11-15T04:56Z (compactor #8247, zeki #8248); it resumes from its stored answers
Notes
Raon-1 nano, served with KRAFTON's vLLM fork krafton-ai/vllm@13a5dd87 (branch raon-1-nano, precompiled; not a vLLM release) from KRAFTON/Raon-1-nano@fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee, vLLM reasoning parser raon_1 (the card lists Korean and English only; cost and speed are on another engine than the stock-vLLM rows), one RTX PRO 6000 on our GPU host alone; thinking on (enable_thinking true, the template default; on/off only, no levels); sampling temperature 0.1, presence_penalty 1.5, sampling from the model card, generation_config has none ('General' row); seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 1.5B / 3.1B active / total (card: 1.5B effective, 3.1B total incl. a 1.5B per-layer embedding table; MTP module 55M not loaded). English translations (DeepSeek + review). SMOKE GATE FAILED (2026-11-15T02:50Z): 9 of 50 bad: 9 cut off, 0 empty, 0 no letter, 0 failed; full runs asked anyway at the same settings (bus #6515: both modes at 32,768, published whichever way they come out; zeki #8211); the smoke answers are kept as these runs' first answers. PAUSED 2026-11-15T04:56:37Z (compactor #8247, zeki #8248): stopped on GPU 0 of our GPU host to free the card; 32 requests were in flight and not stored (the runner has no drain), so they are asked again as fresh requests on resume. Resumes from the stored answers at the same 32,768 settings and seed on its GPU 1 queue entry: a second segment on another card of the same model, not proof of an uninterrupted identical runtime; the pause is not counted in GPU time. PAUSE ACCOUNTING (zeki #8256): runner terminated 2026-11-15T04:56:45Z, server stopped 04:56:48Z; 32 requests in flight at 04:56:45 (read from the server's metrics just before the stop), none stored. The site's GPU time runs from first to last stored answer per stretch, so the active tail from the last stored answer of this benchmark (2026-11-15T04:56:16+00:00) to the termination is not in it: the GPU time of this segment is a lower bound by that tail. On resume only the missing questions are asked, as fresh requests after the interruption. RESUMED 2026-11-15T09:48Z on GPU 0 of our GPU host (booking r_93911c3b1645; compactor #8413/#8415, Umit #8343 item 3), not on the GPU 1 queue entry the pause note names; same 32,768 settings and seed, same KRAFTON vLLM fork; only the missing questions are asked.
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "truncate_history_thinking": false
  },
  "max_tokens": 32768,
  "presence_penalty": 1.5,
  "seed": 0,
  "temperature": 0.1
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · raon-1-nano-64k_en

Name
Raon-1 nano on · chat
Setup
Raon-1 nano (chat, thinking on, KRAFTON vLLM fork, on our GPU) · chat model
Model
raon-1-nano · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.02283 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.00917 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.023. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes†
GPU time
run raon-1-nano-64k_en · first answer 2026-11-15 02:47:32 UTC · last answer 2026-11-15 02:48:05 UTC · span 33.0 s, pauses included (not charged) · charged 33.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0228 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)†
Requests in flight
≈ 5.0 in flight (sum of request latencies / active time)†
Output limit
65,536 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 3.1B dense · 1.5B effective · source https://huggingface.co/KRAFTON/Raon-1-nano · read 2026-11-13 · the card states 1.5B effective parameters (3.1B total, including a 1.5B per-layer embedding table) (the maker's figure is shown); checkpoint count 3.13B (3,130,797,328: per-layer embedding table 1.60B, token embeddings 0.40B tied, layers 1.12B; plus a materialised copy of the tied lm_head 0.40B and the MTP module 0.05B in separate files; dense)
Input
text, English
Endpoint
http://127.0.0.1:18636/v1/chat/completions
Started
15 Nov 2026 02:47 UTC; last answer 15 Nov 2026 02:48 UTC
Progress
10 answered of 1,125 asked; 0 request errors; in progress
Notes
Raon-1 nano, served with KRAFTON's vLLM fork krafton-ai/vllm@13a5dd87 (branch raon-1-nano, precompiled; not a vLLM release) from KRAFTON/Raon-1-nano@fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee, vLLM reasoning parser raon_1 (the card lists Korean and English only; cost and speed are on another engine than the stock-vLLM rows), one RTX PRO 6000 on our GPU host alone; thinking on (enable_thinking true, the template default; on/off only, no levels); sampling temperature 0.1, presence_penalty 1.5, sampling from the model card, generation_config has none ('General' row); seed 0; max_tokens 65,536 (raised from 32,768 once: the smoke failed on cut-offs only); one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 1.5B / 3.1B active / total (card: 1.5B effective, 3.1B total incl. a 1.5B per-layer embedding table; MTP module 55M not loaded). English translations (DeepSeek + review). STOPPED (15 Nov 02:49 UTC): an automatic 65,536-token retry the requested setup excludes (the cap stays 32,768, bus #6515/#8211); not a result of the setup, kept only as the record of what ran.
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "truncate_history_thinking": false
  },
  "max_tokens": 65536,
  "presence_penalty": 1.5,
  "seed": 0,
  "temperature": 0.1
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · karpo-decision-31b_en

Name
Karpo Decision 31B
Setup
karpo-decision-31B · decision model
Model
karpo-decision · provider karpo
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.1446 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0581 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.145. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run karpo-decision-31b_en · first answer 2026-11-15 05:00:11 UTC · last answer 2026-11-15 05:03:40 UTC · span 209.0 s, pauses included (not charged) · charged 209.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1446 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 31B dense · source https://huggingface.co/MachinePulseAI/karpo-decision-31B · read 2026-11-15 · the maker's figure from its name and card title (31B) is shown, as for every model (turkce #4263); not counted from the checkpoint
Input
text, English
Endpoint
http://127.0.0.1:18440
Started
15 Nov 2026 05:00 UTC; last answer 15 Nov 2026 05:03 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup karpo-decision-31B (site run None); runner 815031979; MachinePulseAI/karpo-decision-31B@d69cdd57eab23271cbe65236ca8a00d80d7bb8c7, served by compactor (karpo_decision.serve.server) on GPU 0 of our GPU host for Answer Check and Reading; Logic & Math on the same load (Umit #8157, compactor #8167) SHARED LOAD (turkce #8262, zeki #8260): compactor's karpo-decision-31B server on GPU 0 of our GPU host also answered turkce's Reading runs from 04:58 UTC, one request at a time, while these runs ran (05:00-05:20 UTC), so latency, throughput and GPU-time cost were measured on a shared server, not an isolated load; the answers are unaffected.
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "karpo-decision",
    "--provider",
    "karpo",
    "--timeout",
    "120",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "815031979442b7cd01229dcf844de47d47478ac9",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "karpo-decision",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "karpo",
  "questions": 1125,
  "request_seconds": 197.67897001303209,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "f12c452fa463a10fd6c6c43a8c7357baff6b61befd0437eef958ddb075632793",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 482759,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:18440/v1/systemone
{
 "model": "karpo-decision",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · karpo-decision-31b_en_dontknow

Name
Karpo Decision 31B
Setup
karpo-decision-31B · decision model · with the I don't know option F
Model
karpo-decision · provider karpo
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.1605 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0644 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.160. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run karpo-decision-31b_en_dontknow · first answer 2026-11-15 05:09:43 UTC · last answer 2026-11-15 05:13:35 UTC · span 232.0 s, pauses included (not charged) · charged 232.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1605 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 31B dense · source https://huggingface.co/MachinePulseAI/karpo-decision-31B · read 2026-11-15 · the maker's figure from its name and card title (31B) is shown, as for every model (turkce #4263); not counted from the checkpoint
Input
text, English
Endpoint
http://127.0.0.1:18440
Started
15 Nov 2026 05:09 UTC; last answer 15 Nov 2026 05:13 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup karpo-decision-31B (site run None); runner 815031979; MachinePulseAI/karpo-decision-31B@d69cdd57eab23271cbe65236ca8a00d80d7bb8c7, served by compactor (karpo_decision.serve.server) on GPU 0 of our GPU host for Answer Check and Reading; Logic & Math on the same load (Umit #8157, compactor #8167) SHARED LOAD (turkce #8262, zeki #8260): compactor's karpo-decision-31B server on GPU 0 of our GPU host also answered turkce's Reading runs from 04:58 UTC, one request at a time, while these runs ran (05:00-05:20 UTC), so latency, throughput and GPU-time cost were measured on a shared server, not an isolated load; the answers are unaffected.
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "karpo-decision",
    "--provider",
    "karpo",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "815031979442b7cd01229dcf844de47d47478ac9",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "karpo-decision",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "karpo",
  "questions": 1125,
  "request_seconds": 223.5517578200379,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "21bd740b8ce5f3f8d2052a8aaba981175020629941428dcf264bd2ef58f48ce3",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 539009,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:18440/v1/systemone
{
 "model": "karpo-decision",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · copernicus-31b_en

Name
Copernicus-31B
Setup
Copernicus-31B · decision model
Model
Copernicus-31B · provider copernicus
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.1529 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0614 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.153. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run copernicus-31b_en · first answer 2026-11-15 05:25:01 UTC · last answer 2026-11-15 05:28:42 UTC · span 221.0 s, pauses included (not charged) · charged 221.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1529 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 31B dense · source https://huggingface.co/Poindexter-Labs/Copernicus-31B · read 2026-11-15 · the maker's stated figure (31 billion parameters, the card) is shown, as for every model (turkce #4263); not counted from the checkpoint
Input
text, English
Endpoint
http://127.0.0.1:18441
Started
15 Nov 2026 05:25 UTC; last answer 15 Nov 2026 05:28 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Copernicus-31B (site run None); runner 815031979; Poindexter-Labs/Copernicus-31B@be715259e97a743341f2d7a3e6a6298dab23ca0e, served by compactor (copernicus_server.py --backend hf) on GPU 0 of our GPU host for Answer Check and Reading; Logic & Math on the same load (Umit #8157, compactor #8167)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "Copernicus-31B",
    "--provider",
    "copernicus",
    "--timeout",
    "120",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "815031979442b7cd01229dcf844de47d47478ac9",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "Copernicus-31B",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "copernicus",
  "questions": 1125,
  "request_seconds": 217.3078491869819,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "fd490730c269af35adf5490ffd310dd9d2bf98304dd9281813bbc644b04ccb62",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 836143,
    "output_tokens": 1125
  }
}
Prompt template
POST http://127.0.0.1:18441/v1/systemone
{
 "model": "Copernicus-31B",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · gpt-6-luna-low-sub_en

Name
GPT-6 Luna chat
Setup
GPT-6 Luna (low) · chat model
Model
gpt-6-luna · provider chatgpt-subscription
GPU
—
Cost
$0 for the run · ChatGPT subscription
Answer time
run gpt-6-luna-low-sub_en · first answer 2026-11-15 05:29:04 UTC · last answer 2026-11-15 06:47:37 UTC · span 4713.0 s, pauses included (not charged) · charged 4713.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker (closed weights) · source https://openrouter.ai/openai/gpt-6-luna · read 2026-11-10
Input
text, English
Endpoint
codex-chatgpt-subscription
Started
15 Nov 2026 05:29 UTC; last answer 15 Nov 2026 06:47 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
GPT-6 Luna (low) on the ChatGPT subscription through the Codex CLI (`codex exec`, ChatGPT sign-in from OMP's openai-codex login in a private CODEX_HOME, no refresh token, since 13 Nov 19:35 UTC; as turkce's ask_openai), model gpt-6-luna, model_reasoning_effort low (Reading's setting); no output limit sent (Codex has no setting for it) and no temperature; Codex's agent instructions replaced by a neutral one-line system prompt, web search and tool features off, read-only sandbox, no user config, rules or saved session, API-key variables removed; one question per request, one request at a time; a usage-limit reply stops the run without retrying it (resumable); cost $0 (ChatGPT subscription); api_equivalent_usd is the tokens at OpenAI's API list price {"input": 0.1, "cached_input": 0.01, "cache_write": 0.125, "output": 0.5} per 1M tokens, not a cost. Codex reports no served model id. English translations.
Parameters
{
  "disabled_features": [
    "shell_tool",
    "unified_exec",
    "goals",
    "apps",
    "plugins",
    "remote_plugin",
    "tool_suggest",
    "skill_search",
    "view_image",
    "image_generation",
    "sleep_tool",
    "browser_use",
    "browser_use_external",
    "computer_use",
    "in_app_browser",
    "multi_agent",
    "unbounded_connection_retries"
  ],
  "effort": "low",
  "list_price_per_mtok": {
    "cache_write": 0.125,
    "cached_input": 0.01,
    "input": 0.1,
    "output": 0.5
  },
  "max_output_tokens": null,
  "route": "codex exec (ChatGPT sign-in)",
  "sandbox": "read-only",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message.",
  "web_search": "disabled"
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · copernicus-31b_en_dontknow

Name
Copernicus-31B
Setup
Copernicus-31B · decision model · with the I don't know option F
Model
Copernicus-31B · provider copernicus
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.1688 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0678 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.169. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run copernicus-31b_en_dontknow · first answer 2026-11-15 05:33:17 UTC · last answer 2026-11-15 05:37:21 UTC · span 244.0 s, pauses included (not charged) · charged 244.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1688 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 31B dense · source https://huggingface.co/Poindexter-Labs/Copernicus-31B · read 2026-11-15 · the maker's stated figure (31 billion parameters, the card) is shown, as for every model (turkce #4263); not counted from the checkpoint
Input
text, English
Endpoint
http://127.0.0.1:18441
Started
15 Nov 2026 05:33 UTC; last answer 15 Nov 2026 05:37 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Copernicus-31B (site run None); runner 815031979; Poindexter-Labs/Copernicus-31B@be715259e97a743341f2d7a3e6a6298dab23ca0e, served by compactor (copernicus_server.py --backend hf) on GPU 0 of our GPU host for Answer Check and Reading; Logic & Math on the same load (Umit #8157, compactor #8167)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "Copernicus-31B",
    "--provider",
    "copernicus",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "815031979442b7cd01229dcf844de47d47478ac9",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "Copernicus-31B",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "copernicus",
  "questions": 1125,
  "request_seconds": 241.85325528406247,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "fdabfa84950da431434e3f333f01d3ad8f9406d04ef57d44ff7318e7d153ed5d",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 910393,
    "output_tokens": 1125
  }
}
Prompt template
POST http://127.0.0.1:18441/v1/systemone
{
 "model": "Copernicus-31B",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · qwen3.5-2b-nothink_en

Name
Qwen3.5-2B off · chat
Setup
Qwen3.5-2B (chat, thinking off, on our GPU) · chat model
Model
qwen3.5-2b-nothink · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.805 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.725 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.805. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run qwen3.5-2b-nothink_en · first answer 2026-11-15 06:18:13 UTC · last answer 2026-11-15 07:01:43 UTC · span 2610.0 s, pauses included (not charged) · charged 2610.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.8053 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 30.2 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 2B dense · source https://huggingface.co/Qwen/Qwen3.5-2B · read 2026-11-13 · the card states Number of Parameters: 2B (the maker's figure is shown); checkpoint count 2.27B (2,274,069,824: language model 1.88B incl. tied embeddings, vision encoder 0.33B, MTP head 0.06B; dense)
Input
text, English
Endpoint
http://127.0.0.1:18643/v1/chat/completions
Started
15 Nov 2026 06:18 UTC; last answer 15 Nov 2026 07:01 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Qwen3.5-2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from Qwen/Qwen3.5-2B@15852e8c16360a2fea060d615a32b45270f8a8fc, vLLM reasoning parser qwen3 (vision tower loaded, for Reading's Page images; text-only prompts here), one RTX PRO 6000 on our GPU host alone; thinking off (enable_thinking false, the template default for this size; on/off only, no levels); sampling temperature 1, top_p 0.95, top_k 20, min_p 0, presence_penalty 1.5, repetition_penalty 1, sampling from the model card, generation_config has none (non-thinking mode for reasoning tasks; the repo has no generation_config.json); seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 2.27B / 2.27B active / total (dense; checkpoint 2,274,069,824: language model 1.88B incl. tied embeddings, vision encoder 0.33B, MTP head 0.06B (not loaded); card: 2B). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "presence_penalty": 1.5,
  "repetition_penalty": 1.0,
  "seed": 0,
  "temperature": 1.0,
  "top_k": 20,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · knowline-9b_en

Name
KnowLine-9B-Gen1
Setup
KnowLine-9B-Gen1 · decision model
Model
PelaAI/KnowLine-9B-Gen1 · provider knowline
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.04081 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0164 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.041. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run knowline-9b_en · first answer 2026-11-15 06:19:19 UTC · last answer 2026-11-15 06:20:18 UTC · span 59.0 s, pauses included (not charged) · charged 59.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0408 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/PelaAI/KnowLine-9B-Gen1 · read 2026-11-15 · the maker's figure from its name (9B) is shown, as for every model (turkce #4263); built on Qwen3.5-9B (9.65B counted in its own checkpoint); this checkpoint not counted
Input
text, English
Endpoint
http://127.0.0.1:18450
Started
15 Nov 2026 06:19 UTC; last answer 15 Nov 2026 06:20 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup KnowLine-9B-Gen1 (site run None); runner 815031979; PelaAI/KnowLine-9B-Gen1@325378d128e9f5eb5088196668f5874c5682b6c0, served by compactor on GPU 0 of our GPU host with the card's serve_knowline.sh (SGLang, then knowline_server.py, card defaults, read twice up to 3,000 tokens) in the lmsysorg/sglang image with SGLang 0.5.23; the card's release launcher names SGLang 0.5.21, not available here (compactor #8276). Logic & Math on the same load as Answer Check and Reading (Umit #8157, compactor #8167)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "PelaAI/KnowLine-9B-Gen1",
    "--provider",
    "knowline",
    "--timeout",
    "120",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "815031979442b7cd01229dcf844de47d47478ac9",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "PelaAI/KnowLine-9B-Gen1",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "knowline",
  "questions": 1125,
  "request_seconds": 57.00291736816871,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "71478a2806857dbaa84ef4ea51dbe2a018f3cdad6654380b87af39a4b644863e",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 836764,
    "output_tokens": 1125
  }
}
Prompt template
POST http://127.0.0.1:18450/v1/systemone
{
 "model": "PelaAI/KnowLine-9B-Gen1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · knowline-9b_en_dontknow

Name
KnowLine-9B-Gen1
Setup
KnowLine-9B-Gen1 · decision model · with the I don't know option F
Model
PelaAI/KnowLine-9B-Gen1 · provider knowline
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.04012 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0161 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.040. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run knowline-9b_en_dontknow · first answer 2026-11-15 06:21:48 UTC · last answer 2026-11-15 06:22:46 UTC · span 58.0 s, pauses included (not charged) · charged 58.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0401 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/PelaAI/KnowLine-9B-Gen1 · read 2026-11-15 · the maker's figure from its name (9B) is shown, as for every model (turkce #4263); built on Qwen3.5-9B (9.65B counted in its own checkpoint); this checkpoint not counted
Input
text, English
Endpoint
http://127.0.0.1:18450
Started
15 Nov 2026 06:21 UTC; last answer 15 Nov 2026 06:22 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup KnowLine-9B-Gen1 (site run None); runner 815031979; PelaAI/KnowLine-9B-Gen1@325378d128e9f5eb5088196668f5874c5682b6c0, served by compactor on GPU 0 of our GPU host with the card's serve_knowline.sh (SGLang, then knowline_server.py, card defaults, read twice up to 3,000 tokens) in the lmsysorg/sglang image with SGLang 0.5.23; the card's release launcher names SGLang 0.5.21, not available here (compactor #8276). Logic & Math on the same load as Answer Check and Reading (Umit #8157, compactor #8167)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "PelaAI/KnowLine-9B-Gen1",
    "--provider",
    "knowline",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "815031979442b7cd01229dcf844de47d47478ac9",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "PelaAI/KnowLine-9B-Gen1",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "knowline",
  "questions": 1125,
  "request_seconds": 56.679356671076675,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "58f2556404f7e83bff2b262fedf69c644c1f201775473f952e58f0bd8cea3d3b",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 904264,
    "output_tokens": 1125
  }
}
Prompt template
POST http://127.0.0.1:18450/v1/systemone
{
 "model": "PelaAI/KnowLine-9B-Gen1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · gpt-6-luna-low-sub_en_dontknow

Name
GPT-6 Luna chat
Setup
GPT-6 Luna (low) · chat model · with the I don't know option F
Model
gpt-6-luna · provider chatgpt-subscription
GPU
—
Cost
$0 for the run · ChatGPT subscription
Answer time
run gpt-6-luna-low-sub_en_dontknow · first answer 2026-11-15 06:47:40 UTC · last answer 2026-11-15 08:00:28 UTC · span 4368.0 s, pauses included (not charged) · charged 4368.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker (closed weights) · source https://openrouter.ai/openai/gpt-6-luna · read 2026-11-10
Input
text, English
Endpoint
codex-chatgpt-subscription
Started
15 Nov 2026 06:47 UTC; last answer 15 Nov 2026 08:00 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
GPT-6 Luna (low) on the ChatGPT subscription through the Codex CLI (`codex exec`, ChatGPT sign-in from OMP's openai-codex login in a private CODEX_HOME, no refresh token, since 13 Nov 19:35 UTC; as turkce's ask_openai), model gpt-6-luna, model_reasoning_effort low (Reading's setting); no output limit sent (Codex has no setting for it) and no temperature; Codex's agent instructions replaced by a neutral one-line system prompt, web search and tool features off, read-only sandbox, no user config, rules or saved session, API-key variables removed; one question per request, one request at a time; a usage-limit reply stops the run without retrying it (resumable); cost $0 (ChatGPT subscription); api_equivalent_usd is the tokens at OpenAI's API list price {"input": 0.1, "cached_input": 0.01, "cache_write": 0.125, "output": 0.5} per 1M tokens, not a cost. Codex reports no served model id. English translations.
Parameters
{
  "disabled_features": [
    "shell_tool",
    "unified_exec",
    "goals",
    "apps",
    "plugins",
    "remote_plugin",
    "tool_suggest",
    "skill_search",
    "view_image",
    "image_generation",
    "sleep_tool",
    "browser_use",
    "browser_use_external",
    "computer_use",
    "in_app_browser",
    "multi_agent",
    "unbounded_connection_retries"
  ],
  "effort": "low",
  "list_price_per_mtok": {
    "cache_write": 0.125,
    "cached_input": 0.01,
    "input": 0.1,
    "output": 0.5
  },
  "max_output_tokens": null,
  "route": "codex exec (ChatGPT sign-in)",
  "sandbox": "read-only",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message.",
  "web_search": "disabled"
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · adjudex-9b_en

Name
Adjudex 9B v5
Setup
Adjudex 9B v5 · decision model
Model
jev-latest · provider pplx-decider
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.03528 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0142 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.035. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run adjudex-9b_en · first answer 2026-11-15 06:48:42 UTC · last answer 2026-11-15 06:49:33 UTC · span 51.0 s, pauses included (not charged) · charged 51.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0353 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/newsbang/adjudex-9b-v5 · read 2026-11-15 · the maker's figure from its name (9B) is shown, as for every model (turkce #4263); a fine-tune of Qwen3.5-9B; this checkpoint not counted
Input
text, English
Endpoint
http://127.0.0.1:18451
Started
15 Nov 2026 06:48 UTC; last answer 15 Nov 2026 06:49 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Adjudex 9B v5 (site run None); runner 815031979; newsbang/adjudex-9b-v5@ffdcb8254d1b22e40b2bbbf569f143e0b357aeca (Apache-2.0, a fine-tune of Qwen3.5-9B), served by compactor on GPU 0 of our GPU host with the card's own autojev server (the Perplexity Decider v1.1 source, unchanged: python -m autojev.server in the v1.1 environment, whose requirements are identical to the card's; default settings, max length 8,192); the Perplexity Decider request format (runner provider pplx-decider), model field 'jev-latest' (compactor #8298). Logic & Math on the same load as Answer Check and Reading (Umit #8157, compactor #8167)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "jev-latest",
    "--provider",
    "pplx-decider",
    "--timeout",
    "60",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "815031979442b7cd01229dcf844de47d47478ac9",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "jev-latest",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "pplx-decider",
  "questions": 1125,
  "request_seconds": 45.05185074804467,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "eb2000d3d50c21f546e1afbaa94f20c72b139867d054f6c7d8f1c3ead4a122ac",
  "runner": "benchmark_ollama.py",
  "runner_commit": "815031979442b7cd01229dcf844de47d47478ac9",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 422882,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:18451/v1/systemone
{
 "model": "jev-latest",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · adjudex-9b_en_dontknow

Name
Adjudex 9B v5
Setup
Adjudex 9B v5 · decision model · with the I don't know option F
Model
jev-latest · provider pplx-decider
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.03666 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0147 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.037. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run adjudex-9b_en_dontknow · first answer 2026-11-15 06:51:02 UTC · last answer 2026-11-15 06:51:55 UTC · span 53.0 s, pauses included (not charged) · charged 53.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0367 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/newsbang/adjudex-9b-v5 · read 2026-11-15 · the maker's figure from its name (9B) is shown, as for every model (turkce #4263); a fine-tune of Qwen3.5-9B; this checkpoint not counted
Input
text, English
Endpoint
http://127.0.0.1:18451
Started
15 Nov 2026 06:51 UTC; last answer 15 Nov 2026 06:51 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Adjudex 9B v5 (site run None); runner 815031979; newsbang/adjudex-9b-v5@ffdcb8254d1b22e40b2bbbf569f143e0b357aeca (Apache-2.0, a fine-tune of Qwen3.5-9B), served by compactor on GPU 0 of our GPU host with the card's own autojev server (the Perplexity Decider v1.1 source, unchanged: python -m autojev.server in the v1.1 environment, whose requirements are identical to the card's; default settings, max length 8,192); the Perplexity Decider request format (runner provider pplx-decider), model field 'jev-latest' (compactor #8298). Logic & Math on the same load as Answer Check and Reading (Umit #8157, compactor #8167)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "jev-latest",
    "--provider",
    "pplx-decider",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "815031979442b7cd01229dcf844de47d47478ac9",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "jev-latest",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "pplx-decider",
  "questions": 1125,
  "request_seconds": 47.96552510197944,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "92e55075927826cfb4fa32e09cb7b98da23cfd9a53fa9d554d60d61eda5f8c85",
  "runner": "benchmark_ollama.py",
  "runner_commit": "815031979442b7cd01229dcf844de47d47478ac9",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 456632,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:18451/v1/systemone
{
 "model": "jev-latest",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · jeffan-9b_en

Name
Jeffan-9B-1211
Setup
Jeffan-9B-1211 · decision model
Model
Jeffan-9B-1211 · provider di-serve
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.04427 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0178 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.044. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run jeffan-9b_en · first answer 2026-11-15 07:09:48 UTC · last answer 2026-11-15 07:10:52 UTC · span 64.0 s, pauses included (not charged) · charged 64.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0443 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/tattrongvu/Jeffan-9B-1211 · read 2026-11-15 · the maker's figure from its name (9B) is shown, as for every model (turkce #4263); the card's table states 8.95B; a fine-tune of Qwen3.5-9B whose middle layers run twice per pass (no extra parameters)
Input
text, English
Endpoint
http://127.0.0.1:18453
Started
15 Nov 2026 07:09 UTC; last answer 15 Nov 2026 07:10 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Jeffan-9B-1211 (site run None); runner a75966fe5; tattrongvu/Jeffan-9B-1211@862d1a7a8b64a8d708ddb0b0cd775a740f01841a (Apache-2.0, a fine-tune of Qwen3.5-9B), served by compactor on GPU 0 of our GPU host with his di_serve.py, a thin System One wrapper around the card's own jeffan_engine:DecisionEngine loaded as the Decision Index kit does (decision-index@f8552a1; options max_len and budget 65,536, as the card; transformers 5.17.0, flash-linear-attention 0.5.2, without the optional causal-conv1d, so its PyTorch fallback); returns the engine's own answers, one request at a time (compactor #8319). Runner provider di-serve, which does not send a model field. Logic & Math on the same load as Answer Check and Reading (Umit #8157, compactor #8167)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "Jeffan-9B-1211",
    "--provider",
    "di-serve",
    "--timeout",
    "120",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "Jeffan-9B-1211",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "di-serve",
  "questions": 1125,
  "request_seconds": 59.50900639401516,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "5fb1237cba1163f98cf992313e132e610afc0b8c5ad0860d44f6f92ef2598470",
  "runner": "benchmark_ollama.py",
  "runner_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 428507,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:18453/v1/systemone
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · jeffan-9b_en_dontknow

Name
Jeffan-9B-1211
Setup
Jeffan-9B-1211 · decision model · with the I don't know option F
Model
Jeffan-9B-1211 · provider di-serve
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.04565 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0183 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.046. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run jeffan-9b_en_dontknow · first answer 2026-11-15 07:12:29 UTC · last answer 2026-11-15 07:13:35 UTC · span 66.0 s, pauses included (not charged) · charged 66.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0457 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/tattrongvu/Jeffan-9B-1211 · read 2026-11-15 · the maker's figure from its name (9B) is shown, as for every model (turkce #4263); the card's table states 8.95B; a fine-tune of Qwen3.5-9B whose middle layers run twice per pass (no extra parameters)
Input
text, English
Endpoint
http://127.0.0.1:18453
Started
15 Nov 2026 07:12 UTC; last answer 15 Nov 2026 07:13 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Jeffan-9B-1211 (site run None); runner a75966fe5; tattrongvu/Jeffan-9B-1211@862d1a7a8b64a8d708ddb0b0cd775a740f01841a (Apache-2.0, a fine-tune of Qwen3.5-9B), served by compactor on GPU 0 of our GPU host with his di_serve.py, a thin System One wrapper around the card's own jeffan_engine:DecisionEngine loaded as the Decision Index kit does (decision-index@f8552a1; options max_len and budget 65,536, as the card; transformers 5.17.0, flash-linear-attention 0.5.2, without the optional causal-conv1d, so its PyTorch fallback); returns the engine's own answers, one request at a time (compactor #8319). Runner provider di-serve, which does not send a model field. Logic & Math on the same load as Answer Check and Reading (Umit #8157, compactor #8167)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "Jeffan-9B-1211",
    "--provider",
    "di-serve",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "Jeffan-9B-1211",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "di-serve",
  "questions": 1125,
  "request_seconds": 64.14773532593972,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "8ea2b9b843eeb3a2ae19d9bbdb8ba62750c7abc0ee26fc6ffd01944f566f0083",
  "runner": "benchmark_ollama.py",
  "runner_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 462257,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:18453/v1/systemone
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · qwen3.5-2b-nothink_en_dontknow

Name
Qwen3.5-2B off · chat
Setup
Qwen3.5-2B (chat, thinking off, on our GPU) · chat model · with the I don't know option F
Model
qwen3.5-2b-nothink · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.678 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.674 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.678. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run qwen3.5-2b-nothink_en_dontknow · first answer 2026-11-15 07:31:57 UTC · last answer 2026-11-15 08:12:23 UTC · span 2426.0 s, pauses included (not charged) · charged 2426.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.6780 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.5 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 2B dense · source https://huggingface.co/Qwen/Qwen3.5-2B · read 2026-11-13 · the card states Number of Parameters: 2B (the maker's figure is shown); checkpoint count 2.27B (2,274,069,824: language model 1.88B incl. tied embeddings, vision encoder 0.33B, MTP head 0.06B; dense)
Input
text, English
Endpoint
http://127.0.0.1:18643/v1/chat/completions
Started
15 Nov 2026 07:31 UTC; last answer 15 Nov 2026 08:12 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Qwen3.5-2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from Qwen/Qwen3.5-2B@15852e8c16360a2fea060d615a32b45270f8a8fc, vLLM reasoning parser qwen3 (vision tower loaded, for Reading's Page images; text-only prompts here), one RTX PRO 6000 on our GPU host alone; thinking off (enable_thinking false, the template default for this size; on/off only, no levels); sampling temperature 1, top_p 0.95, top_k 20, min_p 0, presence_penalty 1.5, repetition_penalty 1, sampling from the model card, generation_config has none (non-thinking mode for reasoning tasks; the repo has no generation_config.json); seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 2.27B / 2.27B active / total (dense; checkpoint 2,274,069,824: language model 1.88B incl. tied embeddings, vision encoder 0.33B, MTP head 0.06B (not loaded); card: 2B). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "presence_penalty": 1.5,
  "repetition_penalty": 1.0,
  "seed": 0,
  "temperature": 1.0,
  "top_k": 20,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · chinese-pointer-4b_en

Name
Chinese Decision Pointer 4B
Setup
Chinese Decision Pointer 4B · decision model
Model
Chinese-Decision-Pointer-4B · provider di-serve
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.0332 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0133 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.033. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run chinese-pointer-4b_en · first answer 2026-11-15 07:48:38 UTC · last answer 2026-11-15 07:49:26 UTC · span 48.0 s, pauses included (not charged) · charged 48.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0332 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 4B dense · source https://huggingface.co/quincyqiang/chinese-decision-pointer-4b · read 2026-11-15 · the maker's figure from its name (4B) is shown, as for every model (turkce #4263); a fine-tune of Qwen3.5-4B; this checkpoint not counted
Input
text, English
Endpoint
http://127.0.0.1:18455
Started
15 Nov 2026 07:48 UTC; last answer 15 Nov 2026 07:49 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Chinese Decision Pointer 4B (site run None); runner a75966fe5; quincyqiang/chinese-decision-pointer-4b@889f23e2655e5cba1208bb17f90526092813f5f2 (Apache-2.0, a fine-tune of Qwen3.5-4B), served by compactor on GPU 0 of our GPU host with his di_serve.py around the card's ChineseDecisionEngine (yanqiangmiffy/chinese-decision@c16450b) with debias 'reverse' as in the card's run (averages the original and the reversed option order); Decision Index kit decision-index@9eb2dbe2 (the card names no kit commit), transformers 5.17.0, without causal-conv1d (compactor #8376). Runner provider di-serve, which sends no model field. Logic & Math on the same load as Answer Check and Reading (Umit #8157, compactor #8167)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "Chinese-Decision-Pointer-4B",
    "--provider",
    "di-serve",
    "--timeout",
    "120",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "Chinese-Decision-Pointer-4B",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "di-serve",
  "questions": 1125,
  "request_seconds": 43.5397424249677,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "e47e3d0820ce5af80418507f7cecfb75d3f4464aa204a5ea30cea5f6949061a7",
  "runner": "benchmark_ollama.py",
  "runner_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 791764,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:18455/v1/systemone
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · chinese-pointer-4b_en_dontknow

Name
Chinese Decision Pointer 4B
Setup
Chinese Decision Pointer 4B · decision model · with the I don't know option F
Model
Chinese-Decision-Pointer-4B · provider di-serve
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.03251 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0131 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.033. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run chinese-pointer-4b_en_dontknow · first answer 2026-11-15 07:50:43 UTC · last answer 2026-11-15 07:51:30 UTC · span 47.0 s, pauses included (not charged) · charged 47.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0325 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 4B dense · source https://huggingface.co/quincyqiang/chinese-decision-pointer-4b · read 2026-11-15 · the maker's figure from its name (4B) is shown, as for every model (turkce #4263); a fine-tune of Qwen3.5-4B; this checkpoint not counted
Input
text, English
Endpoint
http://127.0.0.1:18455
Started
15 Nov 2026 07:50 UTC; last answer 15 Nov 2026 07:51 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Chinese Decision Pointer 4B (site run None); runner a75966fe5; quincyqiang/chinese-decision-pointer-4b@889f23e2655e5cba1208bb17f90526092813f5f2 (Apache-2.0, a fine-tune of Qwen3.5-4B), served by compactor on GPU 0 of our GPU host with his di_serve.py around the card's ChineseDecisionEngine (yanqiangmiffy/chinese-decision@c16450b) with debias 'reverse' as in the card's run (averages the original and the reversed option order); Decision Index kit decision-index@9eb2dbe2 (the card names no kit commit), transformers 5.17.0, without causal-conv1d (compactor #8376). Runner provider di-serve, which sends no model field. Logic & Math on the same load as Answer Check and Reading (Umit #8157, compactor #8167)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "Chinese-Decision-Pointer-4B",
    "--provider",
    "di-serve",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "Chinese-Decision-Pointer-4B",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "di-serve",
  "questions": 1125,
  "request_seconds": 42.02176410694665,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "8df3343e12281d4fba9e855301524cfe340dc8079b2af09a3b773625a14f1ef8",
  "runner": "benchmark_ollama.py",
  "runner_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 861514,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:18455/v1/systemone
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · logits-sigma-1-duo_en

Name
Logits Sigma-1 Duo
Setup
Logits Sigma-1 Duo · decision model
Model
Logits-Sigma-1-Duo · provider di-serve
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.2379 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0956 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.238. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run logits-sigma-1-duo_en · first answer 2026-11-15 08:37:08 UTC · last answer 2026-11-15 08:42:52 UTC · span 344.0 s, pauses included (not charged) · charged 344.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.2379 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 57.36B dense · source https://huggingface.co/logitshq/Logits-Sigma-1-Duo · read 2026-11-15 · the card states no total; counted from its two parts, as REAX 1.3 (both used on every question, option probabilities averaged): KnowLine 31B Gen1 with Logits' LoRA plus vLLM-SR d3-ultra (26.09B); stored in FP8
Input
text, English
Endpoint
http://127.0.0.1:18456
Started
15 Nov 2026 08:37 UTC; last answer 15 Nov 2026 08:42 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Logits Sigma-1 Duo (site run None); runner a75966fe5; logitshq/Logits-Sigma-1-Duo@a0c1b6e8242916aafdcf9a5a49c1a952cb56e091 (Apache-2.0; weights and code identical to d9641cd8, the revision its card's run names): two decision models averaged per option, KnowLine 31B Gen1 with Logits' LoRA and vLLM-SR d3-ultra, both stored in FP8. Served by compactor on GPU 0 of our GPU host with his di_serve.py around the card's engine (logits_sigma_1_duo.engine_pair:Duo, model=<snapshot>, as its serve.sh fixes them) in the card's own serving image (code/Dockerfile, kit decision-index@25c6ec7, causal-conv1d 1.7.0) plus only gcc and Python headers, which its Triton kernels need to compile at first use. FP8 matmuls on the engine's own Triton kernel (its cuBLAS block-scaled path is not taken on this GPU; the engine falls back by design); d3-ultra on fla 0.6.0 (compactor #8398). Runner provider di-serve, which sends no model field. Logic & Math on the same load as Answer Check and Reading (Umit #8157, compactor #8167)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "Logits-Sigma-1-Duo",
    "--provider",
    "di-serve",
    "--timeout",
    "120",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "Logits-Sigma-1-Duo",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "di-serve",
  "questions": 1125,
  "request_seconds": 336.9311041710025,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "68172a62ff39595279f1b0000b86a4349affbd49cd8e918fce7e299e40543f03",
  "runner": "benchmark_ollama.py",
  "runner_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 0,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:18456/v1/systemone
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · logits-sigma-1-duo_en_dontknow

Name
Logits Sigma-1 Duo
Setup
Logits Sigma-1 Duo · decision model · with the I don't know option F
Model
Logits-Sigma-1-Duo · provider di-serve
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.2283 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0917 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.228. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run logits-sigma-1-duo_en_dontknow · first answer 2026-11-15 08:50:27 UTC · last answer 2026-11-15 08:55:57 UTC · span 330.0 s, pauses included (not charged) · charged 330.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.2283 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 57.36B dense · source https://huggingface.co/logitshq/Logits-Sigma-1-Duo · read 2026-11-15 · the card states no total; counted from its two parts, as REAX 1.3 (both used on every question, option probabilities averaged): KnowLine 31B Gen1 with Logits' LoRA plus vLLM-SR d3-ultra (26.09B); stored in FP8
Input
text, English
Endpoint
http://127.0.0.1:18456
Started
15 Nov 2026 08:50 UTC; last answer 15 Nov 2026 08:55 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Logits Sigma-1 Duo (site run None); runner a75966fe5; logitshq/Logits-Sigma-1-Duo@a0c1b6e8242916aafdcf9a5a49c1a952cb56e091 (Apache-2.0; weights and code identical to d9641cd8, the revision its card's run names): two decision models averaged per option, KnowLine 31B Gen1 with Logits' LoRA and vLLM-SR d3-ultra, both stored in FP8. Served by compactor on GPU 0 of our GPU host with his di_serve.py around the card's engine (logits_sigma_1_duo.engine_pair:Duo, model=<snapshot>, as its serve.sh fixes them) in the card's own serving image (code/Dockerfile, kit decision-index@25c6ec7, causal-conv1d 1.7.0) plus only gcc and Python headers, which its Triton kernels need to compile at first use. FP8 matmuls on the engine's own Triton kernel (its cuBLAS block-scaled path is not taken on this GPU; the engine falls back by design); d3-ultra on fla 0.6.0 (compactor #8398). Runner provider di-serve, which sends no model field. Logic & Math on the same load as Answer Check and Reading (Umit #8157, compactor #8167)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "Logits-Sigma-1-Duo",
    "--provider",
    "di-serve",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "Logits-Sigma-1-Duo",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "di-serve",
  "questions": 1125,
  "request_seconds": 321.55639248373336,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "53aa1dec35ae9d460360e377d0dc21a6cc79dcb17f6c280b5178e86d121a7b42",
  "runner": "benchmark_ollama.py",
  "runner_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 0,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:18456/v1/systemone
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · qwen3.5-2b_en

Name
Qwen3.5-2B on · chat
Setup
Qwen3.5-2B (chat, thinking, on our GPU) · chat model
Model
qwen3.5-2b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$4.871 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 1.96 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $4.871. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run qwen3.5-2b_en · first answer 2026-11-15 10:00:07 UTC · last answer 2026-11-15 11:57:30 UTC · span 7043.0 s, pauses included (not charged) · charged 7043.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $4.8714 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 30.2 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 101 answers cut off at it with no letter (counted wrong) · 1 empty reply asked once more
Model size
Parameters: 2B dense · source https://huggingface.co/Qwen/Qwen3.5-2B · read 2026-11-13 · the card states Number of Parameters: 2B (the maker's figure is shown); checkpoint count 2.27B (2,274,069,824: language model 1.88B incl. tied embeddings, vision encoder 0.33B, MTP head 0.06B; dense)
Input
text, English
Endpoint
http://127.0.0.1:18635/v1/chat/completions
Started
15 Nov 2026 09:59 UTC; last answer 15 Nov 2026 11:57 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Qwen3.5-2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from Qwen/Qwen3.5-2B@15852e8c16360a2fea060d615a32b45270f8a8fc, vLLM reasoning parser qwen3 (vision tower loaded, for Reading's Page images; text-only prompts here; the card warns the 2B 'is more prone to entering thinking loops' in thinking mode), one RTX PRO 6000 on our GPU host alone; thinking on (enable_thinking true; the template default is off for this size; on/off only, no levels); sampling temperature 1, top_p 0.95, top_k 20, min_p 0, presence_penalty 1.5, repetition_penalty 1, sampling from the model card, generation_config has none (thinking mode, text tasks; the repo has no generation_config.json); seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 2.27B / 2.27B active / total (dense; checkpoint 2,274,069,824: language model 1.88B incl. tied embeddings, vision encoder 0.33B, MTP head 0.06B (not loaded); card: 2B). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "presence_penalty": 1.5,
  "repetition_penalty": 1.0,
  "seed": 0,
  "temperature": 1.0,
  "top_k": 20,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · verdin-4b_en

Name
Verdin-4B-v1.1
Setup
Verdin-4B-v1.1 · decision model
Model
Verdin-4B-v1.1 · provider di-serve
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.03735 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.015 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.037. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run verdin-4b_en · first answer 2026-11-15 11:22:33 UTC · last answer 2026-11-15 11:23:27 UTC · span 54.0 s, pauses included (not charged) · charged 54.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0374 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 4B dense · source https://huggingface.co/victorbennett/Verdin-4B-v1.1 · read 2026-11-15 · the maker's figure from its name (4B) is shown, as for every model (turkce #4263); the card states 4.21B served
Input
text, English
Endpoint
http://127.0.0.1:18457
Started
15 Nov 2026 11:22 UTC; last answer 15 Nov 2026 11:23 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Verdin-4B-v1.1 (site run None); runner a75966fe5; victorbennett/Verdin-4B-v1.1@45830d216bb7e45f43f0a482c131a7cafa6d189f (CC BY-NC 4.0, non-commercial; a fine-tune of Qwen3.5-4B), served by compactor on GPU 2 of our GPU host (jevgammon's booking r_265f5494128e, lent for the index models, #8539) with the card's own server (python -m verdin.server --model .) at the card's versions (torch 2.8.0 cu128, transformers 5.17.0, fla 0.5.2, triton 3.8.0), without causal-conv1d, so its torch conv path (compactor #8576). Runner provider di-serve (the System One request at the default path, no model field). Logic & Math on the same load as Answer Check and Reading (Umit #8157, #8448)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "Verdin-4B-v1.1",
    "--provider",
    "di-serve",
    "--timeout",
    "120",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "Verdin-4B-v1.1",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "di-serve",
  "questions": 1125,
  "request_seconds": 51.23639375001949,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "36198bb211d2c4a6fa1626a02f5764548e25bc36ef9058360f370f3100fcbc12",
  "runner": "benchmark_ollama.py",
  "runner_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 426440,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:18457/v1/systemone
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · verdin-4b_en_dontknow

Name
Verdin-4B-v1.1
Setup
Verdin-4B-v1.1 · decision model · with the I don't know option F
Model
Verdin-4B-v1.1 · provider di-serve
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.04012 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0161 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.040. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run verdin-4b_en_dontknow · first answer 2026-11-15 11:25:20 UTC · last answer 2026-11-15 11:26:18 UTC · span 58.0 s, pauses included (not charged) · charged 58.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0401 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 4B dense · source https://huggingface.co/victorbennett/Verdin-4B-v1.1 · read 2026-11-15 · the maker's figure from its name (4B) is shown, as for every model (turkce #4263); the card states 4.21B served
Input
text, English
Endpoint
http://127.0.0.1:18457
Started
15 Nov 2026 11:25 UTC; last answer 15 Nov 2026 11:26 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Verdin-4B-v1.1 (site run None); runner a75966fe5; victorbennett/Verdin-4B-v1.1@45830d216bb7e45f43f0a482c131a7cafa6d189f (CC BY-NC 4.0, non-commercial; a fine-tune of Qwen3.5-4B), served by compactor on GPU 2 of our GPU host (jevgammon's booking r_265f5494128e, lent for the index models, #8539) with the card's own server (python -m verdin.server --model .) at the card's versions (torch 2.8.0 cu128, transformers 5.17.0, fla 0.5.2, triton 3.8.0), without causal-conv1d, so its torch conv path (compactor #8576). Runner provider di-serve (the System One request at the default path, no model field). Logic & Math on the same load as Answer Check and Reading (Umit #8157, #8448)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "Verdin-4B-v1.1",
    "--provider",
    "di-serve",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "Verdin-4B-v1.1",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "di-serve",
  "questions": 1125,
  "request_seconds": 55.17306871108303,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "776048ad6870724ba8bb1f9d192d7cf7bd6e841104d811482c7327bbe2490b95",
  "runner": "benchmark_ollama.py",
  "runner_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 482690,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:18457/v1/systemone
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · dexy-9b_en

Name
Dexy-Decision-9B-V2
Setup
Dexy-Decision-9B-V2 · decision model
Model
Dexy-Decision-9B-V2 · provider di-serve
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.03804 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0153 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.038. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run dexy-9b_en · first answer 2026-11-15 11:57:01 UTC · last answer 2026-11-15 11:57:56 UTC · span 55.0 s, pauses included (not charged) · charged 55.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0380 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/Patrick-18/Dexy-Decision-9B-V2 · read 2026-11-15 · the maker's figure from its name (9B) is shown, as for every model (turkce #4263); the card's table states 8.394B incl. a 0.456B vision encoder
Input
text, English
Endpoint
http://127.0.0.1:18459
Started
15 Nov 2026 11:57 UTC; last answer 15 Nov 2026 11:57 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Dexy-Decision-9B-V2 (site run None); runner a75966fe5; Patrick-18/Dexy-Decision-9B-V2@4f434e2b5fd9f0addfe3233489571c2dc8ce7a19 (Apache-2.0; built on vllm-sr/d3-flash), served by compactor on GPU 2 of our GPU host (jevgammon's booking r_265f5494128e, lent for the index models, #8539) with the card's Transformers d3_server.py at its requirements.txt pins (torch 2.14.0, transformers 5.19.0) (compactor #8697). Runner provider di-serve (the System One request at the default path, no model field; the server answers as 'Dexy-Decision-9B-V2'). Logic & Math on the same load as Answer Check and Reading (Umit #8157, #8448)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "Dexy-Decision-9B-V2",
    "--provider",
    "di-serve",
    "--timeout",
    "120",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "Dexy-Decision-9B-V2",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "di-serve",
  "questions": 1125,
  "request_seconds": 51.97015240196197,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "f8d4c035247e3b58566033cc650904dfd5ae00e3938d691c230bb9ae5c94a995",
  "runner": "benchmark_ollama.py",
  "runner_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 428507,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:18459/v1/systemone
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · dexy-9b_en_dontknow

Name
Dexy-Decision-9B-V2
Setup
Dexy-Decision-9B-V2 · decision model · with the I don't know option F
Model
Dexy-Decision-9B-V2 · provider di-serve
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.03873 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0156 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.039. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run dexy-9b_en_dontknow · first answer 2026-11-15 11:59:44 UTC · last answer 2026-11-15 12:00:40 UTC · span 56.0 s, pauses included (not charged) · charged 56.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0387 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/Patrick-18/Dexy-Decision-9B-V2 · read 2026-11-15 · the maker's figure from its name (9B) is shown, as for every model (turkce #4263); the card's table states 8.394B incl. a 0.456B vision encoder
Input
text, English
Endpoint
http://127.0.0.1:18459
Started
15 Nov 2026 11:59 UTC; last answer 15 Nov 2026 12:00 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Dexy-Decision-9B-V2 (site run None); runner a75966fe5; Patrick-18/Dexy-Decision-9B-V2@4f434e2b5fd9f0addfe3233489571c2dc8ce7a19 (Apache-2.0; built on vllm-sr/d3-flash), served by compactor on GPU 2 of our GPU host (jevgammon's booking r_265f5494128e, lent for the index models, #8539) with the card's Transformers d3_server.py at its requirements.txt pins (torch 2.14.0, transformers 5.19.0) (compactor #8697). Runner provider di-serve (the System One request at the default path, no model field; the server answers as 'Dexy-Decision-9B-V2'). Logic & Math on the same load as Answer Check and Reading (Umit #8157, #8448)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "Dexy-Decision-9B-V2",
    "--provider",
    "di-serve",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "Dexy-Decision-9B-V2",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "di-serve",
  "questions": 1125,
  "request_seconds": 52.74859690210724,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "21fcebbb2b931ca6715f903eb34ba3cec7fb1cec6042e1dd35a91476fb5f846a",
  "runner": "benchmark_ollama.py",
  "runner_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 462257,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:18459/v1/systemone
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · vera-medium-omni_en

Name
vera-medium-omni
Setup
vera-medium-omni · decision model
Model
vera-medium-omni · provider di-serve
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.02905 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0117 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.029. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run vera-medium-omni_en · first answer 2026-11-15 12:17:28 UTC · last answer 2026-11-15 12:18:10 UTC · span 42.0 s, pauses included (not charged) · charged 42.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0291 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 4.2B dense · source https://huggingface.co/scar-ai/vera-medium-omni · read 2026-11-15 · the maker's stated figure for the text backbone (~4.2B) is shown, as for every model (turkce #4263); its name has none; the vision tower (~0.33B) is not used on text questions
Input
text, English
Endpoint
http://127.0.0.1:18458
Started
15 Nov 2026 12:17 UTC; last answer 15 Nov 2026 12:18 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup vera-medium-omni (site run None); runner c6e3cdddb; scar-ai/vera-medium-omni@1f23a6cf5304e8e53b7ded53980765d5f8f4047c (Apache-2.0; derived from Qwen3.5-4B), served by compactor on GPU 2 of our GPU host (jevgammon's booking r_265f5494128e, lent for the index models, #8539) with the card's own vera-serve from vera-s1 0.1.8, PyTorch engine (compactor #8676). Runner provider di-serve (the System One request at the default path, no model field). Logic & Math on the same load as Answer Check and Reading (Umit #8157, #8448)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "vera-medium-omni",
    "--provider",
    "di-serve",
    "--timeout",
    "120",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "c6e3cdddbbc06c50393f39bfa39d2d995bc5fbc1",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "vera-medium-omni",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "di-serve",
  "questions": 1125,
  "request_seconds": 37.06825608719373,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "2fe322fe5beffd7eae253b96669e45577e7e067eb47933fcac5ac9087bb4e540",
  "runner": "benchmark_ollama.py",
  "runner_commit": "c6e3cdddbbc06c50393f39bfa39d2d995bc5fbc1",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 428507,
    "output_tokens": 38234
  }
}
Prompt template
POST http://127.0.0.1:18458/v1/systemone
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · vera-medium-omni_en_dontknow

Name
vera-medium-omni
Setup
vera-medium-omni · decision model · with the I don't know option F
Model
vera-medium-omni · provider di-serve
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.02628 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0106 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.026. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run vera-medium-omni_en_dontknow · first answer 2026-11-15 12:19:28 UTC · last answer 2026-11-15 12:20:06 UTC · span 38.0 s, pauses included (not charged) · charged 38.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0263 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 4.2B dense · source https://huggingface.co/scar-ai/vera-medium-omni · read 2026-11-15 · the maker's stated figure for the text backbone (~4.2B) is shown, as for every model (turkce #4263); its name has none; the vision tower (~0.33B) is not used on text questions
Input
text, English
Endpoint
http://127.0.0.1:18458
Started
15 Nov 2026 12:19 UTC; last answer 15 Nov 2026 12:20 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup vera-medium-omni (site run None); runner c6e3cdddb; scar-ai/vera-medium-omni@1f23a6cf5304e8e53b7ded53980765d5f8f4047c (Apache-2.0; derived from Qwen3.5-4B), served by compactor on GPU 2 of our GPU host (jevgammon's booking r_265f5494128e, lent for the index models, #8539) with the card's own vera-serve from vera-s1 0.1.8, PyTorch engine (compactor #8676). Runner provider di-serve (the System One request at the default path, no model field). Logic & Math on the same load as Answer Check and Reading (Umit #8157, #8448)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "vera-medium-omni",
    "--provider",
    "di-serve",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "c6e3cdddbbc06c50393f39bfa39d2d995bc5fbc1",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "vera-medium-omni",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "di-serve",
  "questions": 1125,
  "request_seconds": 35.2160874162073,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "2578a02be5d5e824aa53ee0cf731ffc9e8596d656efaf3a1ccc9e46b26c11588",
  "runner": "benchmark_ollama.py",
  "runner_commit": "c6e3cdddbbc06c50393f39bfa39d2d995bc5fbc1",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 462257,
    "output_tokens": 42150
  }
}
Prompt template
POST http://127.0.0.1:18458/v1/systemone
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · valen-9b_en

Name
Valen-Decision-1.5-9B
Setup
Valen-Decision-1.5-9B · decision model
Model
valen · provider valen
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.04773 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0192 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.048. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run valen-9b_en · first answer 2026-11-15 12:44:13 UTC · last answer 2026-11-15 12:45:22 UTC · span 69.0 s, pauses included (not charged) · charged 69.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0477 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/Valen-Team/Valen-Decision-1.5-9B · read 2026-11-15 · the maker's figure from its name (9B) is shown, as for every model (turkce #4263); a fine-tune of Qwen3.5-9B; this checkpoint not counted
Input
text, English
Endpoint
http://127.0.0.1:18460
Started
15 Nov 2026 12:44 UTC; last answer 15 Nov 2026 12:45 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Valen-Decision-1.5-9B (site run None); runner 88e48d7a1; Valen-Team/Valen-Decision-1.5-9B@5a512117ebc0f2b69f311becfcf47cdef8abf568 (Apache-2.0; a fine-tune of Qwen3.5-9B), served by compactor on GPU 2 of our GPU host (jevgammon's booking r_265f5494128e, lent for the index models, #8539) with the card's vLLM deployment valen-vllm (plugin Valen-Team/Valen@773d3a64, installed from a GitHub archive with --no-deps) in the official vllm/vllm-openai:v0.33.0 image, container cmp-valen-decision-1.5-9b (compactor #8773). Runner provider valen (the System One request at the default path with model field 'valen', which this server requires; turkce #8783). Logic & Math on the same load as Answer Check and Reading (Umit #8157, #8448)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "valen",
    "--provider",
    "valen",
    "--timeout",
    "120",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "valen",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "valen",
  "questions": 1125,
  "request_seconds": 66.33889861281205,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "9e468ed39c89679accdc3bca105cc0f72c62b9a82c846535b69c90544f7a95ca",
  "runner": "benchmark_ollama.py",
  "runner_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 373382,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:18460/v1/systemone
{
 "model": "valen",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · valen-9b_en_dontknow

Name
Valen-Decision-1.5-9B
Setup
Valen-Decision-1.5-9B · decision model · with the I don't know option F
Model
valen · provider valen
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.04911 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0197 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.049. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run valen-9b_en_dontknow · first answer 2026-11-15 12:47:39 UTC · last answer 2026-11-15 12:48:50 UTC · span 71.0 s, pauses included (not charged) · charged 71.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0491 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/Valen-Team/Valen-Decision-1.5-9B · read 2026-11-15 · the maker's figure from its name (9B) is shown, as for every model (turkce #4263); a fine-tune of Qwen3.5-9B; this checkpoint not counted
Input
text, English
Endpoint
http://127.0.0.1:18460
Started
15 Nov 2026 12:47 UTC; last answer 15 Nov 2026 12:48 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Valen-Decision-1.5-9B (site run None); runner 88e48d7a1; Valen-Team/Valen-Decision-1.5-9B@5a512117ebc0f2b69f311becfcf47cdef8abf568 (Apache-2.0; a fine-tune of Qwen3.5-9B), served by compactor on GPU 2 of our GPU host (jevgammon's booking r_265f5494128e, lent for the index models, #8539) with the card's vLLM deployment valen-vllm (plugin Valen-Team/Valen@773d3a64, installed from a GitHub archive with --no-deps) in the official vllm/vllm-openai:v0.33.0 image, container cmp-valen-decision-1.5-9b (compactor #8773). Runner provider valen (the System One request at the default path with model field 'valen', which this server requires; turkce #8783). Logic & Math on the same load as Answer Check and Reading (Umit #8157, #8448)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "valen",
    "--provider",
    "valen",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "valen",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "valen",
  "questions": 1125,
  "request_seconds": 68.34337317010795,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "f8fa5b726080c50ce8b76e9368ad279875d403de143c8f6788a838a0b6f2247a",
  "runner": "benchmark_ollama.py",
  "runner_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 404882,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:18460/v1/systemone
{
 "model": "valen",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · startlux-9b_en

Name
StartLux-Decision-v1.1-9B-Instant
Setup
StartLux-Decision v1.1 9B · decision model
Model
startlux-models/StartLux-Decision-v1.1-9B-Instant · provider startlux
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.01129 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0531 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.011 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run startlux-9b_en · first answer 2026-11-15 13:43:24 UTC · last answer 2026-11-15 13:46:35 UTC · span 191.0 s, pauses included (not charged) · charged 191.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0113 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/startlux-models/StartLux-Decision-v1.1-9B-Instant · read 2026-11-15 · the maker's figure from its name (9B) is shown, as for every model (turkce #4263); this checkpoint not counted
Input
text, English
Endpoint
internal address on a shared Spark host
Started
15 Nov 2026 13:43 UTC; last answer 15 Nov 2026 13:46 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup StartLux-Decision v1.1 9B (site run None); runner 88e48d7a1; startlux-models/StartLux-Decision-v1.1-9B-Instant@18a17f734966b213b2f00fa9c915e8b9af2a9b20 (weights CC BY-NC 4.0, non-commercial: run for this non-commercial benchmark; code Apache-2.0) behind the card's server, the same package and environment as StartLux 4B (github.com/StartLuxLabs/StartLux-Decision at a99977d, ~/envs/startlux: vllm==0.31.0): ~/envs/startlux/bin/startlux-decide serve --model ~/decision-models/StartLux-Decision-v1.1-9B-Instant@18a17f734966b213b2f00fa9c915e8b9af2a9b20 --launch-vllm --gpus 0 --host 0.0.0.0 --port 19420; vision on (the server's default); a shared Spark host (DGX Spark, GB10); runner provider 'startlux' (System One at /v1/systemone, model ignored). Logic & Math on the same load as Answer Check and Reading (Umit #8448, compactor #8468)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "startlux-models/StartLux-Decision-v1.1-9B-Instant",
    "--provider",
    "startlux",
    "--timeout",
    "60",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "startlux-models/StartLux-Decision-v1.1-9B-Instant",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "startlux",
  "questions": 1125,
  "request_seconds": 180.14364112273324,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "9931e64362a91c74ceafa5b7aa629169e347174099bde707487256e1c0db423c",
  "runner": "benchmark_ollama.py",
  "runner_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 394757,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19420/v1/systemone
{
 "model": "startlux-models/StartLux-Decision-v1.1-9B-Instant",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · startlux-9b_en_dontknow

Name
StartLux-Decision-v1.1-9B-Instant
Setup
StartLux-Decision v1.1 9B · decision model · with the I don't know option F
Model
startlux-models/StartLux-Decision-v1.1-9B-Instant · provider startlux
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.01147 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0539 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.011 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run startlux-9b_en_dontknow · first answer 2026-11-15 13:51:50 UTC · last answer 2026-11-15 13:55:04 UTC · span 194.0 s, pauses included (not charged) · charged 194.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0115 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/startlux-models/StartLux-Decision-v1.1-9B-Instant · read 2026-11-15 · the maker's figure from its name (9B) is shown, as for every model (turkce #4263); this checkpoint not counted
Input
text, English
Endpoint
internal address on a shared Spark host
Started
15 Nov 2026 13:51 UTC; last answer 15 Nov 2026 13:55 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup StartLux-Decision v1.1 9B (site run None); runner 88e48d7a1; startlux-models/StartLux-Decision-v1.1-9B-Instant@18a17f734966b213b2f00fa9c915e8b9af2a9b20 (weights CC BY-NC 4.0, non-commercial: run for this non-commercial benchmark; code Apache-2.0) behind the card's server, the same package and environment as StartLux 4B (github.com/StartLuxLabs/StartLux-Decision at a99977d, ~/envs/startlux: vllm==0.31.0): ~/envs/startlux/bin/startlux-decide serve --model ~/decision-models/StartLux-Decision-v1.1-9B-Instant@18a17f734966b213b2f00fa9c915e8b9af2a9b20 --launch-vllm --gpus 0 --host 0.0.0.0 --port 19420; vision on (the server's default); a shared Spark host (DGX Spark, GB10); runner provider 'startlux' (System One at /v1/systemone, model ignored). Logic & Math on the same load as Answer Check and Reading (Umit #8448, compactor #8468)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "startlux-models/StartLux-Decision-v1.1-9B-Instant",
    "--provider",
    "startlux",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "startlux-models/StartLux-Decision-v1.1-9B-Instant",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "startlux",
  "questions": 1125,
  "request_seconds": 179.4304655996093,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "2b2aaa4ccb0ec8f6ef73df2cc6b94bd74ba8e43ab532f29ac869f8f5cf7434ca",
  "runner": "benchmark_ollama.py",
  "runner_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 426257,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19420/v1/systemone
{
 "model": "startlux-models/StartLux-Decision-v1.1-9B-Instant",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · qwen3.5-2b_en_dontknow

Name
Qwen3.5-2B on · chat
Setup
Qwen3.5-2B (chat, thinking, on our GPU) · chat model · with the I don't know option F
Model
qwen3.5-2b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$4.072 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 1.64 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $4.072. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run qwen3.5-2b_en_dontknow · first answer 2026-11-15 14:15:31 UTC · last answer 2026-11-15 15:53:38 UTC · span 5887.0 s, pauses included (not charged) · charged 5887.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $4.0718 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.6 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 45 answers cut off at it with no letter (counted wrong) · 1 empty reply asked once more
Model size
Parameters: 2B dense · source https://huggingface.co/Qwen/Qwen3.5-2B · read 2026-11-13 · the card states Number of Parameters: 2B (the maker's figure is shown); checkpoint count 2.27B (2,274,069,824: language model 1.88B incl. tied embeddings, vision encoder 0.33B, MTP head 0.06B; dense)
Input
text, English
Endpoint
http://127.0.0.1:18635/v1/chat/completions
Started
15 Nov 2026 14:14 UTC; last answer 15 Nov 2026 15:53 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Qwen3.5-2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from Qwen/Qwen3.5-2B@15852e8c16360a2fea060d615a32b45270f8a8fc, vLLM reasoning parser qwen3 (vision tower loaded, for Reading's Page images; text-only prompts here; the card warns the 2B 'is more prone to entering thinking loops' in thinking mode), one RTX PRO 6000 on our GPU host alone; thinking on (enable_thinking true; the template default is off for this size; on/off only, no levels); sampling temperature 1, top_p 0.95, top_k 20, min_p 0, presence_penalty 1.5, repetition_penalty 1, sampling from the model card, generation_config has none (thinking mode, text tasks; the repo has no generation_config.json); seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 2.27B / 2.27B active / total (dense; checkpoint 2,274,069,824: language model 1.88B incl. tied embeddings, vision encoder 0.33B, MTP head 0.06B (not loaded); card: 2B). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "presence_penalty": 1.5,
  "repetition_penalty": 1.0,
  "seed": 0,
  "temperature": 1.0,
  "top_k": 20,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · raon-1-nano_en_dontknow

Name
Raon-1 nano on · chat
Setup
Raon-1 nano (chat, thinking on, KRAFTON vLLM fork, on our GPU) · chat model · with the I don't know option F
Model
raon-1-nano · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$2.179 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.875 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $2.179. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes†
GPU time
run raon-1-nano_en_dontknow · first answer 2026-11-15 16:04:46 UTC · last answer 2026-11-15 16:57:17 UTC · span 3151.0 s, pauses included (not charged) · charged 3151.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $2.1794 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)†
Requests in flight
≈ 23.6 in flight (sum of request latencies / active time)†
Output limit
32,768 tokens · 27 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 3.1B dense · 1.5B effective · source https://huggingface.co/KRAFTON/Raon-1-nano · read 2026-11-13 · the card states 1.5B effective parameters (3.1B total, including a 1.5B per-layer embedding table) (the maker's figure is shown); checkpoint count 3.13B (3,130,797,328: per-layer embedding table 1.60B, token embeddings 0.40B tied, layers 1.12B; plus a materialised copy of the tied lm_head 0.40B and the MTP module 0.05B in separate files; dense)
Input
text, English
Endpoint
http://127.0.0.1:18636/v1/chat/completions
Started
15 Nov 2026 16:04 UTC; last answer 15 Nov 2026 16:57 UTC
Progress
84 answered of 1,125 asked; 0 request errors; paused at 84 of 1,125 since 2026-11-15T16:57Z (Ümit #9163: the thinking-off seeds first); it resumes from its stored answers
Notes
Raon-1 nano, served with KRAFTON's vLLM fork krafton-ai/vllm@13a5dd87 (branch raon-1-nano, precompiled; not a vLLM release) from KRAFTON/Raon-1-nano@fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee, vLLM reasoning parser raon_1 (the card lists Korean and English only; cost and speed are on another engine than the stock-vLLM rows), one RTX PRO 6000 on our GPU host alone; thinking on (enable_thinking true, the template default; on/off only, no levels); sampling temperature 0.1, presence_penalty 1.5, sampling from the model card, generation_config has none ('General' row); seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 1.5B / 3.1B active / total (card: 1.5B effective, 3.1B total incl. a 1.5B per-layer embedding table; MTP module 55M not loaded). English translations (DeepSeek + review). PAUSED 2026-11-15T16:57:42Z (Umit #9163: the thinking-off seeds first): runner stopped on GPU 0 of our GPU host with 32 requests in flight, none stored (no drain); resumes on the same card from the stored answers at the same 32,768 settings/seed after the LFM3-1.7B seeds; the pause is not counted in GPU time.
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "truncate_history_thinking": false
  },
  "max_tokens": 32768,
  "presence_penalty": 1.5,
  "seed": 0,
  "temperature": 0.1
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · lfm3-1.7b-vllm-nothink-rtx-seed0_en

Name
LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell
Setup
LFM3-1.7B (chat, thinking off, RTX PRO 6000, on our GPU) · chat model
Model
lfm3-1.7b-vllm-nothink · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.181 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.474 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.181. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run lfm3-1.7b-vllm-nothink-rtx-seed0_en · first answer 2026-11-15 17:00:09 UTC · last answer 2026-11-15 17:28:36 UTC · span 1707.0 s, pauses included (not charged) · charged 1707.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.1807 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 28.8 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 6 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 1.73B dense · from the run's notes (dense; BF16 safetensors 1,728,801,536, text only; card: 1.7B); also https://huggingface.co/LiquidAI/LFM3-1.7B-GGUF · read 2026-11-09 · the released checkpoint's tensors without registered buffers: 1,728,801,536 (the served BF16 GGUF also stores a 64-element RoPE buffer, 1,728,801,600); text only; card: 1.7B
Input
text, English
Endpoint
http://127.0.0.1:18645/v1/chat/completions
Started
15 Nov 2026 17:00 UTC; last answer 15 Nov 2026 17:28 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
LFM3-1.7B, served with vLLM 0.34.0rc1.dev302+gbe47189f3 (nightly, commit be47189f37f6, CUDA 13.5 image; docker vllm/vllm-openai@sha256:7f0618da95bac2f1cc97fcd1791380d7c1a06d24a4998da45aeb7fbe6c49179e), BF16 from LiquidAI/LFM3-1.7B@0dc4268b561499c648c7ba7c9466bf3d8d01517a, vLLM reasoning parser qwen3 (seed 0 of three on the RTX PRO 6000 at 32 in flight (Umit #9150/#9151); the published thinking-off run (lfm3-1.7b-vllm-nothink_en) ran on a DGX Spark, so it is not one of the three), one RTX PRO 6000 on our GPU host alone; thinking off (enable_thinking false; on/off only); sampling temperature 0.1, top_k 50, repetition_penalty 1.1, top_p 1, min_p 0, from generation_config.json, passed 1:1 as vLLM sampling params; seed 0; max_tokens 32,768; one question per request (32 requests in flight), standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 1.73B / 1.73B active / total (dense; BF16 safetensors 1,728,801,536, text only; card: 1.7B). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "repetition_penalty": 1.1,
  "seed": 0,
  "temperature": 0.1,
  "top_k": 50,
  "top_p": 1.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · minicpm5-2b-nothink-seed1_en

Name
MiniCPM5-2B off · chat
Setup
MiniCPM5-2B (chat, thinking off, on our GPU) · chat model
Model
minicpm5-2b-nothink · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.478 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.594 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.478. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run minicpm5-2b-nothink-seed1_en · first answer 2026-11-15 17:00:51 UTC · last answer 2026-11-15 17:36:28 UTC · span 2137.0 s, pauses included (not charged) · charged 2137.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.4781 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 30.0 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 2 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 2.52B dense · from the run's notes (dense; checkpoint 2,516,756,480 incl. untied lm_head, text only; card: 2,516,756,480); also https://huggingface.co/openbmb/MiniCPM5-2B · read 2026-11-12 · counted from the checkpoint (2,516,756,480 incl. untied lm_head, text only; dense); the card: 2,516,756,480
Input
text, English
Endpoint
http://127.0.0.1:18640/v1/chat/completions
Started
15 Nov 2026 17:00 UTC; last answer 15 Nov 2026 17:36 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
MiniCPM5-2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from openbmb/MiniCPM5-2B@42433e90dad95123ab87729fd7011e7110629556, vLLM reasoning parser qwen3 (seed 1 of three (Umit #9150/#9151: seeds 0, 1, 2 on the RTX PRO 6000 at ~30 in flight; seed 0 is minicpm5-2b-nothink_en, which ran there at 32 sent / server 16); otherwise its settings), one RTX PRO 6000 on our GPU host alone; thinking off (enable_thinking false; on/off only, no levels); sampling temperature 1, top_p 0.95, from generation_config.json; seed 1; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 2.52B / 2.52B active / total (dense; checkpoint 2,516,756,480 incl. untied lm_head, text only; card: 2,516,756,480). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "max_tokens": 32768,
  "seed": 1,
  "temperature": 1.0,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · lfm3-1.7b-vllm-nothink-rtx-seed1_en

Name
LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell
Setup
LFM3-1.7B (chat, thinking off, RTX PRO 6000, on our GPU) · chat model
Model
lfm3-1.7b-vllm-nothink · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.13 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.454 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.130. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run lfm3-1.7b-vllm-nothink-rtx-seed1_en · first answer 2026-11-15 17:53:45 UTC · last answer 2026-11-15 18:20:59 UTC · span 1634.0 s, pauses included (not charged) · charged 1634.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.1302 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 29.1 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 7 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 1.73B dense · from the run's notes (dense; BF16 safetensors 1,728,801,536, text only; card: 1.7B); also https://huggingface.co/LiquidAI/LFM3-1.7B-GGUF · read 2026-11-09 · the released checkpoint's tensors without registered buffers: 1,728,801,536 (the served BF16 GGUF also stores a 64-element RoPE buffer, 1,728,801,600); text only; card: 1.7B
Input
text, English
Endpoint
http://127.0.0.1:18645/v1/chat/completions
Started
15 Nov 2026 17:53 UTC; last answer 15 Nov 2026 18:20 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
LFM3-1.7B, served with vLLM 0.34.0rc1.dev302+gbe47189f3 (nightly, commit be47189f37f6, CUDA 13.5 image; docker vllm/vllm-openai@sha256:7f0618da95bac2f1cc97fcd1791380d7c1a06d24a4998da45aeb7fbe6c49179e), BF16 from LiquidAI/LFM3-1.7B@0dc4268b561499c648c7ba7c9466bf3d8d01517a, vLLM reasoning parser qwen3 (seed 1 of three on the RTX PRO 6000 at 32 in flight (Umit #9150/#9151); the published thinking-off run (lfm3-1.7b-vllm-nothink_en) ran on a DGX Spark, so it is not one of the three), one RTX PRO 6000 on our GPU host alone; thinking off (enable_thinking false; on/off only); sampling temperature 0.1, top_k 50, repetition_penalty 1.1, top_p 1, min_p 0, from generation_config.json, passed 1:1 as vLLM sampling params; seed 1; max_tokens 32,768; one question per request (32 requests in flight), standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 1.73B / 1.73B active / total (dense; BF16 safetensors 1,728,801,536, text only; card: 1.7B). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "repetition_penalty": 1.1,
  "seed": 1,
  "temperature": 0.1,
  "top_k": 50,
  "top_p": 1.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · minicpm5-2b-nothink-seed2_en

Name
MiniCPM5-2B off · chat
Setup
MiniCPM5-2B (chat, thinking off, on our GPU) · chat model
Model
minicpm5-2b-nothink · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.406 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.565 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.406. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run minicpm5-2b-nothink-seed2_en · first answer 2026-11-15 18:05:24 UTC · last answer 2026-11-15 18:39:17 UTC · span 2033.0 s, pauses included (not charged) · charged 2033.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.4062 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 29.4 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 2.52B dense · from the run's notes (dense; checkpoint 2,516,756,480 incl. untied lm_head, text only; card: 2,516,756,480); also https://huggingface.co/openbmb/MiniCPM5-2B · read 2026-11-12 · counted from the checkpoint (2,516,756,480 incl. untied lm_head, text only; dense); the card: 2,516,756,480
Input
text, English
Endpoint
http://127.0.0.1:18640/v1/chat/completions
Started
15 Nov 2026 18:05 UTC; last answer 15 Nov 2026 18:39 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
MiniCPM5-2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from openbmb/MiniCPM5-2B@42433e90dad95123ab87729fd7011e7110629556, vLLM reasoning parser qwen3 (seed 2 of three (Umit #9150/#9151: seeds 0, 1, 2 on the RTX PRO 6000 at ~30 in flight; seed 0 is minicpm5-2b-nothink_en, which ran there at 32 sent / server 16); otherwise its settings), one RTX PRO 6000 on our GPU host alone; thinking off (enable_thinking false; on/off only, no levels); sampling temperature 1, top_p 0.95, from generation_config.json; seed 2; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 2.52B / 2.52B active / total (dense; checkpoint 2,516,756,480 incl. untied lm_head, text only; card: 2,516,756,480). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "max_tokens": 32768,
  "seed": 2,
  "temperature": 1.0,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · lfm3-1.7b-vllm-nothink-rtx-seed2_en

Name
LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell
Setup
LFM3-1.7B (chat, thinking off, RTX PRO 6000, on our GPU) · chat model
Model
lfm3-1.7b-vllm-nothink · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.198 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.481 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.198. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run lfm3-1.7b-vllm-nothink-rtx-seed2_en · first answer 2026-11-15 18:37:07 UTC · last answer 2026-11-15 19:05:59 UTC · span 1732.0 s, pauses included (not charged) · charged 1732.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.1980 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 29.0 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 8 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 1.73B dense · from the run's notes (dense; BF16 safetensors 1,728,801,536, text only; card: 1.7B); also https://huggingface.co/LiquidAI/LFM3-1.7B-GGUF · read 2026-11-09 · the released checkpoint's tensors without registered buffers: 1,728,801,536 (the served BF16 GGUF also stores a 64-element RoPE buffer, 1,728,801,600); text only; card: 1.7B
Input
text, English
Endpoint
http://127.0.0.1:18645/v1/chat/completions
Started
15 Nov 2026 18:37 UTC; last answer 15 Nov 2026 19:05 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
LFM3-1.7B, served with vLLM 0.34.0rc1.dev302+gbe47189f3 (nightly, commit be47189f37f6, CUDA 13.5 image; docker vllm/vllm-openai@sha256:7f0618da95bac2f1cc97fcd1791380d7c1a06d24a4998da45aeb7fbe6c49179e), BF16 from LiquidAI/LFM3-1.7B@0dc4268b561499c648c7ba7c9466bf3d8d01517a, vLLM reasoning parser qwen3 (seed 2 of three on the RTX PRO 6000 at 32 in flight (Umit #9150/#9151); the published thinking-off run (lfm3-1.7b-vllm-nothink_en) ran on a DGX Spark, so it is not one of the three), one RTX PRO 6000 on our GPU host alone; thinking off (enable_thinking false; on/off only); sampling temperature 0.1, top_k 50, repetition_penalty 1.1, top_p 1, min_p 0, from generation_config.json, passed 1:1 as vLLM sampling params; seed 2; max_tokens 32,768; one question per request (32 requests in flight), standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 1.73B / 1.73B active / total (dense; BF16 safetensors 1,728,801,536, text only; card: 1.7B). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "repetition_penalty": 1.1,
  "seed": 2,
  "temperature": 0.1,
  "top_k": 50,
  "top_p": 1.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · knowline-9b-gen2_en

Name
KnowLine-9B-Gen2
Setup
KnowLine-9B-Gen2 · decision model
Model
PelaAI/KnowLine-9B-Gen2 · provider knowline
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.04496 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0181 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.045. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run knowline-9b-gen2_en · first answer 2026-11-16 10:19:37 UTC · last answer 2026-11-16 10:20:42 UTC · span 65.0 s, pauses included (not charged) · charged 65.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0450 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/PelaAI/KnowLine-9B-Gen2 · read 2026-11-16 · the maker's figure (9B) is shown, as for every model (turkce #4263); this checkpoint not counted
Input
text, English
Endpoint
http://127.0.0.1:18461
Started
16 Nov 2026 10:19 UTC; last answer 16 Nov 2026 10:20 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup KnowLine-9B-Gen2 (site run None); runner 88e48d7a1; PelaAI/KnowLine-9B-Gen2@8ec8eb609bc10aa6050ae3754aee419d458fae42 (Apache-2.0; built on Qwen3.5-9B), served by compactor on GPU 3 of our GPU host (his booking r_d665bb34a8c2) with the card's serve_knowline.sh (SGLang, then knowline_server.py, card defaults, read twice) in the lmsysorg/sglang image with SGLang 0.5.23, as KnowLine-9B-Gen1; the card names SGLang 0.5.21 (compactor #9859). Logic & Math on the same load as Answer Check and Reading (Umit #9814)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "PelaAI/KnowLine-9B-Gen2",
    "--provider",
    "knowline",
    "--timeout",
    "120",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "PelaAI/KnowLine-9B-Gen2",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "knowline",
  "questions": 1125,
  "request_seconds": 59.4028170343081,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "195c15046b41109bc6346ef70bb7f6af1d5ac6211d45cc76d849b0b2e09c86d3",
  "runner": "benchmark_ollama.py",
  "runner_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 836764,
    "output_tokens": 1125
  }
}
Prompt template
POST http://127.0.0.1:18461/v1/systemone
{
 "model": "PelaAI/KnowLine-9B-Gen2",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · knowline-9b-gen2_en_dontknow

Name
KnowLine-9B-Gen2
Setup
KnowLine-9B-Gen2 · decision model · with the I don't know option F
Model
PelaAI/KnowLine-9B-Gen2 · provider knowline
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.04496 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0181 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.045. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run knowline-9b-gen2_en_dontknow · first answer 2026-11-16 10:22:29 UTC · last answer 2026-11-16 10:23:34 UTC · span 65.0 s, pauses included (not charged) · charged 65.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0450 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/PelaAI/KnowLine-9B-Gen2 · read 2026-11-16 · the maker's figure (9B) is shown, as for every model (turkce #4263); this checkpoint not counted
Input
text, English
Endpoint
http://127.0.0.1:18461
Started
16 Nov 2026 10:22 UTC; last answer 16 Nov 2026 10:23 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup KnowLine-9B-Gen2 (site run None); runner 88e48d7a1; PelaAI/KnowLine-9B-Gen2@8ec8eb609bc10aa6050ae3754aee419d458fae42 (Apache-2.0; built on Qwen3.5-9B), served by compactor on GPU 3 of our GPU host (his booking r_d665bb34a8c2) with the card's serve_knowline.sh (SGLang, then knowline_server.py, card defaults, read twice) in the lmsysorg/sglang image with SGLang 0.5.23, as KnowLine-9B-Gen1; the card names SGLang 0.5.21 (compactor #9859). Logic & Math on the same load as Answer Check and Reading (Umit #9814)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "PelaAI/KnowLine-9B-Gen2",
    "--provider",
    "knowline",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "PelaAI/KnowLine-9B-Gen2",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "knowline",
  "questions": 1125,
  "request_seconds": 60.67173783047474,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "29873c5c12c2695860bf716883856856e3268a38424f15261eea3b19cd8f3709",
  "runner": "benchmark_ollama.py",
  "runner_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 904264,
    "output_tokens": 1125
  }
}
Prompt template
POST http://127.0.0.1:18461/v1/systemone
{
 "model": "PelaAI/KnowLine-9B-Gen2",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · knowline-12b-gen1_en

Name
KnowLine-12B-Gen1
Setup
KnowLine-12B-Gen1 · decision model
Model
PelaAI/KnowLine-12B-Gen1 · provider knowline
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.0581 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0233 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.058. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run knowline-12b-gen1_en · first answer 2026-11-16 10:22:35 UTC · last answer 2026-11-16 10:23:59 UTC · span 84.0 s, pauses included (not charged) · charged 84.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0581 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 12B dense · source https://huggingface.co/PelaAI/KnowLine-12B-Gen1 · read 2026-11-16 · the maker's figure (12B) is shown, as for every model (turkce #4263); this checkpoint not counted
Input
text, English
Endpoint
http://127.0.0.1:18462
Started
16 Nov 2026 10:22 UTC; last answer 16 Nov 2026 10:23 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup KnowLine-12B-Gen1 (site run None); runner 88e48d7a1; PelaAI/KnowLine-12B-Gen1@f26fbc2d64092bb2f8caaedbec243172bb3e908f (Apache-2.0; built on Gemma 4 12B), served by compactor on GPU 2 of our GPU host (his booking r_298d21956009) with the card's serve_knowline.sh (SGLang, then knowline_server.py, card defaults, read twice) in the lmsysorg/sglang image with SGLang 0.5.23; the card names SGLang 0.5.21 (compactor #9859). Logic & Math on the same load as Answer Check and Reading (Umit #9814)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "PelaAI/KnowLine-12B-Gen1",
    "--provider",
    "knowline",
    "--timeout",
    "120",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "PelaAI/KnowLine-12B-Gen1",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "knowline",
  "questions": 1125,
  "request_seconds": 78.75357625729521,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "bd3c81d1d85b872a1781d8855541c45b9174b36167744d25cc525a4b69b91bbd",
  "runner": "benchmark_ollama.py",
  "runner_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 836143,
    "output_tokens": 1125
  }
}
Prompt template
POST http://127.0.0.1:18462/v1/systemone
{
 "model": "PelaAI/KnowLine-12B-Gen1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · knowline-12b-gen1_en_dontknow

Name
KnowLine-12B-Gen1
Setup
KnowLine-12B-Gen1 · decision model · with the I don't know option F
Model
PelaAI/KnowLine-12B-Gen1 · provider knowline
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.06018 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0242 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.060. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run knowline-12b-gen1_en_dontknow · first answer 2026-11-16 10:25:59 UTC · last answer 2026-11-16 10:27:26 UTC · span 87.0 s, pauses included (not charged) · charged 87.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0602 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 12B dense · source https://huggingface.co/PelaAI/KnowLine-12B-Gen1 · read 2026-11-16 · the maker's figure (12B) is shown, as for every model (turkce #4263); this checkpoint not counted
Input
text, English
Endpoint
http://127.0.0.1:18462
Started
16 Nov 2026 10:25 UTC; last answer 16 Nov 2026 10:27 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup KnowLine-12B-Gen1 (site run None); runner 88e48d7a1; PelaAI/KnowLine-12B-Gen1@f26fbc2d64092bb2f8caaedbec243172bb3e908f (Apache-2.0; built on Gemma 4 12B), served by compactor on GPU 2 of our GPU host (his booking r_298d21956009) with the card's serve_knowline.sh (SGLang, then knowline_server.py, card defaults, read twice) in the lmsysorg/sglang image with SGLang 0.5.23; the card names SGLang 0.5.21 (compactor #9859). Logic & Math on the same load as Answer Check and Reading (Umit #9814)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "PelaAI/KnowLine-12B-Gen1",
    "--provider",
    "knowline",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "PelaAI/KnowLine-12B-Gen1",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "knowline",
  "questions": 1125,
  "request_seconds": 83.05571010589483,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "9b96d457e417038befcfaac5d4f46e14d5dcc698237d194fce82088a65705532",
  "runner": "benchmark_ollama.py",
  "runner_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 910393,
    "output_tokens": 1125
  }
}
Prompt template
POST http://127.0.0.1:18462/v1/systemone
{
 "model": "PelaAI/KnowLine-12B-Gen1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · knowline-31b-gen2_en

Name
KnowLine-31B-Gen2
Setup
KnowLine-31B-Gen2 · decision model
Model
PelaAI/KnowLine-31B-Gen2 · provider knowline
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.1259 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0506 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.126. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run knowline-31b-gen2_en · first answer 2026-11-16 10:40:17 UTC · last answer 2026-11-16 10:43:19 UTC · span 182.0 s, pauses included (not charged) · charged 182.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1259 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 31B dense · source https://huggingface.co/PelaAI/KnowLine-31B-Gen2 · read 2026-11-16 · the maker's figure (31B) is shown, as for every model (turkce #4263); this checkpoint not counted
Input
text, English
Endpoint
http://127.0.0.1:18463
Started
16 Nov 2026 10:40 UTC; last answer 16 Nov 2026 10:43 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup KnowLine-31B-Gen2 (site run None); runner 88e48d7a1; PelaAI/KnowLine-31B-Gen2@fd99a9ed2255d8a67dd47a44cc429e638f611593 (Apache-2.0; built on Gemma 4 31B), served by compactor on GPU 1 of our GPU host with the card's serve_knowline.sh (SGLang, then knowline_server.py, card defaults, read twice) in the lmsysorg/sglang image with SGLang 0.5.23; the card names SGLang 0.5.21 (compactor #9859). Logic & Math on the same load as Answer Check and Reading (Umit #9814)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "PelaAI/KnowLine-31B-Gen2",
    "--provider",
    "knowline",
    "--timeout",
    "120",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "PelaAI/KnowLine-31B-Gen2",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "knowline",
  "questions": 1125,
  "request_seconds": 179.33634044381324,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "183962ed7c91118b06e94e496c7c9b29f8c01e9c9663c5b95f742fd76e50912a",
  "runner": "benchmark_ollama.py",
  "runner_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 836143,
    "output_tokens": 1125
  }
}
Prompt template
POST http://127.0.0.1:18463/v1/systemone
{
 "model": "PelaAI/KnowLine-31B-Gen2",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · knowline-31b-gen2_en_dontknow

Name
KnowLine-31B-Gen2
Setup
KnowLine-31B-Gen2 · decision model · with the I don't know option F
Model
PelaAI/KnowLine-31B-Gen2 · provider knowline
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.1404 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0564 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.140. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run knowline-31b-gen2_en_dontknow · first answer 2026-11-16 10:47:53 UTC · last answer 2026-11-16 10:51:16 UTC · span 203.0 s, pauses included (not charged) · charged 203.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1404 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 31B dense · source https://huggingface.co/PelaAI/KnowLine-31B-Gen2 · read 2026-11-16 · the maker's figure (31B) is shown, as for every model (turkce #4263); this checkpoint not counted
Input
text, English
Endpoint
http://127.0.0.1:18463
Started
16 Nov 2026 10:47 UTC; last answer 16 Nov 2026 10:51 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup KnowLine-31B-Gen2 (site run None); runner 88e48d7a1; PelaAI/KnowLine-31B-Gen2@fd99a9ed2255d8a67dd47a44cc429e638f611593 (Apache-2.0; built on Gemma 4 31B), served by compactor on GPU 1 of our GPU host with the card's serve_knowline.sh (SGLang, then knowline_server.py, card defaults, read twice) in the lmsysorg/sglang image with SGLang 0.5.23; the card names SGLang 0.5.21 (compactor #9859). Logic & Math on the same load as Answer Check and Reading (Umit #9814)
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "PelaAI/KnowLine-31B-Gen2",
    "--provider",
    "knowline",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "PelaAI/KnowLine-31B-Gen2",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "knowline",
  "questions": 1125,
  "request_seconds": 195.57054590291227,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "e7b7c5eb00cc010b145f91b0f0532b9bd65486de3e248e4d3938ccfb8e42e15b",
  "runner": "benchmark_ollama.py",
  "runner_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 910393,
    "output_tokens": 1125
  }
}
Prompt template
POST http://127.0.0.1:18463/v1/systemone
{
 "model": "PelaAI/KnowLine-31B-Gen2",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · inkling-2-small_en

Name
Inkling 2 Small high · chat
Setup
Inkling 2 Small · chat model
Model
thinkingmachines/inkling-2-small · provider openrouter
GPU
—
Cost
$4.637 for the run · provider price × reported usage
Answer time
run inkling-2-small_en · first answer 2026-11-16 15:27:41 UTC · last answer 2026-11-16 17:17:53 UTC · span 6612.0 s, pauses included (not charged) · charged 6612.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 7.7 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 14 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 594B MoE · 17B active · from the run's notes (Thinking Machines (Hugging Face card)); also https://huggingface.co/thinkingmachines/Inkling-2-Small · read 2026-11-16 · the card at 2ad1157016e1aac77b6ff1b770e7c05229403134: '594B total, 17B active'
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
16 Nov 2026 15:27 UTC; last answer 16 Nov 2026 17:17 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Inkling 2 Small through OpenRouter (thinkingmachines/inkling-2-small), provider pinned to nebius/fp4, fallbacks off; reasoning {"effort": "high"}; temperature 1.0; max_tokens 32768; one question per request; parameters 17B / 594B active / total (Thinking Machines (Hugging Face card)). English translations.
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "nebius/fp4"
    ]
  },
  "reasoning": {
    "effort": "high"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · qwen4-max_en

Name
Qwen4-Max high · chat
Setup
Qwen4-Max · chat model
Model
qwen/qwen4-max · provider openrouter
GPU
—
Cost
$6.401 for the run · provider price × reported usage
Answer time
run qwen4-max_en · first answer 2026-11-16 15:27:34 UTC · last answer 2026-11-16 16:39:50 UTC · span 4336.0 s, pauses included (not charged) · charged 4336.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 6.4 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 1 answer cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: not disclosed by maker (closed weights) · source https://openrouter.ai/qwen/qwen4-max · read 2026-11-16
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
16 Nov 2026 15:27 UTC; last answer 16 Nov 2026 16:39 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Qwen4-Max through OpenRouter (qwen/qwen4-max), provider pinned to alibaba, fallbacks off; reasoning {"effort": "high"}; temperature 1.0; max_tokens 32768; one question per request; parameters not disclosed (closed weights). English translations.
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "alibaba"
    ]
  },
  "reasoning": {
    "effort": "high"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · gemma-4-e2b-it-think-32k-sampled-seed0_en

Name
Gemma 4 E2B on · chat · 32k cap · sampled
Setup
Gemma 4 E2B (chat, thinking on, 32k, sampled, on our GPU) · chat model
Model
gemma-4-e2b-it-think-32k-sampled-seed0 · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.6426 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.258 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.643. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gemma-4-e2b-it-think-32k-sampled-seed0_en · first answer 2026-11-16 15:49:35 UTC · last answer 2026-11-16 16:05:04 UTC · span 929.0 s, pauses included (not charged) · charged 929.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.6426 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 28.9 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 5.1B dense · 2.3B effective · source https://huggingface.co/google/gemma-4-E2B-it · read 2026-11-10 · the card states 2.3B effective (5.1B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 5.12B (counted from the checkpoint (5,123,178,979: language model 4.65B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 5.1B with embeddings, 2.3B effective; not a mixture of experts)
Input
text, English
Endpoint
http://127.0.0.1:18670/v1/chat/completions
Started
16 Nov 2026 15:49 UTC; last answer 16 Nov 2026 16:05 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Gemma 4 E2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from google/gemma-4-E2B-it@4adf15748de9e1a0097b6274cee9c15836c41c37, vLLM reasoning parser gemma4 (seed 0 of three at the maker's sampling (Umit #10533, logic #10541); the published temperature-0 run is not one of them), one RTX PRO 6000 on our GPU host alone; thinking on (chat_template_kwargs enable_thinking true, the template's own switch: it opens a system turn with <|think|>; the template default is off; on/off only, no levels); sampling temperature 1, top_p 0.95, top_k 64, from generation_config.json (temperature 1.0, top_p 0.95, top_k 64); seed 0; max_tokens 32,768; one question per request (32 requests in flight), standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 5.12B / 5.12B active / total (dense; checkpoint 5,123,178,979 incl. vision 0.17B and audio 0.31B towers; card: 5.1B with per-layer embeddings, 2.3B effective). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "seed": 0,
  "temperature": 1.0,
  "top_k": 64,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · gemma-4-e4b-it-think-32k-sampled-seed0_en

Name
Gemma 4 E4B on · chat · 32k cap · sampled
Setup
Gemma 4 E4B (chat, thinking on, 32k, sampled, on our GPU) · chat model
Model
gemma-4-e4b-it-think-32k-sampled-seed0 · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.091 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.438 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.091. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gemma-4-e4b-it-think-32k-sampled-seed0_en · first answer 2026-11-16 16:04:16 UTC · last answer 2026-11-16 16:30:34 UTC · span 1578.0 s, pauses included (not charged) · charged 1578.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.0915 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 28.2 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 8B dense · 4.5B effective · source https://huggingface.co/google/gemma-4-E4B-it · read 2026-11-10 · the card states 4.5B effective (8B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 8.00B (counted from the checkpoint (7,996,157,418: language model 7.52B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 8B with embeddings, 4.5B effective; not a mixture of experts)
Input
text, English
Endpoint
http://127.0.0.1:18671/v1/chat/completions
Started
16 Nov 2026 16:04 UTC; last answer 16 Nov 2026 16:30 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Gemma 4 E4B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from google/gemma-4-E4B-it@4ecc2ba63bcac23ba4a767fc87025b1450f84c9c, vLLM reasoning parser gemma4 (seed 0 of three at the maker's sampling (Umit #10533, logic #10541); the published temperature-0 run is not one of them), one RTX PRO 6000 on our GPU host alone; thinking on (chat_template_kwargs enable_thinking true, the template's own switch: it opens a system turn with <|think|>; the template default is off; on/off only, no levels); sampling temperature 1, top_p 0.95, top_k 64, from generation_config.json (temperature 1.0, top_p 0.95, top_k 64); seed 0; max_tokens 32,768; one question per request (32 requests in flight), standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 8.00B / 8.00B active / total (dense; checkpoint 7,996,157,418 incl. vision 0.17B and audio 0.31B towers; card: 8B with per-layer embeddings, 4.5B effective). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "seed": 0,
  "temperature": 1.0,
  "top_k": 64,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · gemma-4-e2b-it-think-32k-sampled-seed0_en_dontknow

Name
Gemma 4 E2B on · chat · 32k cap · sampled
Setup
Gemma 4 E2B (chat, thinking on, 32k, sampled, on our GPU) · chat model · with the I don't know option F
Model
gemma-4-e2b-it-think-32k-sampled-seed0 · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.5831 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.234 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.583. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gemma-4-e2b-it-think-32k-sampled-seed0_en_dontknow · first answer 2026-11-16 16:20:57 UTC · last answer 2026-11-16 16:35:00 UTC · span 843.0 s, pauses included (not charged) · charged 843.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.5831 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 30.9 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 5.1B dense · 2.3B effective · source https://huggingface.co/google/gemma-4-E2B-it · read 2026-11-10 · the card states 2.3B effective (5.1B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 5.12B (counted from the checkpoint (5,123,178,979: language model 4.65B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 5.1B with embeddings, 2.3B effective; not a mixture of experts)
Input
text, English
Endpoint
http://127.0.0.1:18670/v1/chat/completions
Started
16 Nov 2026 16:20 UTC; last answer 16 Nov 2026 16:35 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Gemma 4 E2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from google/gemma-4-E2B-it@4adf15748de9e1a0097b6274cee9c15836c41c37, vLLM reasoning parser gemma4 (seed 0 of three at the maker's sampling (Umit #10533, logic #10541); the published temperature-0 run is not one of them), one RTX PRO 6000 on our GPU host alone; thinking on (chat_template_kwargs enable_thinking true, the template's own switch: it opens a system turn with <|think|>; the template default is off; on/off only, no levels); sampling temperature 1, top_p 0.95, top_k 64, from generation_config.json (temperature 1.0, top_p 0.95, top_k 64); seed 0; max_tokens 32,768; one question per request (32 requests in flight), standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 5.12B / 5.12B active / total (dense; checkpoint 5,123,178,979 incl. vision 0.17B and audio 0.31B towers; card: 5.1B with per-layer embeddings, 2.3B effective). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "seed": 0,
  "temperature": 1.0,
  "top_k": 64,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · qwen4-max_en_dontknow

Name
Qwen4-Max high · chat
Setup
Qwen4-Max · chat model · with the I don't know option F
Model
qwen/qwen4-max · provider openrouter
GPU
—
Cost
$6.286 for the run · provider price × reported usage
Answer time
run qwen4-max_en_dontknow · first answer 2026-11-16 16:39:58 UTC · last answer 2026-11-16 17:36:22 UTC · span 3384.0 s, pauses included (not charged) · charged 3384.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 7.5 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 2 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: not disclosed by maker (closed weights) · source https://openrouter.ai/qwen/qwen4-max · read 2026-11-16
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
16 Nov 2026 16:39 UTC; last answer 16 Nov 2026 17:36 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Qwen4-Max through OpenRouter (qwen/qwen4-max), provider pinned to alibaba, fallbacks off; reasoning {"effort": "high"}; temperature 1.0; max_tokens 32768; one question per request; parameters not disclosed (closed weights). English translations.
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "alibaba"
    ]
  },
  "reasoning": {
    "effort": "high"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · gemma-4-e2b-it-think-32k-sampled-seed1_en

Name
Gemma 4 E2B on · chat · 32k cap · sampled
Setup
Gemma 4 E2B (chat, thinking on, 32k, sampled, on our GPU) · chat model
Model
gemma-4-e2b-it-think-32k-sampled-seed1 · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.6557 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.263 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.656. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gemma-4-e2b-it-think-32k-sampled-seed1_en · first answer 2026-11-16 16:58:46 UTC · last answer 2026-11-16 17:14:34 UTC · span 948.0 s, pauses included (not charged) · charged 948.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.6557 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 28.7 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 1 answer cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 5.1B dense · 2.3B effective · source https://huggingface.co/google/gemma-4-E2B-it · read 2026-11-10 · the card states 2.3B effective (5.1B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 5.12B (counted from the checkpoint (5,123,178,979: language model 4.65B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 5.1B with embeddings, 2.3B effective; not a mixture of experts)
Input
text, English
Endpoint
http://127.0.0.1:18670/v1/chat/completions
Started
16 Nov 2026 16:58 UTC; last answer 16 Nov 2026 17:14 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Gemma 4 E2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from google/gemma-4-E2B-it@4adf15748de9e1a0097b6274cee9c15836c41c37, vLLM reasoning parser gemma4 (seed 1 of three at the maker's sampling (Umit #10533, logic #10541); the published temperature-0 run is not one of them), one RTX PRO 6000 on our GPU host alone; thinking on (chat_template_kwargs enable_thinking true, the template's own switch: it opens a system turn with <|think|>; the template default is off; on/off only, no levels); sampling temperature 1, top_p 0.95, top_k 64, from generation_config.json (temperature 1.0, top_p 0.95, top_k 64); seed 1; max_tokens 32,768; one question per request (32 requests in flight), standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 5.12B / 5.12B active / total (dense; checkpoint 5,123,178,979 incl. vision 0.17B and audio 0.31B towers; card: 5.1B with per-layer embeddings, 2.3B effective). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "seed": 1,
  "temperature": 1.0,
  "top_k": 64,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · gemma-4-e4b-it-think-32k-sampled-seed0_en_dontknow

Name
Gemma 4 E4B on · chat · 32k cap · sampled
Setup
Gemma 4 E4B (chat, thinking on, 32k, sampled, on our GPU) · chat model · with the I don't know option F
Model
gemma-4-e4b-it-think-32k-sampled-seed0 · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.9732 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.391 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.973. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gemma-4-e4b-it-think-32k-sampled-seed0_en_dontknow · first answer 2026-11-16 17:01:33 UTC · last answer 2026-11-16 17:25:00 UTC · span 1407.0 s, pauses included (not charged) · charged 1407.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.9732 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.4 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 8B dense · 4.5B effective · source https://huggingface.co/google/gemma-4-E4B-it · read 2026-11-10 · the card states 4.5B effective (8B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 8.00B (counted from the checkpoint (7,996,157,418: language model 7.52B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 8B with embeddings, 4.5B effective; not a mixture of experts)
Input
text, English
Endpoint
http://127.0.0.1:18671/v1/chat/completions
Started
16 Nov 2026 17:01 UTC; last answer 16 Nov 2026 17:25 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Gemma 4 E4B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from google/gemma-4-E4B-it@4ecc2ba63bcac23ba4a767fc87025b1450f84c9c, vLLM reasoning parser gemma4 (seed 0 of three at the maker's sampling (Umit #10533, logic #10541); the published temperature-0 run is not one of them), one RTX PRO 6000 on our GPU host alone; thinking on (chat_template_kwargs enable_thinking true, the template's own switch: it opens a system turn with <|think|>; the template default is off; on/off only, no levels); sampling temperature 1, top_p 0.95, top_k 64, from generation_config.json (temperature 1.0, top_p 0.95, top_k 64); seed 0; max_tokens 32,768; one question per request (32 requests in flight), standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 8.00B / 8.00B active / total (dense; checkpoint 7,996,157,418 incl. vision 0.17B and audio 0.31B towers; card: 8B with per-layer embeddings, 4.5B effective). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "seed": 0,
  "temperature": 1.0,
  "top_k": 64,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · inkling-2-small_en_dontknow

Name
Inkling 2 Small high · chat
Setup
Inkling 2 Small · chat model · with the I don't know option F
Model
thinkingmachines/inkling-2-small · provider openrouter
GPU
—
Cost
$4.411 for the run · provider price × reported usage
Answer time
run inkling-2-small_en_dontknow · first answer 2026-11-16 17:17:59 UTC · last answer 2026-11-16 18:24:43 UTC · span 4004.0 s, pauses included (not charged) · charged 4004.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 7.9 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 13 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 594B MoE · 17B active · from the run's notes (Thinking Machines (Hugging Face card)); also https://huggingface.co/thinkingmachines/Inkling-2-Small · read 2026-11-16 · the card at 2ad1157016e1aac77b6ff1b770e7c05229403134: '594B total, 17B active'
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
16 Nov 2026 17:17 UTC; last answer 16 Nov 2026 18:24 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Inkling 2 Small through OpenRouter (thinkingmachines/inkling-2-small), provider pinned to nebius/fp4, fallbacks off; reasoning {"effort": "high"}; temperature 1.0; max_tokens 32768; one question per request; parameters 17B / 594B active / total (Thinking Machines (Hugging Face card)). English translations.
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "nebius/fp4"
    ]
  },
  "reasoning": {
    "effort": "high"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · gemma-4-e2b-it-think-32k-sampled-seed2_en

Name
Gemma 4 E2B on · chat · 32k cap · sampled
Setup
Gemma 4 E2B (chat, thinking on, 32k, sampled, on our GPU) · chat model
Model
gemma-4-e2b-it-think-32k-sampled-seed2 · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.6391 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.257 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.639. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gemma-4-e2b-it-think-32k-sampled-seed2_en · first answer 2026-11-16 17:36:04 UTC · last answer 2026-11-16 17:51:28 UTC · span 924.0 s, pauses included (not charged) · charged 924.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.6391 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 28.6 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 5.1B dense · 2.3B effective · source https://huggingface.co/google/gemma-4-E2B-it · read 2026-11-10 · the card states 2.3B effective (5.1B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 5.12B (counted from the checkpoint (5,123,178,979: language model 4.65B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 5.1B with embeddings, 2.3B effective; not a mixture of experts)
Input
text, English
Endpoint
http://127.0.0.1:18670/v1/chat/completions
Started
16 Nov 2026 17:36 UTC; last answer 16 Nov 2026 17:51 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Gemma 4 E2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from google/gemma-4-E2B-it@4adf15748de9e1a0097b6274cee9c15836c41c37, vLLM reasoning parser gemma4 (seed 2 of three at the maker's sampling (Umit #10533, logic #10541); the published temperature-0 run is not one of them), one RTX PRO 6000 on our GPU host alone; thinking on (chat_template_kwargs enable_thinking true, the template's own switch: it opens a system turn with <|think|>; the template default is off; on/off only, no levels); sampling temperature 1, top_p 0.95, top_k 64, from generation_config.json (temperature 1.0, top_p 0.95, top_k 64); seed 2; max_tokens 32,768; one question per request (32 requests in flight), standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 5.12B / 5.12B active / total (dense; checkpoint 5,123,178,979 incl. vision 0.17B and audio 0.31B towers; card: 5.1B with per-layer embeddings, 2.3B effective). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "seed": 2,
  "temperature": 1.0,
  "top_k": 64,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · gemma-4-e4b-it-think-32k-sampled-seed1_en

Name
Gemma 4 E4B on · chat · 32k cap · sampled
Setup
Gemma 4 E4B (chat, thinking on, 32k, sampled, on our GPU) · chat model
Model
gemma-4-e4b-it-think-32k-sampled-seed1 · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.091 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.438 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.091. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gemma-4-e4b-it-think-32k-sampled-seed1_en · first answer 2026-11-16 17:50:31 UTC · last answer 2026-11-16 18:16:49 UTC · span 1578.0 s, pauses included (not charged) · charged 1578.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.0915 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 28.5 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 8B dense · 4.5B effective · source https://huggingface.co/google/gemma-4-E4B-it · read 2026-11-10 · the card states 4.5B effective (8B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 8.00B (counted from the checkpoint (7,996,157,418: language model 7.52B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 8B with embeddings, 4.5B effective; not a mixture of experts)
Input
text, English
Endpoint
http://127.0.0.1:18671/v1/chat/completions
Started
16 Nov 2026 17:50 UTC; last answer 16 Nov 2026 18:16 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Gemma 4 E4B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from google/gemma-4-E4B-it@4ecc2ba63bcac23ba4a767fc87025b1450f84c9c, vLLM reasoning parser gemma4 (seed 1 of three at the maker's sampling (Umit #10533, logic #10541); the published temperature-0 run is not one of them), one RTX PRO 6000 on our GPU host alone; thinking on (chat_template_kwargs enable_thinking true, the template's own switch: it opens a system turn with <|think|>; the template default is off; on/off only, no levels); sampling temperature 1, top_p 0.95, top_k 64, from generation_config.json (temperature 1.0, top_p 0.95, top_k 64); seed 1; max_tokens 32,768; one question per request (32 requests in flight), standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 8.00B / 8.00B active / total (dense; checkpoint 7,996,157,418 incl. vision 0.17B and audio 0.31B towers; card: 8B with per-layer embeddings, 4.5B effective). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "seed": 1,
  "temperature": 1.0,
  "top_k": 64,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · raon-1-nano-nothink_en_dontknow

Name
Raon-1 nano off · chat
Setup
Raon-1 nano (chat, thinking off, KRAFTON vLLM fork, on our GPU) · chat model · with the I don't know option F
Model
raon-1-nano-nothink · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$7.525 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 3.02 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $7.525. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes†
GPU time
run raon-1-nano-nothink_en_dontknow · first answer 2026-11-16 17:59:34 UTC · last answer 2026-11-16 21:00:53 UTC · span 10879.0 s, pauses included (not charged) · charged 10879.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $7.5246 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)†
Requests in flight
≈ 30.2 in flight (sum of request latencies / active time)†
Output limit
32,768 tokens · 62 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 3.1B dense · 1.5B effective · source https://huggingface.co/KRAFTON/Raon-1-nano · read 2026-11-13 · the card states 1.5B effective parameters (3.1B total, including a 1.5B per-layer embedding table) (the maker's figure is shown); checkpoint count 3.13B (3,130,797,328: per-layer embedding table 1.60B, token embeddings 0.40B tied, layers 1.12B; plus a materialised copy of the tied lm_head 0.40B and the MTP module 0.05B in separate files; dense)
Input
text, English
Endpoint
http://127.0.0.1:18637/v1/chat/completions
Started
16 Nov 2026 17:59 UTC; last answer 16 Nov 2026 21:00 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Raon-1 nano, served with KRAFTON's vLLM fork krafton-ai/vllm@13a5dd87 (branch raon-1-nano, precompiled; not a vLLM release) from KRAFTON/Raon-1-nano@fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee, vLLM reasoning parser raon_1 (the card lists Korean and English only; cost and speed are on another engine than the stock-vLLM rows), one RTX PRO 6000 on our GPU host alone; thinking off (enable_thinking false; the card: 'designed for thinking mode; disabling it is not recommended'); sampling temperature 0.1, presence_penalty 1.5, sampling from the model card, generation_config has none ('General' row); seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 1.5B / 3.1B active / total (card: 1.5B effective, 3.1B total incl. a 1.5B per-layer embedding table; MTP module 55M not loaded). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "truncate_history_thinking": false
  },
  "max_tokens": 32768,
  "presence_penalty": 1.5,
  "seed": 0,
  "temperature": 0.1
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Logic · gemma-4-e4b-it-think-32k-sampled-seed2_en

Name
Gemma 4 E4B on · chat · 32k cap · sampled
Setup
Gemma 4 E4B (chat, thinking on, 32k, sampled, on our GPU) · chat model
Model
gemma-4-e4b-it-think-32k-sampled-seed2 · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.167 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.469 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.167. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gemma-4-e4b-it-think-32k-sampled-seed2_en · first answer 2026-11-16 18:44:28 UTC · last answer 2026-11-16 19:12:35 UTC · span 1687.0 s, pauses included (not charged) · charged 1687.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.1668 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 27.8 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 8B dense · 4.5B effective · source https://huggingface.co/google/gemma-4-E4B-it · read 2026-11-10 · the card states 4.5B effective (8B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 8.00B (counted from the checkpoint (7,996,157,418: language model 7.52B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 8B with embeddings, 4.5B effective; not a mixture of experts)
Input
text, English
Endpoint
http://127.0.0.1:18671/v1/chat/completions
Started
16 Nov 2026 18:44 UTC; last answer 16 Nov 2026 19:12 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Gemma 4 E4B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from google/gemma-4-E4B-it@4ecc2ba63bcac23ba4a767fc87025b1450f84c9c, vLLM reasoning parser gemma4 (seed 2 of three at the maker's sampling (Umit #10533, logic #10541); the published temperature-0 run is not one of them), one RTX PRO 6000 on our GPU host alone; thinking on (chat_template_kwargs enable_thinking true, the template's own switch: it opens a system turn with <|think|>; the template default is off; on/off only, no levels); sampling temperature 1, top_p 0.95, top_k 64, from generation_config.json (temperature 1.0, top_p 0.95, top_k 64); seed 2; max_tokens 32,768; one question per request (32 requests in flight), standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 8.00B / 8.00B active / total (dense; checkpoint 7,996,157,418 incl. vision 0.17B and audio 0.31B towers; card: 8B with per-layer embeddings, 4.5B effective). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "seed": 2,
  "temperature": 1.0,
  "top_k": 64,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · d1.2-ultra_en

Name
d1.2-ultra
Setup
Liquid d1.2-ultra · decision model
Model
d1.2-ultra · provider liquid
GPU
—
Cost
$0.01422 for the run · provider price × reported usage
Answer time
run d1.2-ultra_en · first answer 2026-11-16 18:58:20 UTC · last answer 2026-11-16 19:03:09 UTC · span 289.0 s, pauses included (not charged) · charged 289.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://www.liquid.ai/blog/d1-2 · read 2026-11-16
Input
text, English
Endpoint
https://api.liquid.ai/decisions
Started
16 Nov 2026 18:58 UTC; last answer 16 Nov 2026 19:03 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Liquid d1.2-ultra (site run None); runner 1cc2ce5d9; Liquid's hosted decision API, released 16 November 2026 (liquid.ai/blog/d1-2); run at once on Umit's word, whatever the cost (#10926, zeki #10928), one request at a time as d1-ultra ran
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "d1.2-ultra",
    "--provider",
    "liquid",
    "--timeout",
    "30",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "0f7123a347725d64b71f99d55f89d66f32fa9eb0",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "d1.2-ultra",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Liquid AI's d1.2-ultra price",
    "date": "2026-11-16",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.042,
      "cache_write_input_tokens": 0.042,
      "cached_input_tokens": 0.042,
      "input_tokens": 0.042,
      "output_tokens": 0.0
    },
    "source": "https://www.liquid.ai/blog/d1-2",
    "text": "$0.042 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "liquid",
  "questions": 1125,
  "request_seconds": 286.2100326210166,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "c406b80525de5071eaa66dac08d64f6d865272d89190332d07c0d9d0206911f4",
  "runner": "benchmark_ollama.py",
  "runner_commit": "1cc2ce5d9e9148092d004abf953cbbbffa12110a",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 338651,
    "output_tokens": 0
  }
}
Prompt template
POST https://api.liquid.ai/decisions/v1/systemone
{
 "model": "d1.2-ultra",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · d1.2-ultra_en_dontknow

Name
d1.2-ultra
Setup
Liquid d1.2-ultra · decision model · with the I don't know option F
Model
d1.2-ultra · provider liquid
GPU
—
Cost
$0.0155 for the run · provider price × reported usage
Answer time
run d1.2-ultra_en_dontknow · first answer 2026-11-16 19:12:48 UTC · last answer 2026-11-16 19:17:54 UTC · span 306.0 s, pauses included (not charged) · charged 306.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://www.liquid.ai/blog/d1-2 · read 2026-11-16
Input
text, English
Endpoint
https://api.liquid.ai/decisions
Started
16 Nov 2026 19:12 UTC; last answer 16 Nov 2026 19:17 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Liquid d1.2-ultra (site run None); runner 1cc2ce5d9; Liquid's hosted decision API, released 16 November 2026 (liquid.ai/blog/d1-2); run at once on Umit's word, whatever the cost (#10926, zeki #10928), one request at a time as d1-ultra ran
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "d1.2-ultra",
    "--provider",
    "liquid",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "0f7123a347725d64b71f99d55f89d66f32fa9eb0",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "d1.2-ultra",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Liquid AI's d1.2-ultra price",
    "date": "2026-11-16",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.042,
      "cache_write_input_tokens": 0.042,
      "cached_input_tokens": 0.042,
      "input_tokens": 0.042,
      "output_tokens": 0.0
    },
    "source": "https://www.liquid.ai/blog/d1-2",
    "text": "$0.042 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "liquid",
  "questions": 1125,
  "request_seconds": 293.6367528460287,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "af50cdb23e0e99b114433d9bee4d5501106c129fe3d0a0c62fabe8f1b25088dd",
  "runner": "benchmark_ollama.py",
  "runner_commit": "1cc2ce5d9e9148092d004abf953cbbbffa12110a",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 368953,
    "output_tokens": 0
  }
}
Prompt template
POST https://api.liquid.ai/decisions/v1/systemone
{
 "model": "d1.2-ultra",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · d1.2_en

Name
d1.2
Setup
Liquid d1.2 · decision model
Model
d1.2 · provider liquid
GPU
—
Cost
$0.006773 for the run · provider price × reported usage
Answer time
run d1.2_en · first answer 2026-11-16 19:32:21 UTC · last answer 2026-11-16 19:37:23 UTC · span 302.0 s, pauses included (not charged) · charged 302.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://www.liquid.ai/blog/d1-2 · read 2026-11-16
Input
text, English
Endpoint
https://api.liquid.ai/decisions
Started
16 Nov 2026 19:32 UTC; last answer 16 Nov 2026 19:37 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Liquid d1.2 (site run None); runner 1cc2ce5d9; Liquid's hosted decision API, released 16 November 2026 (liquid.ai/blog/d1-2); run at once on Umit's word, both models whatever the cost (#10926), one request at a time as d1.1 ran
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "d1.2",
    "--provider",
    "liquid",
    "--timeout",
    "30",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "0f7123a347725d64b71f99d55f89d66f32fa9eb0",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "d1.2",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Liquid AI's d1.2 price",
    "date": "2026-11-16",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.02,
      "cache_write_input_tokens": 0.02,
      "cached_input_tokens": 0.02,
      "input_tokens": 0.02,
      "output_tokens": 0.0
    },
    "source": "https://www.liquid.ai/blog/d1-2",
    "text": "$0.02 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "liquid",
  "questions": 1125,
  "request_seconds": 288.7145658520931,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "255310a19909313e2747a47eb199ecf7a7a7b12054f48f1d3bea90dddc18d106",
  "runner": "benchmark_ollama.py",
  "runner_commit": "1cc2ce5d9e9148092d004abf953cbbbffa12110a",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 338651,
    "output_tokens": 0
  }
}
Prompt template
POST https://api.liquid.ai/decisions/v1/systemone
{
 "model": "d1.2",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Logic · granite-4.2-3b-nothink_en

Name
Granite-4.2-3B off · chat
Setup
Granite-4.2-3B (chat, thinking off, on our GPU) · chat model
Model
granite-4.2-3b-nothink · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.3244 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.13 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.324. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run granite-4.2-3b-nothink_en · first answer 2026-11-16 19:39:30 UTC · last answer 2026-11-16 19:47:19 UTC · span 469.0 s, pauses included (not charged) · charged 469.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.3244 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 11.7 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 4 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 3B dense · source https://huggingface.co/ibm-granite/granite-4.2-3b · read 2026-11-12 · the card states 3B (the maker's figure is shown); checkpoint count 3.66B (3,659,737,600 incl. untied lm_head, text only; dense)
Input
text, English
Endpoint
http://127.0.0.1:18641/v1/chat/completions
Started
16 Nov 2026 19:39 UTC; last answer 16 Nov 2026 19:47 UTC
Progress
25 answered of 1,125 asked; 0 request errors; smoke test at 32,768 tokens only; the full run uses a higher limit (run granite-4.2-3b-nothink-64k_en)
Notes
Granite-4.2-3B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from ibm-granite/granite-4.2-3b@4e9c275e232f10211ec88644d106a1f6a1b60e06, vLLM reasoning parser granite_thinking_parser, one RTX PRO 6000 on our GPU host alone; thinking off (enable_thinking false, the template's own switch); sampling temperature 1, top_p 0.95, from generation_config.json; seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 3.66B / 3.66B active / total (dense; checkpoint 3,659,737,600 incl. untied lm_head, text only; card: 3B). English translations (DeepSeek + review). SUPERSEDED by granite-4.2-3b-nothink-64k_en (cut-offs only; one retry at 65,536).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "max_tokens": 32768,
  "seed": 0,
  "temperature": 1.0,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Logic · d1.2_en_dontknow

Name
d1.2
Setup
Liquid d1.2 · decision model · with the I don't know option F
Model
d1.2 · provider liquid
GPU
—
Cost
$0.007379 for the run · provider price × reported usage
Answer time
run d1.2_en_dontknow · first answer 2026-11-16 19:46:48 UTC · last answer 2026-11-16 19:51:16 UTC · span 268.0 s, pauses included (not charged) · charged 268.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://www.liquid.ai/blog/d1-2 · read 2026-11-16
Input
text, English
Endpoint
https://api.liquid.ai/decisions
Started
16 Nov 2026 19:46 UTC; last answer 16 Nov 2026 19:51 UTC
Progress
1,125 answered of 1,125 asked; 0 request errors; complete
Notes
Decision model, published setup Liquid d1.2 (site run None); runner 1cc2ce5d9; Liquid's hosted decision API, released 16 November 2026 (liquid.ai/blog/d1-2); run at once on Umit's word, both models whatever the cost (#10926), one request at a time as d1.1 ran
Parameters
{
  "bank": "logic_en.sqlite",
  "bank_sha256": "22e405875f10e1c8b4f401786b85b203b9001479c0dbd22ad09300ec75693172",
  "batch_size": 1,
  "command": [
    "d1.2",
    "--provider",
    "liquid",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/logic_en.jsonl",
    "--questions",
    "banks/logic_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "9435c6570e1f49c0dafeff7c17dedf1c58df4a1fefbdafb7f9251e81c37d9ebc",
  "harness_commit": "0f7123a347725d64b71f99d55f89d66f32fa9eb0",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "d1.2",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Liquid AI's d1.2 price",
    "date": "2026-11-16",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.02,
      "cache_write_input_tokens": 0.02,
      "cached_input_tokens": 0.02,
      "input_tokens": 0.02,
      "output_tokens": 0.0
    },
    "source": "https://www.liquid.ai/blog/d1-2",
    "text": "$0.02 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "liquid",
  "questions": 1125,
  "request_seconds": 264.3088338079724,
  "requests": 1125,
  "results_db": "(local file, not published)",
  "run_key": "4b225c622bcafec40028800a244dcdd2f64f64ccd7e04e511f7a4b9f1a4b6438",
  "runner": "benchmark_ollama.py",
  "runner_commit": "1cc2ce5d9e9148092d004abf953cbbbffa12110a",
  "runner_run_id": 2,
  "selection_sha256": "6a30d4def0bdb04adb92407cff7341c4cef595219cd7daf19cca161acda5f274",
  "site_run": null,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 368953,
    "output_tokens": 0
  }
}
Prompt template
POST https://api.liquid.ai/decisions/v1/systemone
{
 "model": "d1.2",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Logic · granite-4.2-3b-nothink-64k_en

Name
Granite-4.2-3B off · chat
Setup
Granite-4.2-3B (chat, thinking off, on our GPU) · chat model
Model
granite-4.2-3b-nothink · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$5.755 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 2.31 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $5.755. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run granite-4.2-3b-nothink-64k_en · first answer 2026-11-16 19:48:45 UTC · last answer 2026-11-16 22:07:26 UTC · span 8321.0 s, pauses included (not charged) · charged 8321.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $5.7554 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 27.6 in flight (sum of request latencies / active time)
Output limit
65,536 tokens · 2 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 3B dense · source https://huggingface.co/ibm-granite/granite-4.2-3b · read 2026-11-12 · the card states 3B (the maker's figure is shown); checkpoint count 3.66B (3,659,737,600 incl. untied lm_head, text only; dense)
Input
text, English
Endpoint
http://127.0.0.1:18641/v1/chat/completions
Started
16 Nov 2026 19:48 UTC; last answer 16 Nov 2026 22:07 UTC
Progress
259 answered of 1,125 asked; 0 request errors; in progress
Notes
Granite-4.2-3B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from ibm-granite/granite-4.2-3b@4e9c275e232f10211ec88644d106a1f6a1b60e06, vLLM reasoning parser granite_thinking_parser, one RTX PRO 6000 on our GPU host alone; thinking off (enable_thinking false, the template's own switch); sampling temperature 1, top_p 0.95, from generation_config.json; seed 0; max_tokens 65,536 (raised from 32,768 once: the smoke failed on cut-offs only); one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 3.66B / 3.66B active / total (dense; checkpoint 3,659,737,600 incl. untied lm_head, text only; card: 3B). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "max_tokens": 65536,
  "seed": 0,
  "temperature": 1.0,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · qwen3.8-27b-bf16_en_low

Name
Qwen3.8-27B low · chat
Setup
Qwen3.8-27B (chat, reasoning low, on our GPU) · chat model
Model
qwen38-27b-bf16 · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.146 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.46 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.146. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-06 22:08:53–11:29:53 UTC
GPU time
run qwen3.8-27b-bf16_en_low · first answer 2026-11-06 21:41:16 UTC · last answer 2026-11-12 11:29:53 UTC · span 481717.0 s, pauses included (not charged) · charged 1657.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.1461 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 16.2 in flight (sum of request latencies / active time)
Output limit
60,000 tokens · 0 answers cut off at it with no letter (counted wrong) · 1 empty reply asked once more
Model size
Parameters: 27B dense · source https://huggingface.co/Qwen/Qwen3.8-27B · read 2026-11-10 · the card states Number of Parameters: 27B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 27.78B (counted from the checkpoint (27,781,427,952, the vision encoder included); the card says 27B)
Input
text, English
Endpoint
http://127.0.0.1:8022/v1/chat/completions
Started
6 Nov 2026 21:41 UTC; last answer 12 Nov 2026 11:29 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Qwen3.8-27B BF16 (vLLM v0.30.0, MTP, GPU 1 of our GPU host), thinking on at reasoning_effort low, model generation config sampling (1.0/0.95/20, seed 0), JSON answer schema; English translations (DeepSeek + review); empty-reply retry under the 8 Nov rule done on 12 Nov 2026 (11:18–11:30 UTC, same server and request settings): question kk1-temel-kavramlar-test-15-14, first reply empty at 30,440 tokens (kept in empty_retries), retry also empty (finish stop, 35,217 tokens), scored wrong; score unchanged 2,146/2,163 Original latency of the empty first reply to kk1-temel-kavramlar-test-15-14 (overwritten by the 12 Nov retry write): 576.77 s, recovered from the 11 Nov backup ~/T9/logic-backup/2026-11-11/math-benchmark.db (row created 2026-11-06T22:15:53Z, 30,440 completion tokens) (zeki #5304).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "reasoning_effort": "low"
  },
  "max_tokens": 60000,
  "response_format": {
    "json_schema": {
      "name": "answer",
      "schema": {
        "additionalProperties": false,
        "properties": {
          "answer": {
            "enum": [
              "A",
              "B",
              "C",
              "D",
              "E"
            ],
            "type": "string"
          }
        },
        "required": [
          "answer"
        ],
        "type": "object"
      },
      "strict": true
    },
    "type": "json_schema"
  },
  "seed": 0,
  "temperature": 1.0,
  "top_k": 20,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. Give your answer only as JSON {{"answer": "X"}} (X = A, B, C, D or E).

{question}

Math · gemma-4-31b-it_en_t0

Name
Gemma 4 31B chat
Setup
Gemma 4 31B (chat, on our GPU) · chat model
Model
gemma-4-31B-it · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.148 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.461 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.148. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gemma-4-31b-it_en_t0 · first answer 2026-11-06 21:41:18 UTC · last answer 2026-11-06 22:08:58 UTC · span 1660.0 s, pauses included (not charged) · charged 1660.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.1482 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 16.0 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 4 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 30.7B dense · source https://huggingface.co/google/gemma-4-31B-it · read 2026-11-10 · the card states Total Parameters 30.7B (plus a vision encoder of about 550M) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 31.27B (counted from the checkpoint (31,273,088,876, the vision encoder included); the card gives 30.7B plus a vision encoder of about 550M)
Input
text, English
Endpoint
http://127.0.0.1:8021/v1/chat/completions
Started
6 Nov 2026 21:41 UTC; last answer 6 Nov 2026 22:08 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Gemma 4 31B-it FP8 weights + FP8 KV (vLLM v0.30.0, GPU 0 of our GPU host), thinking off (server default), temperature 0, English translations (DeepSeek + review)
Parameters
{
  "max_tokens": 8192,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · decider-flash-v1.2_en

Name
Perplexity Decider Flash v1.2
Setup
Perplexity Decider Flash v1.2 · decision model
Model
pplx-decider-flash-v1.2 · provider pplx-decider
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.004707 for the run, list · maker's list price; RTX PRO 6000 Blackwell: $0.06502 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.0261 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.065. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run decider-flash-v1.2_en · first answer 2026-11-07 13:44:53 UTC · last answer 2026-11-07 13:46:27 UTC · span 94.0 s, pauses included (not charged) · charged 94.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0650 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 4B dense · source https://huggingface.co/perplexity-ai/pplx-decider-flash-v1.2 · read 2026-11-10 · the card: built on Qwen3.5-4B, 'about one seventh of the size' of the 27B models; Ümit/turkce #4263: 4B (turkce #4263: the maker's stated total is shown); counted from the released files: 4.54B (counted from the released checkpoint: 4,539,918,336 (based on Qwen3.5-4B))
Input
text, English
Endpoint
http://127.0.0.1:19180
Started
7 Nov 2026 13:44 UTC; last answer 7 Nov 2026 13:46 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Perplexity Decider Flash v1.2 (site run 223); runner ffc307cb7; published on a DGX Spark (GB10); gpu-lanes runs it on an RTX PRO 6000
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "pplx-decider-flash-v1.2",
    "--provider",
    "pplx-decider",
    "--timeout",
    "30",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "pplx-decider-flash-v1.2",
  "perplexity_mode": null,
  "price": {
    "applies_to": "maker",
    "basis": "Perplexity's list price for the same model",
    "date": "2026-10-18",
    "extra_input_per_request": 3,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.01,
      "cache_write_input_tokens": 0.01,
      "cached_input_tokens": 0.01,
      "input_tokens": 0.01,
      "output_tokens": 0.0
    },
    "source": "https://docs.perplexity.ai/docs/getting-started/pricing",
    "text": "$0.01 input, output free per 1M tokens, plus 3 billed input tokens per request"
  },
  "prompt_version": "47f5776e71db",
  "provider": "pplx-decider",
  "questions": 2163,
  "request_seconds": 88.49582223361358,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "570d1952dc511cd96d377067d5dfda22a9d0e3eb9b291c6b660afb637b234668",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 223,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 464254,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19180/v1/systemone
{
 "model": "pplx-decider-flash-v1.2",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · decider-flash-v1.2_en_dontknow

Name
Perplexity Decider Flash v1.2
Setup
Perplexity Decider Flash v1.2 · decision model · with the I don't know option F
Model
pplx-decider-flash-v1.2 · provider pplx-decider
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.005356 for the run, list · maker's list price; RTX PRO 6000 Blackwell: $0.06848 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.0275 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.068. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run decider-flash-v1.2_en_dontknow · first answer 2026-11-07 13:46:33 UTC · last answer 2026-11-07 13:48:12 UTC · span 99.0 s, pauses included (not charged) · charged 99.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0685 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 4B dense · source https://huggingface.co/perplexity-ai/pplx-decider-flash-v1.2 · read 2026-11-10 · the card: built on Qwen3.5-4B, 'about one seventh of the size' of the 27B models; Ümit/turkce #4263: 4B (turkce #4263: the maker's stated total is shown); counted from the released files: 4.54B (counted from the released checkpoint: 4,539,918,336 (based on Qwen3.5-4B))
Input
text, English
Endpoint
http://127.0.0.1:19180
Started
7 Nov 2026 13:46 UTC; last answer 7 Nov 2026 13:48 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Perplexity Decider Flash v1.2 (site run 223); runner ffc307cb7; published on a DGX Spark (GB10); gpu-lanes runs it on an RTX PRO 6000
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "pplx-decider-flash-v1.2",
    "--provider",
    "pplx-decider",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "pplx-decider-flash-v1.2",
  "perplexity_mode": null,
  "price": {
    "applies_to": "maker",
    "basis": "Perplexity's list price for the same model",
    "date": "2026-10-18",
    "extra_input_per_request": 3,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.01,
      "cache_write_input_tokens": 0.01,
      "cached_input_tokens": 0.01,
      "input_tokens": 0.01,
      "output_tokens": 0.0
    },
    "source": "https://docs.perplexity.ai/docs/getting-started/pricing",
    "text": "$0.01 input, output free per 1M tokens, plus 3 billed input tokens per request"
  },
  "prompt_version": "413d6775fc58",
  "provider": "pplx-decider",
  "questions": 2163,
  "request_seconds": 92.80165477038827,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "03c5b73856e0285cf69e8f2fff16db6b7106b9f6a7ace3b56f72bc8153afadd4",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 223,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 529144,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19180/v1/systemone
{
 "model": "pplx-decider-flash-v1.2",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · d1-2b_en

Name
d1-2B
Setup
Liquid d1-2B (open weights) · decision model
Model
LiquidAI/d1-2B · provider d1-server
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.03043 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0122 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.030. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run d1-2b_en · first answer 2026-11-07 13:51:57 UTC · last answer 2026-11-07 13:52:41 UTC · span 44.0 s, pauses included (not charged) · charged 44.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0304 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.8 in flight (sum of request latencies / active time)
Model size
Parameters: 1.92B dense · source https://huggingface.co/LiquidAI/d1-2B · read 2026-11-09 · the card: 1.92B parameters; counted from the checkpoint 1,918,181,888
Input
text, English
Endpoint
http://127.0.0.1:19260
Started
7 Nov 2026 13:51 UTC; last answer 7 Nov 2026 13:52 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Liquid d1-2B (open weights) (site run 340); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "LiquidAI/d1-2B",
    "--provider",
    "d1-server",
    "--timeout",
    "30",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "LiquidAI/d1-2B",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "47f5776e71db",
  "provider": "d1-server",
  "questions": 2163,
  "request_seconds": 37.160787550034,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "351ec7792a905117620f3defd4b9ddc5085566e3f18d5cbabc182273903b97d1",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 340,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 311943,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19260/v1/systemone
{
 "model": "LiquidAI/d1-2B",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · d1-2b_en_dontknow

Name
d1-2B
Setup
Liquid d1-2B (open weights) · decision model · with the I don't know option F
Model
LiquidAI/d1-2B · provider d1-server
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.02974 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0119 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.030. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run d1-2b_en_dontknow · first answer 2026-11-07 13:52:45 UTC · last answer 2026-11-07 13:53:28 UTC · span 43.0 s, pauses included (not charged) · charged 43.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0297 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.8 in flight (sum of request latencies / active time)
Model size
Parameters: 1.92B dense · source https://huggingface.co/LiquidAI/d1-2B · read 2026-11-09 · the card: 1.92B parameters; counted from the checkpoint 1,918,181,888
Input
text, English
Endpoint
http://127.0.0.1:19260
Started
7 Nov 2026 13:52 UTC; last answer 7 Nov 2026 13:53 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Liquid d1-2B (open weights) (site run 340); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "LiquidAI/d1-2B",
    "--provider",
    "d1-server",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "LiquidAI/d1-2B",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "413d6775fc58",
  "provider": "d1-server",
  "questions": 2163,
  "request_seconds": 36.294632641016506,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "085113febb9849c28b0eb36df398455ba1b24114c026f99bc0e4d8a1c1875551",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 340,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 369209,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19260/v1/systemone
{
 "model": "LiquidAI/d1-2B",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · glide-v1.2_en

Name
GLiDE v1.2
Setup
Fastino GLiDE v1.2 · decision model
Model
glide-v1.2 · provider glide
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.01632 for the run, list · maker's list price; RTX PRO 6000 Blackwell: $0.1252 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.0503 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.125. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run glide-v1.2_en · first answer 2026-11-07 13:54:15 UTC · last answer 2026-11-07 13:57:16 UTC · span 181.0 s, pauses included (not charged) · charged 181.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1252 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/fastino/GLiDE · read 2026-11-10 · the card states a 27B GLiDE decision model (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 26.09B (counted from the checkpoint (26,086,635,760: language model, vision encoder, readout); Fastino's blog says 26.09B)
Input
text, English
Endpoint
http://127.0.0.1:19110
Started
7 Nov 2026 13:54 UTC; last answer 7 Nov 2026 13:57 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Fastino GLiDE v1.2 (site run 248); runner ffc307cb7; published on a DGX Spark (GB10); gpu-lanes runs it on an RTX PRO 6000
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "glide-v1.2",
    "--provider",
    "glide",
    "--timeout",
    "60",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "glide-v1.2",
  "perplexity_mode": null,
  "price": {
    "applies_to": "maker",
    "basis": "Fastino's list price for the same model",
    "date": "2026-10-22",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.04,
      "cache_write_input_tokens": 0.04,
      "cached_input_tokens": 0.04,
      "input_tokens": 0.04,
      "output_tokens": 0.0
    },
    "source": "https://docs.fastino.ai/pricing",
    "text": "$0.04 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "glide",
  "questions": 2163,
  "request_seconds": 174.96084069646895,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "f04fc73f56e3d4138b2aefe644f6cfb8480931d8fddd56841e1ec388d99fa422",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 248,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 408016,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19110/v1/systemone
{
 "model": "glide-v1.2",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · glide-v1.2_en_dontknow

Name
GLiDE v1.2
Setup
Fastino GLiDE v1.2 · decision model · with the I don't know option F
Model
glide-v1.2 · provider glide
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.01892 for the run, list · maker's list price; RTX PRO 6000 Blackwell: $0.1307 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.0525 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.131. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run glide-v1.2_en_dontknow · first answer 2026-11-07 13:57:22 UTC · last answer 2026-11-07 14:00:31 UTC · span 189.0 s, pauses included (not charged) · charged 189.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1307 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/fastino/GLiDE · read 2026-11-10 · the card states a 27B GLiDE decision model (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 26.09B (counted from the checkpoint (26,086,635,760: language model, vision encoder, readout); Fastino's blog says 26.09B)
Input
text, English
Endpoint
http://127.0.0.1:19110
Started
7 Nov 2026 13:57 UTC; last answer 7 Nov 2026 14:00 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Fastino GLiDE v1.2 (site run 248); runner ffc307cb7; published on a DGX Spark (GB10); gpu-lanes runs it on an RTX PRO 6000
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "glide-v1.2",
    "--provider",
    "glide",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "glide-v1.2",
  "perplexity_mode": null,
  "price": {
    "applies_to": "maker",
    "basis": "Fastino's list price for the same model",
    "date": "2026-10-22",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.04,
      "cache_write_input_tokens": 0.04,
      "cached_input_tokens": 0.04,
      "input_tokens": 0.04,
      "output_tokens": 0.0
    },
    "source": "https://docs.fastino.ai/pricing",
    "text": "$0.04 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "glide",
  "questions": 2163,
  "request_seconds": 183.4991300853435,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "6d9527bb33589b06882ebb2e774b80c40ff58924e767f5a84bf8e6d692b89605",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 248,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 472906,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19110/v1/systemone
{
 "model": "glide-v1.2",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · strands_en

Name
Strands Decider 2B
Setup
Strands Decider 2B · decision model
Model
strands-decider-2b · provider strands
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.05672 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0228 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.057. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run strands_en · first answer 2026-11-07 14:00:06 UTC · last answer 2026-11-07 14:01:28 UTC · span 82.0 s, pauses included (not charged) · charged 82.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0567 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 2B dense · source our local copy of the model files · read 2026-11-10 · the official model name: Strands Decider 2B (turkce #4263: the maker's stated total is shown); counted from the released files: 2.29B (counted from the checkpoint: Qwen3.5-2B-Base with its LoRA and head (2,291,942,208); the Strands card states no total)
Input
text, English
Endpoint
http://127.0.0.1:19270
Started
7 Nov 2026 14:00 UTC; last answer 7 Nov 2026 14:01 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Strands Decider 2B (site run 141); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "strands-decider-2b",
    "--provider",
    "strands",
    "--timeout",
    "120",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "strands-decider-2b",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "47f5776e71db",
  "provider": "strands",
  "questions": 2163,
  "request_seconds": 77.04395882436074,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "4ba416997e5e3d0ec08fe0cfb0d9536304bd1ba0c006fad412accb33759a745d",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 141,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 407906,
    "output_tokens": 2163
  }
}
Prompt template
POST http://127.0.0.1:19270/v1/systemone
{
 "model": "strands-decider-2b",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · strands_en_dontknow

Name
Strands Decider 2B
Setup
Strands Decider 2B · decision model · with the I don't know option F
Model
strands-decider-2b · provider strands
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.05672 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0228 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.057. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run strands_en_dontknow · first answer 2026-11-07 14:01:33 UTC · last answer 2026-11-07 14:02:55 UTC · span 82.0 s, pauses included (not charged) · charged 82.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0567 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 2B dense · source our local copy of the model files · read 2026-11-10 · the official model name: Strands Decider 2B (turkce #4263: the maker's stated total is shown); counted from the released files: 2.29B (counted from the checkpoint: Qwen3.5-2B-Base with its LoRA and head (2,291,942,208); the Strands card states no total)
Input
text, English
Endpoint
http://127.0.0.1:19270
Started
7 Nov 2026 14:01 UTC; last answer 7 Nov 2026 14:02 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Strands Decider 2B (site run 141); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "strands-decider-2b",
    "--provider",
    "strands",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "strands-decider-2b",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "413d6775fc58",
  "provider": "strands",
  "questions": 2163,
  "request_seconds": 76.55758669122588,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "05986a28dd828569e51993b6233c6f054507c4a79ca7a5efdb8734af8c456130",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 141,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 472796,
    "output_tokens": 2163
  }
}
Prompt template
POST http://127.0.0.1:19270/v1/systemone
{
 "model": "strands-decider-2b",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · decider-v1.2-local_en

Name
Perplexity Decider 27B v1.2 NVIDIA RTX PRO 6000 Blackwell
Setup
Perplexity Decider 27B v1.2 (local GPU) · decision model
Model
pplx-decider-v1.2 · provider pplx-decider
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.009415 for the run, list · maker's list price; RTX PRO 6000 Blackwell: $0.1515 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.0608 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.151. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run decider-v1.2-local_en · first answer 2026-11-07 14:21:35 UTC · last answer 2026-11-07 14:25:14 UTC · span 219.0 s, pauses included (not charged) · charged 219.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1515 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/perplexity-ai/pplx-decider-v1.2 · read 2026-11-10 · the official model name: 27B (pplx-decider 27B v1.2; the card states no count of its own) (turkce #4263: the maker's stated total is shown); counted from the released files: 26.09B (counted from the released checkpoint: backbone 26,085,330,160 (its 11 safetensors shards) + decision head (readout.safetensors) 1,305,600 = 26,086,635,760; the API setups use the same count)
Input
text, English
Endpoint
http://127.0.0.1:19170
Started
7 Nov 2026 14:21 UTC; last answer 7 Nov 2026 14:25 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Perplexity Decider 27B v1.2 (local GPU) (site run 217); runner ffc307cb7; published on a DGX Spark (GB10); gpu-lanes runs it on an RTX PRO 6000
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "pplx-decider-v1.2",
    "--provider",
    "pplx-decider",
    "--timeout",
    "30",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "pplx-decider-v1.2",
  "perplexity_mode": null,
  "price": {
    "applies_to": "maker",
    "basis": "Perplexity's list price for the same model",
    "date": "2026-10-18",
    "extra_input_per_request": 3,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.02,
      "cache_write_input_tokens": 0.02,
      "cached_input_tokens": 0.02,
      "input_tokens": 0.02,
      "output_tokens": 0.0
    },
    "source": "https://docs.perplexity.ai/docs/getting-started/pricing",
    "text": "$0.02 input, output free per 1M tokens, plus 3 billed input tokens per request"
  },
  "prompt_version": "47f5776e71db",
  "provider": "pplx-decider",
  "questions": 2163,
  "request_seconds": 213.26625031325966,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "1a07ef5a04f19e8ecbb43b4a8bb610b8286ac707e78832fa863ab1661fb54567",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 217,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 464254,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19170/v1/systemone
{
 "model": "pplx-decider-v1.2",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · decider-v1.2-local_en_dontknow

Name
Perplexity Decider 27B v1.2 NVIDIA RTX PRO 6000 Blackwell
Setup
Perplexity Decider 27B v1.2 (local GPU) · decision model · with the I don't know option F
Model
pplx-decider-v1.2 · provider pplx-decider
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.01071 for the run, list · maker's list price; RTX PRO 6000 Blackwell: $0.1653 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.0664 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.165. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run decider-v1.2-local_en_dontknow · first answer 2026-11-07 14:25:21 UTC · last answer 2026-11-07 14:29:20 UTC · span 239.0 s, pauses included (not charged) · charged 239.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1653 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/perplexity-ai/pplx-decider-v1.2 · read 2026-11-10 · the official model name: 27B (pplx-decider 27B v1.2; the card states no count of its own) (turkce #4263: the maker's stated total is shown); counted from the released files: 26.09B (counted from the released checkpoint: backbone 26,085,330,160 (its 11 safetensors shards) + decision head (readout.safetensors) 1,305,600 = 26,086,635,760; the API setups use the same count)
Input
text, English
Endpoint
http://127.0.0.1:19170
Started
7 Nov 2026 14:25 UTC; last answer 7 Nov 2026 14:29 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Perplexity Decider 27B v1.2 (local GPU) (site run 217); runner ffc307cb7; published on a DGX Spark (GB10); gpu-lanes runs it on an RTX PRO 6000
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "pplx-decider-v1.2",
    "--provider",
    "pplx-decider",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "pplx-decider-v1.2",
  "perplexity_mode": null,
  "price": {
    "applies_to": "maker",
    "basis": "Perplexity's list price for the same model",
    "date": "2026-10-18",
    "extra_input_per_request": 3,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.02,
      "cache_write_input_tokens": 0.02,
      "cached_input_tokens": 0.02,
      "input_tokens": 0.02,
      "output_tokens": 0.0
    },
    "source": "https://docs.perplexity.ai/docs/getting-started/pricing",
    "text": "$0.02 input, output free per 1M tokens, plus 3 billed input tokens per request"
  },
  "prompt_version": "413d6775fc58",
  "provider": "pplx-decider",
  "questions": 2163,
  "request_seconds": 230.58549142512493,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "b45c37ba099429f357ff643b7867ae5bbf06712b901bfdcc7be031dcc2242005",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 217,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 529144,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19170/v1/systemone
{
 "model": "pplx-decider-v1.2",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · metask_en

Name
Metask-Jev-4B
Setup
Metask-Jev 4B · decision model
Model
metask-jev-4b-policy-mix · provider metask
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.1798 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0722 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.180. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run metask_en · first answer 2026-11-07 14:37:06 UTC · last answer 2026-11-07 14:41:26 UTC · span 260.0 s, pauses included (not charged) · charged 260.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1798 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 4.54B dense · source our local copy of the model files · read 2026-11-09 · the card: 4.54B parameters (Qwen3.5-4B with its LoRA merged)
Input
text, English
Endpoint
http://127.0.0.1:19250
Started
7 Nov 2026 14:37 UTC; last answer 7 Nov 2026 14:41 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Metask-Jev 4B (site run 126); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "metask-jev-4b-policy-mix",
    "--provider",
    "metask",
    "--timeout",
    "120",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "metask-jev-4b-policy-mix",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "47f5776e71db",
  "provider": "metask",
  "questions": 2163,
  "request_seconds": 252.0949217779562,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "15d8388ce0d2b80f49af6fa6f2a57c67c530f8460bcc9b59d957e9e218dec5e4",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 126,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 733739,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19250/v1/systemone
{
 "model": "metask-jev-4b-policy-mix",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · metask_en_dontknow

Name
Metask-Jev-4B
Setup
Metask-Jev 4B · decision model · with the I don't know option F
Model
metask-jev-4b-policy-mix · provider metask
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.1902 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0764 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.190. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run metask_en_dontknow · first answer 2026-11-07 14:41:33 UTC · last answer 2026-11-07 14:46:08 UTC · span 275.0 s, pauses included (not charged) · charged 275.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1902 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 4.54B dense · source our local copy of the model files · read 2026-11-09 · the card: 4.54B parameters (Qwen3.5-4B with its LoRA merged)
Input
text, English
Endpoint
http://127.0.0.1:19250
Started
7 Nov 2026 14:41 UTC; last answer 7 Nov 2026 14:46 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Metask-Jev 4B (site run 126); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "metask-jev-4b-policy-mix",
    "--provider",
    "metask",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "metask-jev-4b-policy-mix",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "413d6775fc58",
  "provider": "metask",
  "questions": 2163,
  "request_seconds": 269.421553739463,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "0307e5813740c3591c14f14a105e8e2e5d8c2a2e97b2da558349f341c5b661a3",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 126,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 825327,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19250/v1/systemone
{
 "model": "metask-jev-4b-policy-mix",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · drex-max-v1_en

Name
Drex Max
Setup
Drex Max v1 · decision model
Model
drex-max-v1 · provider drex-max
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.02824 for the run, list · maker's list price; RTX PRO 6000 Blackwell: $0.1183 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.0475 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.118. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run drex-max-v1_en · first answer 2026-11-07 14:46:06 UTC · last answer 2026-11-07 14:48:57 UTC · span 171.0 s, pauses included (not charged) · charged 171.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1183 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/nace-ai/drex-max-v1 · read 2026-11-10 · the card states a dense 27B-parameter model (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 26.09B (counted from the released checkpoint: backbone 26,085,330,160 (its 11 safetensors shards) + decision head (readout.safetensors) 1,305,600 = 26,086,635,760)
Input
text, English
Endpoint
http://127.0.0.1:19140
Started
7 Nov 2026 14:46 UTC; last answer 7 Nov 2026 14:48 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Drex Max v1 (site run 322); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "drex-max-v1",
    "--provider",
    "drex-max",
    "--timeout",
    "30",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "drex-max-v1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "maker",
    "basis": "Nace.AI's list price for the same model",
    "date": "2026-10-30",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.06,
      "cache_write_input_tokens": 0.06,
      "cached_input_tokens": 0.06,
      "input_tokens": 0.06,
      "output_tokens": 0.0
    },
    "source": "https://nace.ai/drex",
    "text": "$0.06 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "drex-max",
  "questions": 2163,
  "request_seconds": 166.13710108329542,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "5c1c78e14d39f505661e13e0a7759935c28f61687405da21168d3bf44772a947",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 322,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 470743,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19140/v1/systemone
{
 "model": "drex-max-v1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · drex-max-v1_en_dontknow

Name
Drex Max
Setup
Drex Max v1 · decision model · with the I don't know option F
Model
drex-max-v1 · provider drex-max
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.03214 for the run, list · maker's list price; RTX PRO 6000 Blackwell: $0.1612 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.0647 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.161. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run drex-max-v1_en_dontknow · first answer 2026-11-07 14:49:03 UTC · last answer 2026-11-07 14:52:56 UTC · span 233.0 s, pauses included (not charged) · charged 233.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1612 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.7 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/nace-ai/drex-max-v1 · read 2026-11-10 · the card states a dense 27B-parameter model (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 26.09B (counted from the released checkpoint: backbone 26,085,330,160 (its 11 safetensors shards) + decision head (readout.safetensors) 1,305,600 = 26,086,635,760)
Input
text, English
Endpoint
http://127.0.0.1:19140
Started
7 Nov 2026 14:49 UTC; last answer 7 Nov 2026 14:52 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Drex Max v1 (site run 322); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "drex-max-v1",
    "--provider",
    "drex-max",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "drex-max-v1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "maker",
    "basis": "Nace.AI's list price for the same model",
    "date": "2026-10-30",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.06,
      "cache_write_input_tokens": 0.06,
      "cached_input_tokens": 0.06,
      "input_tokens": 0.06,
      "output_tokens": 0.0
    },
    "source": "https://nace.ai/drex",
    "text": "$0.06 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "drex-max",
  "questions": 2163,
  "request_seconds": 154.01120182021987,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "9ee4f9157cd02497a3b6939d8d30845ba40b69e0e74c4788a9f99371e75574fd",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 322,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 535633,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19140/v1/systemone
{
 "model": "drex-max-v1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · glide_en

Name
GLiDE API
Setup
Fastino GLiDE · decision model
Model
fastino/GLiDE · provider fastino
GPU
—
Cost
$0.0612 for the run · provider price × reported usage
Answer time
run glide_en · first answer 2026-11-07 14:58:12 UTC · last answer 2026-11-07 15:00:39 UTC · span 147.0 s, pauses included (not charged) · charged 147.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 4.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://docs.fastino.ai/inference/systemone · read 2026-11-09 · not disclosed by maker for the hosted version we ran (6 October 2026); Fastino's 5 November blog gives 26.09B for what it serves now, a later change
Input
text, English
Endpoint
https://api.fastino.ai
Started
7 Nov 2026 14:58 UTC; last answer 7 Nov 2026 15:00 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Fastino GLiDE (site run 129); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "fastino/GLiDE",
    "--provider",
    "fastino",
    "--timeout",
    "300",
    "--concurrency",
    "4",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 4,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "fastino/GLiDE",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Fastino's GLiDE price",
    "date": "2026-10-08",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.15,
      "cache_write_input_tokens": 0.15,
      "cached_input_tokens": 0.15,
      "input_tokens": 0.15,
      "output_tokens": 0.0
    },
    "source": "https://docs.fastino.ai/pricing",
    "text": "$0.15 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "fastino",
  "questions": 2163,
  "request_seconds": 147.813721301849,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "937fcfc50498548a00a26e54c826a1bc48a17f743e5ca8c5d703af87491d088c",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 129,
  "status": "completed",
  "timeout": 300.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 408016,
    "output_tokens": 0
  }
}
Prompt template
POST https://api.fastino.ai/v1/systemone
{
 "model": "fastino/GLiDE",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · glide_en_dontknow

Name
GLiDE API
Setup
Fastino GLiDE · decision model · with the I don't know option F
Model
fastino/GLiDE · provider fastino
GPU
—
Cost
$0.07094 for the run · provider price × reported usage
Answer time
run glide_en_dontknow · first answer 2026-11-07 15:00:44 UTC · last answer 2026-11-07 15:03:09 UTC · span 145.0 s, pauses included (not charged) · charged 145.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 4.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://docs.fastino.ai/inference/systemone · read 2026-11-09 · not disclosed by maker for the hosted version we ran (6 October 2026); Fastino's 5 November blog gives 26.09B for what it serves now, a later change
Input
text, English
Endpoint
https://api.fastino.ai
Started
7 Nov 2026 15:00 UTC; last answer 7 Nov 2026 15:03 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Fastino GLiDE (site run 129); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "fastino/GLiDE",
    "--provider",
    "fastino",
    "--timeout",
    "300",
    "--concurrency",
    "4",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 4,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "fastino/GLiDE",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Fastino's GLiDE price",
    "date": "2026-10-08",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.15,
      "cache_write_input_tokens": 0.15,
      "cached_input_tokens": 0.15,
      "input_tokens": 0.15,
      "output_tokens": 0.0
    },
    "source": "https://docs.fastino.ai/pricing",
    "text": "$0.15 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "fastino",
  "questions": 2163,
  "request_seconds": 145.15789324895013,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "13a754211bc3074ea852a3bfc279253b4b511ae71faf6bab15325e3848a6d62d",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 129,
  "status": "completed",
  "timeout": 300.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 472906,
    "output_tokens": 0
  }
}
Prompt template
POST https://api.fastino.ai/v1/systemone
{
 "model": "fastino/GLiDE",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · jev-1.14_en

Name
Jev 1.14
Setup
Jev 1.14.0 · decision model
Model
jev-1.14.0 · provider jev
GPU
—
Cost
$0.04108 for the run · provider price × reported usage
Answer time
run jev-1.14_en · first answer 2026-11-07 15:13:58 UTC · last answer 2026-11-07 15:20:39 UTC · span 401.0 s, pauses included (not charged) · charged 401.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://developers.cloudflare.com/ai/models/typesafe/jev-1.14.0/ · read 2026-11-09
Input
text, English
Endpoint
https://api.typesafe.ai
Started
7 Nov 2026 15:13 UTC; last answer 7 Nov 2026 15:20 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Jev 1.14.0 (site run 194); runner ffc307cb7; the run script reads the key from JEV_KEY_FILE; run_setup.sh takes JEV_API_KEY from the environment only
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "jev-1.14.0",
    "--provider",
    "jev",
    "--timeout",
    "30",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "jev-1.14.0",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "TypeSafe's System One price",
    "date": "2026-10-16",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.042,
      "cache_write_input_tokens": 0.042,
      "cached_input_tokens": 0.042,
      "input_tokens": 0.042,
      "output_tokens": 0.0
    },
    "source": "https://docs.typesafe.ai/models",
    "text": "$0.042 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "jev",
  "questions": 2163,
  "request_seconds": 357.25007506343536,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "1799c1975914893a91c4d5f1a8c33dc4588b276d9748d623c09a8a53f97cc4df",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 194,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 978187,
    "output_tokens": 112476
  }
}
Prompt template
POST https://api.typesafe.ai/v1/systemone
{
 "model": "jev-1.14.0",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · d3-ultra_en

Name
d3-ultra
Setup
vLLM-SR d3-ultra · decision model
Model
vllm-sr/d3-ultra · provider d3-server
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.1916 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0769 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.192. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-07 15:19:13–17:53:08 UTC
GPU time
run d3-ultra_en · first answer 2026-11-07 15:16:53 UTC · last answer 2026-11-07 17:55:25 UTC · span 9512.0 s, pauses included (not charged) · charged 277.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1916 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 26.09B dense · source https://huggingface.co/vllm-sr/d3-ultra · read 2026-11-09 · counted from the released checkpoint: backbone 26,085,330,160 (its 11 safetensors shards) + decision head (readout.safetensors) 1,305,600 = 26,086,635,760
Input
text, English
Endpoint
http://127.0.0.1:19130
Started
7 Nov 2026 15:16 UTC; last answer 7 Nov 2026 17:55 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup vLLM-SR d3-ultra (site run 269); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "vllm-sr/d3-ultra",
    "--provider",
    "d3-server",
    "--timeout",
    "60",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "vllm-sr/d3-ultra",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "47f5776e71db",
  "provider": "d3-server",
  "questions": 2163,
  "request_seconds": 269.67673703091486,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "6ca58457ad77dfd2d16b1b2767fc51af7efffd1843b11d6a810711af89fabe65",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 269,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 636084,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19130/v1/systemone
{
 "model": "vllm-sr/d3-ultra",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · d1-ultra_en

Name
d1-ultra
Setup
Liquid d1-ultra · decision model
Model
d1-ultra · provider liquid
GPU
—
Cost
$0.01328 for the run · provider price × reported usage
Answer time
run d1-ultra_en · first answer 2026-11-07 15:19:54 UTC · last answer 2026-11-07 15:28:31 UTC · span 517.0 s, pauses included (not charged) · charged 517.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) · source https://www.liquid.ai/blog/d1-ultra · read 2026-11-09
Input
text, English
Endpoint
https://api.liquid.ai/decisions
Started
7 Nov 2026 15:19 UTC; last answer 7 Nov 2026 15:28 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Liquid d1-ultra (site run 199); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "d1-ultra",
    "--provider",
    "liquid",
    "--timeout",
    "30",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "d1-ultra",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Liquid AI's d1-ultra price",
    "date": "2026-10-16",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.042,
      "cache_write_input_tokens": 0.042,
      "cached_input_tokens": 0.042,
      "input_tokens": 0.042,
      "output_tokens": 0.0
    },
    "source": "https://www.liquid.ai/blog/d1-ultra",
    "text": "$0.042 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "liquid",
  "questions": 2163,
  "request_seconds": 510.24304912041407,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "36cc01d39a3fc92e6b7f9af79ec9038ec4b7d65976290480a903dc5bfe83f66c",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 199,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 316269,
    "output_tokens": 0
  }
}
Prompt template
POST https://api.liquid.ai/decisions/v1/systemone
{
 "model": "d1-ultra",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · mercury-decide_en

Name
Mercury Decide
Setup
Inception Mercury Decide · decision model
Model
inception/mercury-decide · provider openrouter
GPU
—
Cost
$0.009483 for the run · provider price × reported usage
Answer time
run mercury-decide_en · first answer 2026-11-07 15:19:58 UTC · last answer 2026-11-07 15:30:59 UTC · span 661.0 s, pauses included (not charged) · charged 661.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://www.inceptionlabs.ai/ · read 2026-11-09
Input
text, English
Endpoint
https://openrouter.ai/api
Started
7 Nov 2026 15:19 UTC; last answer 7 Nov 2026 15:30 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Inception Mercury Decide (site run 204); runner ffc307cb7; via OpenRouter; latency includes OpenRouter's time
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "inception/mercury-decide",
    "--provider",
    "openrouter",
    "--timeout",
    "30",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "inception/mercury-decide",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "OpenRouter's Mercury Decide price",
    "date": "2026-10-17",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.02,
      "cache_write_input_tokens": 0.02,
      "cached_input_tokens": 0.02,
      "input_tokens": 0.02,
      "output_tokens": 0.0
    },
    "source": "https://openrouter.ai/inception/mercury-decide",
    "text": "$0.02 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "openrouter",
  "questions": 2163,
  "request_seconds": 654.9507486914517,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "a18ae6f4895023c2457c0dc4608bb7b85d71ea79725a562751826c7f84f9227c",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 204,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 474166,
    "output_tokens": 2163
  }
}
Prompt template
POST https://openrouter.ai/api/v1/systemone
{
 "model": "inception/mercury-decide",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · celeris_en

Name
celeris-1-decision
Setup
Celeris-1 Decision · decision model
Model
celeris-1-decision · provider celeris
GPU
—
Cost
$0.01968 for the run · provider price × reported usage
Answer time
run celeris_en · first answer 2026-11-07 15:19:59 UTC · last answer 2026-11-07 15:34:02 UTC · span 843.0 s, pauses included (not charged) · charged 843.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://docs.celeris.ai/decisions · read 2026-11-09
Input
text, English
Endpoint
https://inference.celeris.ai/celeris-1-decision
Started
7 Nov 2026 15:19 UTC; last answer 7 Nov 2026 15:34 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Celeris-1 Decision (site run 150); runner ffc307cb7; probabilities come rounded to two decimals
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "celeris-1-decision",
    "--provider",
    "celeris",
    "--timeout",
    "30",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "celeris-1-decision",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Celeris's decision model price",
    "date": "2026-10-10",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.04,
      "cache_write_input_tokens": 0.04,
      "cached_input_tokens": 0.04,
      "input_tokens": 0.04,
      "output_tokens": 0.0
    },
    "source": "https://docs.celeris.ai/pricing",
    "text": "$0.04 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "celeris",
  "questions": 2163,
  "request_seconds": 837.7112978210207,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "883a34e093ecfbbbab667bd30194d1bed2f75b06724b120c6d8d50567ec4a370",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 150,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 491901,
    "output_tokens": 0
  }
}
Prompt template
POST https://inference.celeris.ai/celeris-1-decision/v1/systemone
{
 "model": "celeris-1-decision",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · subq-1.1_en

Name
SubQ Snap 1.1
Setup
SubQ Snap 1.1 · decision model
Model
subq-snap-1.1 · provider subq
GPU
—
Cost
$0.01857 for the run · provider price × reported usage
Answer time
run subq-1.1_en · first answer 2026-11-07 15:19:59 UTC · last answer 2026-11-07 15:33:52 UTC · span 833.0 s, pauses included (not charged) · charged 833.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://docs.subq.ai/using-the-api/models/ · read 2026-11-09
Input
text, English
Endpoint
https://api.subq.ai
Started
7 Nov 2026 15:19 UTC; last answer 7 Nov 2026 15:33 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup SubQ Snap 1.1 (site run 346); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "subq-snap-1.1",
    "--provider",
    "subq",
    "--timeout",
    "30",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "subq-snap-1.1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "SubQ's list price",
    "date": "2026-11-01",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.04,
      "cache_write_input_tokens": 0.04,
      "cached_input_tokens": 0.04,
      "input_tokens": 0.04,
      "output_tokens": 0.0
    },
    "source": "https://docs.subq.ai/using-the-api/models/",
    "text": "$0.04 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "subq",
  "questions": 2163,
  "request_seconds": 827.4245663249167,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "92f226627e80c646fe5d34d8df12dc82c7a22071c20905feff9a542ca6ada002",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 346,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 464254,
    "output_tokens": 0
  }
}
Prompt template
POST https://api.subq.ai/v1/systemone
{
 "model": "subq-snap-1.1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · jev-1.14_en_dontknow

Name
Jev 1.14
Setup
Jev 1.14.0 · decision model · with the I don't know option F
Model
jev-1.14.0 · provider jev
GPU
—
Cost
$0.04434 for the run · provider price × reported usage
Answer time
run jev-1.14_en_dontknow · first answer 2026-11-07 15:20:46 UTC · last answer 2026-11-07 15:26:45 UTC · span 359.0 s, pauses included (not charged) · charged 359.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://developers.cloudflare.com/ai/models/typesafe/jev-1.14.0/ · read 2026-11-09
Input
text, English
Endpoint
https://api.typesafe.ai
Started
7 Nov 2026 15:20 UTC; last answer 7 Nov 2026 15:26 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Jev 1.14.0 (site run 194); runner ffc307cb7; the run script reads the key from JEV_KEY_FILE; run_setup.sh takes JEV_API_KEY from the environment only
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "jev-1.14.0",
    "--provider",
    "jev",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "jev-1.14.0",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "TypeSafe's System One price",
    "date": "2026-10-16",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.042,
      "cache_write_input_tokens": 0.042,
      "cached_input_tokens": 0.042,
      "input_tokens": 0.042,
      "output_tokens": 0.0
    },
    "source": "https://docs.typesafe.ai/models",
    "text": "$0.042 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "jev",
  "questions": 2163,
  "request_seconds": 353.45370735169854,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "e4b4b749431ba2706d28492602bc4291c94f0d50bacf5ef0b88f9ce0537b506c",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 194,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 1055762,
    "output_tokens": 127617
  }
}
Prompt template
POST https://api.typesafe.ai/v1/systemone
{
 "model": "jev-1.14.0",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · drex-max-1.1_en

Name
Drex Max v1.1
Setup
Drex Max 1.1 · decision model
Model
drex-max-v1.1 · provider nace
GPU
—
Cost
$0.02786 for the run · provider price × reported usage
Answer time
run drex-max-1.1_en · first answer 2026-11-07 15:26:35 UTC · last answer 2026-11-07 15:44:54 UTC · span 1099.0 s, pauses included (not charged) · charged 1099.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://console.nace.ai/docs/drex/reference/models · read 2026-11-09
Input
text, English
Endpoint
https://console.nace.ai
Started
7 Nov 2026 15:26 UTC; last answer 7 Nov 2026 15:44 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Drex Max 1.1 (site run 326); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "drex-max-v1.1",
    "--provider",
    "nace",
    "--timeout",
    "30",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "drex-max-v1.1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Nace.AI's list price (launch price $0.03 per 1M input tokens, 50% off, is what Nace bills)",
    "date": "2026-10-30",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.06,
      "cache_write_input_tokens": 0.06,
      "cached_input_tokens": 0.06,
      "input_tokens": 0.06,
      "output_tokens": 0.0
    },
    "source": "https://nace.ai/drex",
    "text": "$0.06 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "nace",
  "questions": 2163,
  "request_seconds": 1092.7092372337356,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "558eb92e89b436138699e3d870115fdce198bb29f552a2f608cf622e3c96c401",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 326,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 464254,
    "output_tokens": 354281
  }
}
Prompt template
POST https://console.nace.ai/v1/systemone
{
 "model": "drex-max-v1.1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · d1-ultra_en_dontknow

Name
d1-ultra
Setup
Liquid d1-ultra · decision model · with the I don't know option F
Model
d1-ultra · provider liquid
GPU
—
Cost
$0.01569 for the run · provider price × reported usage
Answer time
run d1-ultra_en_dontknow · first answer 2026-11-07 15:28:40 UTC · last answer 2026-11-07 15:38:00 UTC · span 560.0 s, pauses included (not charged) · charged 560.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker (built on a larger open-weight model, not named with a count) · source https://www.liquid.ai/blog/d1-ultra · read 2026-11-09
Input
text, English
Endpoint
https://api.liquid.ai/decisions
Started
7 Nov 2026 15:28 UTC; last answer 7 Nov 2026 15:38 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Liquid d1-ultra (site run 199); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "d1-ultra",
    "--provider",
    "liquid",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "d1-ultra",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Liquid AI's d1-ultra price",
    "date": "2026-10-16",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.042,
      "cache_write_input_tokens": 0.042,
      "cached_input_tokens": 0.042,
      "input_tokens": 0.042,
      "output_tokens": 0.0
    },
    "source": "https://www.liquid.ai/blog/d1-ultra",
    "text": "$0.042 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "liquid",
  "questions": 2163,
  "request_seconds": 554.2210104055703,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "3172a31db59b954eb15fecc67d4018cce6663b43f7d758556fd48031b83f78a8",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 199,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 373535,
    "output_tokens": 0
  }
}
Prompt template
POST https://api.liquid.ai/decisions/v1/systemone
{
 "model": "d1-ultra",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · mercury-decide_en_dontknow

Name
Mercury Decide
Setup
Inception Mercury Decide · decision model · with the I don't know option F
Model
inception/mercury-decide · provider openrouter
GPU
—
Cost
$0.01114 for the run · provider price × reported usage
Answer time
run mercury-decide_en_dontknow · first answer 2026-11-07 15:31:10 UTC · last answer 2026-11-07 15:42:20 UTC · span 670.0 s, pauses included (not charged) · charged 670.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://www.inceptionlabs.ai/ · read 2026-11-09
Input
text, English
Endpoint
https://openrouter.ai/api
Started
7 Nov 2026 15:31 UTC; last answer 7 Nov 2026 15:42 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Inception Mercury Decide (site run 204); runner ffc307cb7; via OpenRouter; latency includes OpenRouter's time
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "inception/mercury-decide",
    "--provider",
    "openrouter",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "inception/mercury-decide",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "OpenRouter's Mercury Decide price",
    "date": "2026-10-17",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.02,
      "cache_write_input_tokens": 0.02,
      "cached_input_tokens": 0.02,
      "input_tokens": 0.02,
      "output_tokens": 0.0
    },
    "source": "https://openrouter.ai/inception/mercury-decide",
    "text": "$0.02 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "openrouter",
  "questions": 2163,
  "request_seconds": 663.1415279645007,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "4ae0638959e90991437cad2b4f9611d3b4cb3df1e3b343a1a4199c122f229356",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 204,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 557127,
    "output_tokens": 2163
  }
}
Prompt template
POST https://openrouter.ai/api/v1/systemone
{
 "model": "inception/mercury-decide",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · jev-1.13_en

Name
Jev 1.13
Setup
Jev 1.13.0 · decision model
Model
jev-1.13.0 · provider jev
GPU
—
Cost
$0.04108 for the run · provider price × reported usage
Answer time
run jev-1.13_en · first answer 2026-11-07 15:33:42 UTC · last answer 2026-11-07 15:40:11 UTC · span 389.0 s, pauses included (not charged) · charged 389.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://typesafe.ai/blog/jev-1-14 · read 2026-11-09
Input
text, English
Endpoint
https://api.typesafe.ai
Started
7 Nov 2026 15:33 UTC; last answer 7 Nov 2026 15:40 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Jev 1.13.0 (site run 49); runner ffc307cb7; the run script reads the key from JEV_KEY_FILE; run_setup.sh takes JEV_API_KEY from the environment only
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "jev-1.13.0",
    "--provider",
    "jev",
    "--timeout",
    "30",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "jev-1.13.0",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "TypeSafe's System One price",
    "date": "2026-10-08",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.042,
      "cache_write_input_tokens": 0.042,
      "cached_input_tokens": 0.042,
      "input_tokens": 0.042,
      "output_tokens": 0.0
    },
    "source": "https://typesafe.ai/blog/introducing-system-one-models-and-jev",
    "text": "$0.042 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "jev",
  "questions": 2163,
  "request_seconds": 381.6008429659996,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "642727ac96d84a0651cb3c5cfa205aafc2d49ea3ccabdcfb91df2a7409e3b3c1",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 49,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 978187,
    "output_tokens": 112476
  }
}
Prompt template
POST https://api.typesafe.ai/v1/systemone
{
 "model": "jev-1.13.0",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · subq-1.1_en_dontknow

Name
SubQ Snap 1.1
Setup
SubQ Snap 1.1 · decision model · with the I don't know option F
Model
subq-snap-1.1 · provider subq
GPU
—
Cost
$0.02117 for the run · provider price × reported usage
Answer time
run subq-1.1_en_dontknow · first answer 2026-11-07 15:34:05 UTC · last answer 2026-11-07 15:48:01 UTC · span 836.0 s, pauses included (not charged) · charged 836.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://docs.subq.ai/using-the-api/models/ · read 2026-11-09
Input
text, English
Endpoint
https://api.subq.ai
Started
7 Nov 2026 15:34 UTC; last answer 7 Nov 2026 15:48 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup SubQ Snap 1.1 (site run 346); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "subq-snap-1.1",
    "--provider",
    "subq",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "subq-snap-1.1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "SubQ's list price",
    "date": "2026-11-01",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.04,
      "cache_write_input_tokens": 0.04,
      "cached_input_tokens": 0.04,
      "input_tokens": 0.04,
      "output_tokens": 0.0
    },
    "source": "https://docs.subq.ai/using-the-api/models/",
    "text": "$0.04 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "subq",
  "questions": 2163,
  "request_seconds": 830.3051180056063,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "11974dd270a18268277d1096761b3ca395f03e53f3b4018e55b6d72b694361ff",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 346,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 529144,
    "output_tokens": 0
  }
}
Prompt template
POST https://api.subq.ai/v1/systemone
{
 "model": "subq-snap-1.1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · celeris_en_dontknow

Name
celeris-1-decision
Setup
Celeris-1 Decision · decision model · with the I don't know option F
Model
celeris-1-decision · provider celeris
GPU
—
Cost
$0.02262 for the run · provider price × reported usage
Answer time
run celeris_en_dontknow · first answer 2026-11-07 15:34:14 UTC · last answer 2026-11-07 15:48:22 UTC · span 848.0 s, pauses included (not charged) · charged 848.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://docs.celeris.ai/decisions · read 2026-11-09
Input
text, English
Endpoint
https://inference.celeris.ai/celeris-1-decision
Started
7 Nov 2026 15:34 UTC; last answer 7 Nov 2026 15:48 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Celeris-1 Decision (site run 150); runner ffc307cb7; probabilities come rounded to two decimals
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "celeris-1-decision",
    "--provider",
    "celeris",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "celeris-1-decision",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Celeris's decision model price",
    "date": "2026-10-10",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.04,
      "cache_write_input_tokens": 0.04,
      "cached_input_tokens": 0.04,
      "input_tokens": 0.04,
      "output_tokens": 0.0
    },
    "source": "https://docs.celeris.ai/pricing",
    "text": "$0.04 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "celeris",
  "questions": 2163,
  "request_seconds": 841.1620566984639,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "7f2b4ec630e3b9fd11b9bd98f695837f36ea236cbbf1fb80fb85fc90d39ebe1f",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 150,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 565443,
    "output_tokens": 0
  }
}
Prompt template
POST https://inference.celeris.ai/celeris-1-decision/v1/systemone
{
 "model": "celeris-1-decision",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · jev-1.13_en_dontknow

Name
Jev 1.13
Setup
Jev 1.13.0 · decision model · with the I don't know option F
Model
jev-1.13.0 · provider jev
GPU
—
Cost
$0.04434 for the run · provider price × reported usage
Answer time
run jev-1.13_en_dontknow · first answer 2026-11-07 15:40:18 UTC · last answer 2026-11-07 15:46:41 UTC · span 383.0 s, pauses included (not charged) · charged 383.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://typesafe.ai/blog/jev-1-14 · read 2026-11-09
Input
text, English
Endpoint
https://api.typesafe.ai
Started
7 Nov 2026 15:40 UTC; last answer 7 Nov 2026 15:46 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Jev 1.13.0 (site run 49); runner ffc307cb7; the run script reads the key from JEV_KEY_FILE; run_setup.sh takes JEV_API_KEY from the environment only
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "jev-1.13.0",
    "--provider",
    "jev",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "jev-1.13.0",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "TypeSafe's System One price",
    "date": "2026-10-08",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.042,
      "cache_write_input_tokens": 0.042,
      "cached_input_tokens": 0.042,
      "input_tokens": 0.042,
      "output_tokens": 0.0
    },
    "source": "https://typesafe.ai/blog/introducing-system-one-models-and-jev",
    "text": "$0.042 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "jev",
  "questions": 2163,
  "request_seconds": 377.1248158457456,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "09118bc3b46a6f43244ed63ea85bd73bd6d089d494b46ca554edfa1b538df912",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 49,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 1055762,
    "output_tokens": 127617
  }
}
Prompt template
POST https://api.typesafe.ai/v1/systemone
{
 "model": "jev-1.13.0",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · drex-max-1.1_en_dontknow

Name
Drex Max v1.1
Setup
Drex Max 1.1 · decision model · with the I don't know option F
Model
drex-max-v1.1 · provider nace
GPU
—
Cost
$0.03175 for the run · provider price × reported usage
Answer time
run drex-max-1.1_en_dontknow · first answer 2026-11-07 15:45:07 UTC · last answer 2026-11-07 16:03:19 UTC · span 1092.0 s, pauses included (not charged) · charged 1092.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://console.nace.ai/docs/drex/reference/models · read 2026-11-09
Input
text, English
Endpoint
https://console.nace.ai
Started
7 Nov 2026 15:45 UTC; last answer 7 Nov 2026 16:03 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Drex Max 1.1 (site run 326); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "drex-max-v1.1",
    "--provider",
    "nace",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "drex-max-v1.1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Nace.AI's list price (launch price $0.03 per 1M input tokens, 50% off, is what Nace bills)",
    "date": "2026-10-30",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.06,
      "cache_write_input_tokens": 0.06,
      "cached_input_tokens": 0.06,
      "input_tokens": 0.06,
      "output_tokens": 0.0
    },
    "source": "https://nace.ai/drex",
    "text": "$0.06 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "nace",
  "questions": 2163,
  "request_seconds": 1082.9731227885932,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "581a968f8b0ed6efe414e5c554f17113d2869affd68533e318718f2deee17de8",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 326,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 529144,
    "output_tokens": 406830
  }
}
Prompt template
POST https://console.nace.ai/v1/systemone
{
 "model": "drex-max-v1.1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · d1.1_en

Name
d1.1
Setup
Liquid d1.1 · decision model
Model
d1.1 · provider liquid
GPU
—
Cost
$0.006325 for the run · provider price × reported usage
Answer time
run d1.1_en · first answer 2026-11-07 15:46:16 UTC · last answer 2026-11-07 15:53:51 UTC · span 455.0 s, pauses included (not charged) · charged 455.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://docs.liquid.ai/lfm/models/decision-models · read 2026-11-09
Input
text, English
Endpoint
https://api.liquid.ai/decisions
Started
7 Nov 2026 15:46 UTC; last answer 7 Nov 2026 15:53 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Liquid d1.1 (site run 254); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "d1.1",
    "--provider",
    "liquid",
    "--timeout",
    "30",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "d1.1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Liquid AI's price for d1.1, as its API bills it",
    "date": "2026-10-24",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.02,
      "cache_write_input_tokens": 0.02,
      "cached_input_tokens": 0.02,
      "input_tokens": 0.02,
      "output_tokens": 0.0
    },
    "source": "https://www.liquid.ai/pricing",
    "text": "$0.02 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "liquid",
  "questions": 2163,
  "request_seconds": 443.88119161175564,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "23aa977a2c4b1e643101911c3ea84fd569ff7d2fe9764fd7dc7dd7b50d56384c",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 254,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 316269,
    "output_tokens": 0
  }
}
Prompt template
POST https://api.liquid.ai/decisions/v1/systemone
{
 "model": "d1.1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · d1.1_en_dontknow

Name
d1.1
Setup
Liquid d1.1 · decision model · with the I don't know option F
Model
d1.1 · provider liquid
GPU
—
Cost
$0.007471 for the run · provider price × reported usage
Answer time
run d1.1_en_dontknow · first answer 2026-11-07 15:54:00 UTC · last answer 2026-11-07 16:02:33 UTC · span 513.0 s, pauses included (not charged) · charged 513.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://docs.liquid.ai/lfm/models/decision-models · read 2026-11-09
Input
text, English
Endpoint
https://api.liquid.ai/decisions
Started
7 Nov 2026 15:54 UTC; last answer 7 Nov 2026 16:02 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Liquid d1.1 (site run 254); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "d1.1",
    "--provider",
    "liquid",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "d1.1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Liquid AI's price for d1.1, as its API bills it",
    "date": "2026-10-24",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.02,
      "cache_write_input_tokens": 0.02,
      "cached_input_tokens": 0.02,
      "input_tokens": 0.02,
      "output_tokens": 0.0
    },
    "source": "https://www.liquid.ai/pricing",
    "text": "$0.02 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "liquid",
  "questions": 2163,
  "request_seconds": 502.62377833074424,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "3ddcbfb087dc7b217b8fa0582b5d179631a952ae575fc78d55ae7ca0bf84f43b",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 254,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 373535,
    "output_tokens": 0
  }
}
Prompt template
POST https://api.liquid.ai/decisions/v1/systemone
{
 "model": "d1.1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · ms-decision-1_en

Name
Microsoft-Decision-1
Setup
Microsoft-Decision-1 · decision model
Model
microsoft/microsoft-decision-1 · provider openrouter
GPU
—
Cost
$0.01472 for the run · provider price × reported usage
Answer time
run ms-decision-1_en · first answer 2026-11-07 15:55:10 UTC · last answer 2026-11-07 16:06:52 UTC · span 702.0 s, pauses included (not charged) · charged 702.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://openrouter.ai/microsoft/microsoft-decision-1 · read 2026-11-09 · not disclosed by maker; Microsoft names only its base model (post-trained from Qwen3.5-9B) and says its weights are updated continually
Input
text, English
Endpoint
https://openrouter.ai/api
Started
7 Nov 2026 15:55 UTC; last answer 7 Nov 2026 16:06 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Microsoft-Decision-1 (site run 205); runner ffc307cb7; via OpenRouter; latency includes OpenRouter's time
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "microsoft/microsoft-decision-1",
    "--provider",
    "openrouter",
    "--timeout",
    "30",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "microsoft/microsoft-decision-1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "OpenRouter's Microsoft-Decision-1 price",
    "date": "2026-10-17",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.042,
      "cache_write_input_tokens": 0.042,
      "cached_input_tokens": 0.042,
      "input_tokens": 0.042,
      "output_tokens": 0.0
    },
    "source": "https://openrouter.ai/microsoft/microsoft-decision-1",
    "text": "$0.042 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "openrouter",
  "questions": 2163,
  "request_seconds": 693.0105550908484,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "5341341780f0f810a42c47856b0eaf5ece2a20f609e93d4b0615aa9ed3cb55d2",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 205,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 350588,
    "output_tokens": 2163
  }
}
Prompt template
POST https://openrouter.ai/api/v1/systemone
{
 "model": "microsoft/microsoft-decision-1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · subq-preview_en

Name
SubQ Snap Preview
Setup
SubQ Snap Preview · decision model
Model
subq-snap-preview · provider subq
GPU
—
Cost
$0.02321 for the run · provider price × reported usage
Answer time
run subq-preview_en · first answer 2026-11-07 16:03:08 UTC · last answer 2026-11-07 16:17:13 UTC · span 845.0 s, pauses included (not charged) · charged 845.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://docs.subq.ai/using-the-api/models/ · read 2026-11-09
Input
text, English
Endpoint
https://api.subq.ai
Started
7 Nov 2026 16:03 UTC; last answer 7 Nov 2026 16:17 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup SubQ Snap Preview (site run 177); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "subq-snap-preview",
    "--provider",
    "subq",
    "--timeout",
    "30",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "subq-snap-preview",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "SubQ's Decisions API price",
    "date": "2026-10-13",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.05,
      "cache_write_input_tokens": 0.05,
      "cached_input_tokens": 0.05,
      "input_tokens": 0.05,
      "output_tokens": 0.0
    },
    "source": "https://docs.subq.ai/using-the-api/models/",
    "text": "$0.05 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "subq",
  "questions": 2163,
  "request_seconds": 837.6735406697262,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "0ded5a1be905d6b8928ed916e653e46815fae41ee87115467d33e890ad991929",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 177,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 464254,
    "output_tokens": 0
  }
}
Prompt template
POST https://api.subq.ai/v1/systemone
{
 "model": "subq-snap-preview",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · ms-decision-1_en_dontknow

Name
Microsoft-Decision-1
Setup
Microsoft-Decision-1 · decision model · with the I don't know option F
Model
microsoft/microsoft-decision-1 · provider openrouter
GPU
—
Cost
$0.0172 for the run · provider price × reported usage
Answer time
run ms-decision-1_en_dontknow · first answer 2026-11-07 16:07:03 UTC · last answer 2026-11-07 16:18:37 UTC · span 694.0 s, pauses included (not charged) · charged 694.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://openrouter.ai/microsoft/microsoft-decision-1 · read 2026-11-09 · not disclosed by maker; Microsoft names only its base model (post-trained from Qwen3.5-9B) and says its weights are updated continually
Input
text, English
Endpoint
https://openrouter.ai/api
Started
7 Nov 2026 16:07 UTC; last answer 7 Nov 2026 16:18 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Microsoft-Decision-1 (site run 205); runner ffc307cb7; via OpenRouter; latency includes OpenRouter's time
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "microsoft/microsoft-decision-1",
    "--provider",
    "openrouter",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "microsoft/microsoft-decision-1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "OpenRouter's Microsoft-Decision-1 price",
    "date": "2026-10-17",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.042,
      "cache_write_input_tokens": 0.042,
      "cached_input_tokens": 0.042,
      "input_tokens": 0.042,
      "output_tokens": 0.0
    },
    "source": "https://openrouter.ai/microsoft/microsoft-decision-1",
    "text": "$0.042 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "openrouter",
  "questions": 2163,
  "request_seconds": 685.2481700479984,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "a4ae1ca120f541f59c8ecd00097992d993c96f822cf414c38c00e684ab8217f1",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 205,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 409448,
    "output_tokens": 2163
  }
}
Prompt template
POST https://openrouter.ai/api/v1/systemone
{
 "model": "microsoft/microsoft-decision-1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · d1_en

Name
d1
Setup
Liquid d1 · decision model
Model
d1 · provider liquid
GPU
—
Cost
$0.006325 for the run · provider price × reported usage
Answer time
run d1_en · first answer 2026-11-07 16:11:21 UTC · last answer 2026-11-07 16:19:07 UTC · span 466.0 s, pauses included (not charged) · charged 466.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://www.liquid.ai/blog/d1-decision-model · read 2026-11-09
Input
text, English
Endpoint
https://api.liquid.ai/decisions
Started
7 Nov 2026 16:11 UTC; last answer 7 Nov 2026 16:19 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Liquid d1 (site run 52); runner ffc307cb7; published as d1:free (free tier), which Liquid's API no longer serves (HTTP 404, 2 November 2026); run_english.sh's d1 lane asks for d1 with the same settings, priced at Liquid's d1 list price
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "d1",
    "--provider",
    "liquid",
    "--timeout",
    "30",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "d1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Liquid's price for d1, as its API bills it",
    "date": "2026-11-02",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.02,
      "cache_write_input_tokens": 0.02,
      "cached_input_tokens": 0.02,
      "input_tokens": 0.02,
      "output_tokens": 0.0
    },
    "source": "https://www.liquid.ai/pricing",
    "text": "$0.02 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "liquid",
  "questions": 2163,
  "request_seconds": 459.3261034428142,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "e73ffc275ce75822ec1f72b2bcec6e4eb722e80803d0b86d11fef9ae44d6a126",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 52,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 316269,
    "output_tokens": 0
  }
}
Prompt template
POST https://api.liquid.ai/decisions/v1/systemone
{
 "model": "d1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · subq-preview_en_dontknow

Name
SubQ Snap Preview
Setup
SubQ Snap Preview · decision model · with the I don't know option F
Model
subq-snap-preview · provider subq
GPU
—
Cost
$0.02646 for the run · provider price × reported usage
Answer time
run subq-preview_en_dontknow · first answer 2026-11-07 16:17:26 UTC · last answer 2026-11-07 16:31:28 UTC · span 842.0 s, pauses included (not charged) · charged 842.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://docs.subq.ai/using-the-api/models/ · read 2026-11-09
Input
text, English
Endpoint
https://api.subq.ai
Started
7 Nov 2026 16:17 UTC; last answer 7 Nov 2026 16:31 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup SubQ Snap Preview (site run 177); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "subq-snap-preview",
    "--provider",
    "subq",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "subq-snap-preview",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "SubQ's Decisions API price",
    "date": "2026-10-13",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.05,
      "cache_write_input_tokens": 0.05,
      "cached_input_tokens": 0.05,
      "input_tokens": 0.05,
      "output_tokens": 0.0
    },
    "source": "https://docs.subq.ai/using-the-api/models/",
    "text": "$0.05 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "subq",
  "questions": 2163,
  "request_seconds": 832.2515498150606,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "783bc7ffb0e97bb73ed71f70aaa777dbbc1653ac43636213b20990dd030896a2",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 177,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 529144,
    "output_tokens": 0
  }
}
Prompt template
POST https://api.subq.ai/v1/systemone
{
 "model": "subq-snap-preview",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · d1_en_dontknow

Name
d1
Setup
Liquid d1 · decision model · with the I don't know option F
Model
d1 · provider liquid
GPU
—
Cost
$0.007471 for the run · provider price × reported usage
Answer time
run d1_en_dontknow · first answer 2026-11-07 16:19:15 UTC · last answer 2026-11-07 16:26:55 UTC · span 460.0 s, pauses included (not charged) · charged 460.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://www.liquid.ai/blog/d1-decision-model · read 2026-11-09
Input
text, English
Endpoint
https://api.liquid.ai/decisions
Started
7 Nov 2026 16:19 UTC; last answer 7 Nov 2026 16:26 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Liquid d1 (site run 52); runner ffc307cb7; published as d1:free (free tier), which Liquid's API no longer serves (HTTP 404, 2 November 2026); run_english.sh's d1 lane asks for d1 with the same settings, priced at Liquid's d1 list price
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "d1",
    "--provider",
    "liquid",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "d1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Liquid's price for d1, as its API bills it",
    "date": "2026-11-02",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.02,
      "cache_write_input_tokens": 0.02,
      "cached_input_tokens": 0.02,
      "input_tokens": 0.02,
      "output_tokens": 0.0
    },
    "source": "https://www.liquid.ai/pricing",
    "text": "$0.02 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "liquid",
  "questions": 2163,
  "request_seconds": 451.77858549100347,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "dbdf159a94943ebabca0111b46639286752671f5a1f5df9cd92ef4dd44e721a1",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 52,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 373535,
    "output_tokens": 0
  }
}
Prompt template
POST https://api.liquid.ai/decisions/v1/systemone
{
 "model": "d1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · drex-2.0_en

Name
Drex 2.0
Setup
Drex 2.0 · decision model
Model
drex-v2.0 · provider nace
GPU
—
Cost
$0.01929 for the run · provider price × reported usage
Answer time
run drex-2.0_en · first answer 2026-11-07 16:22:50 UTC · last answer 2026-11-07 17:09:48 UTC · span 2818.0 s, pauses included (not charged) · charged 1074.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 35B MoE · 3B active · source https://huggingface.co/nace-ai/drex-v2.0 · read 2026-11-10 · the card states 35B parameters in total, about 3B active per token (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 35.11B / 3B active (counted from the released checkpoint (nace-ai/drex-v2.0 @ e646674): 35,107,181,936, including a vision tower of 446,571,248 that the card says is not loaded for text; mixture of experts, about 3B active per token (the card: 35B total))
Input
text, English
Endpoint
https://console.nace.ai
Started
7 Nov 2026 16:22 UTC; last answer 7 Nov 2026 17:09 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Drex 2.0 (site run 334); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "drex-v2.0",
    "--provider",
    "nace",
    "--timeout",
    "30",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "drex-v2.0",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Nace.AI's list price",
    "date": "2026-10-30",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.06,
      "cache_write_input_tokens": 0.06,
      "cached_input_tokens": 0.06,
      "input_tokens": 0.06,
      "output_tokens": 0.0
    },
    "source": "https://console.nace.ai/docs/drex/reference/pricing",
    "text": "$0.06 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "nace",
  "questions": 2163,
  "request_seconds": 1060.192734989544,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "a98681d89a87ea5fd85f0cd37fe2e456999568c14b838824414e2b505d8b98ab",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 334,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 321496,
    "output_tokens": 182338
  }
}
Prompt template
POST https://console.nace.ai/v1/systemone
{
 "model": "drex-v2.0",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · deepseek-v4-flash_en_t0

Name
DeepSeek-V4-Flash chat
Setup
deepseek-v4-flash-dspark · chat model
Model
deepseek-v4-flash-dspark
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.6377 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 3 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.638 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run deepseek-v4-flash_en_t0 · first answer 2026-11-07 17:28:52 UTC · last answer 2026-11-07 20:28:39 UTC · span 10787.0 s, pauses included (not charged) · charged 10787.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.6377 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 2.0 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 1 answer cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 285B MoE · 13B active · source https://fe-static.deepseek.com/chat/transparency/deepseek-V4-model-card-EN.pdf · read 2026-11-09 · DeepSeek's count for DeepSeek-V4-Flash (285B, 13B activated per token); the DeepSeek-V4-Flash-Vision-Exp checkpoint served here states no count of its own and adds a vision module
Input
text, English
Endpoint
internal address on a shared Spark host, path /v1/chat/completions
Started
7 Nov 2026 17:01 UTC; last answer 7 Nov 2026 20:28 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
DeepSeek-V4-Flash (shared vLLM service on two shared Spark hosts, not ours), thinking off (server default), temperature 0, English translations | INCIDENT 2026-11-07 17:01-17:30 UTC: after the reboot the runner could not store answers (responses had a new probs_json column; run_eval.py inserted without column names), so the answers generated in that window were discarded; not recoverable (vLLM logs keep no reply text). the shared Spark host's vLLM log: 362 completed chat requests from our GPU host in 17:00-17:30, an upper bound since other users of our GPU host share that address. Restarted 17:30 with the same settings; every stored answer is from that restart.
Parameters
{
  "max_tokens": 8192,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · drex-2.0_en_dontknow

Name
Drex 2.0
Setup
Drex 2.0 · decision model · with the I don't know option F
Model
drex-v2.0 · provider nace
GPU
—
Cost
$0.02305 for the run · provider price × reported usage
Answer time
run drex-2.0_en_dontknow · first answer 2026-11-07 17:09:57 UTC · last answer 2026-11-07 17:28:48 UTC · span 1131.0 s, pauses included (not charged) · charged 1131.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 35B MoE · 3B active · source https://huggingface.co/nace-ai/drex-v2.0 · read 2026-11-10 · the card states 35B parameters in total, about 3B active per token (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 35.11B / 3B active (counted from the released checkpoint (nace-ai/drex-v2.0 @ e646674): 35,107,181,936, including a vision tower of 446,571,248 that the card says is not loaded for text; mixture of experts, about 3B active per token (the card: 35B total))
Input
text, English
Endpoint
https://console.nace.ai
Started
7 Nov 2026 17:09 UTC; last answer 7 Nov 2026 17:28 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Drex 2.0 (site run 334); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "drex-v2.0",
    "--provider",
    "nace",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "drex-v2.0",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Nace.AI's list price",
    "date": "2026-10-30",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.06,
      "cache_write_input_tokens": 0.06,
      "cached_input_tokens": 0.06,
      "input_tokens": 0.06,
      "output_tokens": 0.0
    },
    "source": "https://console.nace.ai/docs/drex/reference/pricing",
    "text": "$0.06 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "nace",
  "questions": 2163,
  "request_seconds": 1116.9875631979978,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "77691347a961dcef823e79e8837953ba79a104adb1a97294f1c1533cc7da8090",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 334,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 384223,
    "output_tokens": 205806
  }
}
Prompt template
POST https://console.nace.ai/v1/systemone
{
 "model": "drex-v2.0",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · reax13_en

Name
REAX 1.3
Setup
REAX 1.3 · decision model
Model
reax13 · provider reax
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.3071 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.123 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.307. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run reax13_en · first answer 2026-11-07 17:12:29 UTC · last answer 2026-11-07 17:19:53 UTC · span 444.0 s, pauses included (not charged) · charged 444.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.3071 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 57.36B dense · source https://huggingface.co/decisionmodels/REAX-1.3 · read 2026-11-09 · counted from the checkpoint's two parts (57,359,724,636); the card states no count; Fastino's blog says 57.36B
Input
text, English
Endpoint
http://127.0.0.1:19100
Started
7 Nov 2026 17:12 UTC; last answer 7 Nov 2026 17:19 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup REAX 1.3 (site run 827); runner ffc307cb7; Docker with the model card's protections; the published run dropped no-new-privileges (an approved exception): servers.sh refuses to start it without the protections
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "reax13",
    "--provider",
    "reax",
    "--timeout",
    "1800",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "reax13",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "47f5776e71db",
  "provider": "reax",
  "questions": 2163,
  "request_seconds": 426.1396214560026,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "42a587dd2c9ddcd2e581b6b5698143a95bf24e67095e90edab83900a10c247d7",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 827,
  "status": "completed",
  "timeout": 1800.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 0,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19100/v1/systemone
{
 "model": "reax13",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · reax13_en_dontknow

Name
REAX 1.3
Setup
REAX 1.3 · decision model · with the I don't know option F
Model
reax13 · provider reax
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.3002 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.121 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.300. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run reax13_en_dontknow · first answer 2026-11-07 17:20:00 UTC · last answer 2026-11-07 17:27:14 UTC · span 434.0 s, pauses included (not charged) · charged 434.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.3002 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 57.36B dense · source https://huggingface.co/decisionmodels/REAX-1.3 · read 2026-11-09 · counted from the checkpoint's two parts (57,359,724,636); the card states no count; Fastino's blog says 57.36B
Input
text, English
Endpoint
http://127.0.0.1:19100
Started
7 Nov 2026 17:20 UTC; last answer 7 Nov 2026 17:27 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup REAX 1.3 (site run 827); runner ffc307cb7; Docker with the model card's protections; the published run dropped no-new-privileges (an approved exception): servers.sh refuses to start it without the protections
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "reax13",
    "--provider",
    "reax",
    "--timeout",
    "1800",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "reax13",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "413d6775fc58",
  "provider": "reax",
  "questions": 2163,
  "request_seconds": 417.0931140640032,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "0980d329514e1705cb8f5c2a4ffffca0526570426a01e7bd9eb1a4c59acbfe21",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 827,
  "status": "completed",
  "timeout": 1800.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 0,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19100/v1/systemone
{
 "model": "reax13",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · mammoth-1_en

Name
Mammoth-1
Setup
Mammoth-1 · decision model
Model
baselabs/mammoth-1 · provider baselabs
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.0488 for the run, list · maker's list price; RTX PRO 6000 Blackwell: $0.2248 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.0903 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.225. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run mammoth-1_en · first answer 2026-11-07 17:31:48 UTC · last answer 2026-11-07 17:37:13 UTC · span 325.0 s, pauses included (not charged) · charged 325.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.2248 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/baselabs/mammoth-1 · read 2026-11-10 · the card states a 27B parameter decision model (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 27.36B (counted from the checkpoint: language model 26,895,998,464 + vision encoder 460,730,096; the card says a 27B decision model)
Input
text, English
Endpoint
http://127.0.0.1:19150
Started
7 Nov 2026 17:31 UTC; last answer 7 Nov 2026 17:37 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Mammoth-1 (site run 374); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "baselabs/mammoth-1",
    "--provider",
    "baselabs",
    "--timeout",
    "30",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "baselabs/mammoth-1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "maker",
    "basis": "Baseten's list price for the same model",
    "date": "2026-11-01",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.1,
      "cache_write_input_tokens": 0.1,
      "cached_input_tokens": 0.1,
      "input_tokens": 0.1,
      "output_tokens": 0.0
    },
    "source": "https://www.baseten.co/library/mammoth-1/",
    "text": "$0.10 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "baselabs",
  "questions": 2163,
  "request_seconds": 317.69227257799616,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "6b47ec283c6dd07c7f8bebfd2c61469cea4a316aa0eddb1855d51e1fb2e61c10",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 374,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 488047,
    "output_tokens": 2163
  }
}
Prompt template
POST http://127.0.0.1:19150/v1/systemone
{
 "model": "baselabs/mammoth-1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · mammoth-1_en_dontknow

Name
Mammoth-1
Setup
Mammoth-1 · decision model · with the I don't know option F
Model
baselabs/mammoth-1 · provider baselabs
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.05573 for the run, list · maker's list price; RTX PRO 6000 Blackwell: $0.2117 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.085 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.212. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run mammoth-1_en_dontknow · first answer 2026-11-07 17:37:18 UTC · last answer 2026-11-07 17:42:24 UTC · span 306.0 s, pauses included (not charged) · charged 306.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.2117 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/baselabs/mammoth-1 · read 2026-11-10 · the card states a 27B parameter decision model (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 27.36B (counted from the checkpoint: language model 26,895,998,464 + vision encoder 460,730,096; the card says a 27B decision model)
Input
text, English
Endpoint
http://127.0.0.1:19150
Started
7 Nov 2026 17:37 UTC; last answer 7 Nov 2026 17:42 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Mammoth-1 (site run 374); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "baselabs/mammoth-1",
    "--provider",
    "baselabs",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "baselabs/mammoth-1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "maker",
    "basis": "Baseten's list price for the same model",
    "date": "2026-11-01",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.1,
      "cache_write_input_tokens": 0.1,
      "cached_input_tokens": 0.1,
      "input_tokens": 0.1,
      "output_tokens": 0.0
    },
    "source": "https://www.baseten.co/library/mammoth-1/",
    "text": "$0.10 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "baselabs",
  "questions": 2163,
  "request_seconds": 298.6905725330048,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "8ae1cae02b50e2fb73a7573ec2cb1957dd4382e8ecb3da43c15c4ae8e42cc7d6",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 374,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 557263,
    "output_tokens": 2163
  }
}
Prompt template
POST http://127.0.0.1:19150/v1/systemone
{
 "model": "baselabs/mammoth-1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · decider-v1.2-web_en

Name
Perplexity Decider 27B v1.2 API · with web
Setup
Perplexity Decider 27B v1.2 (API, with web) · decision model
Model
pplx-decider-v1.2 · provider perplexity
GPU
—
Cost
$2.212 for the run · provider price × reported usage
Answer time
run decider-v1.2-web_en · first answer 2026-11-07 17:41:03 UTC · last answer 2026-11-07 18:01:36 UTC · span 1233.0 s, pauses included (not charged) · charged 1233.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/perplexity-ai/pplx-decider-v1.2 · read 2026-11-10 · the official model name: 27B (pplx-decider 27B v1.2; the card states no count of its own) (turkce #4263: the maker's stated total is shown); counted from the released files: 26.09B (counted from the released checkpoint: backbone 26,085,330,160 (its 11 safetensors shards) + decision head (readout.safetensors) 1,305,600 = 26,086,635,760; the API setups use the same count)
Input
text, English
Endpoint
https://api.perplexity.ai
Started
7 Nov 2026 17:41 UTC; last answer 7 Nov 2026 18:01 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Perplexity Decider 27B v1.2 (API, with web) (site run 640); runner ffc307cb7; web grounding: one Perplexity search per question ($0.001 each)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "pplx-decider-v1.2",
    "--provider",
    "perplexity",
    "--timeout",
    "30",
    "--perplexity-mode",
    "web_grounding",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "pplx-decider-v1.2",
  "perplexity_mode": "web_grounding",
  "price": {
    "applies_to": "billed",
    "basis": "Perplexity's Decisions API price with web grounding",
    "date": "2026-11-05",
    "extra_input_per_request": 0,
    "per_request": 0.001,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.02,
      "cache_write_input_tokens": 0.02,
      "cached_input_tokens": 0.02,
      "input_tokens": 0.02,
      "output_tokens": 0.0
    },
    "source": "https://docs.perplexity.ai/docs/getting-started/pricing",
    "text": "$0.02 input, output free per 1M tokens, plus $0.001 per request (one web search)"
  },
  "prompt_version": "47f5776e71db",
  "provider": "perplexity",
  "questions": 2163,
  "request_seconds": 1067.403302654996,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "b4958d731cc8f41e897707afb64380f6d6bc774f1905fecad30e3dd3b57c8967",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 640,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 2465070,
    "output_tokens": 2163
  }
}
Prompt template
POST https://api.perplexity.ai/v1/decisions
{
 "model": "pplx-decider-v1.2",
 "mode": "web_grounding",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · d3_en

Name
d3
Setup
vLLM-SR d3 · decision model
Model
vllm-sr/d3 · provider vllm-sr
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.08784 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0353 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.088. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run d3_en · first answer 2026-11-07 17:47:43 UTC · last answer 2026-11-07 17:49:50 UTC · span 127.0 s, pauses included (not charged) · charged 127.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0878 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 26.09B dense · source https://huggingface.co/vllm-sr/d3 · read 2026-11-09 · counted from the released checkpoint: backbone 26,085,330,160 (its 11 safetensors shards) + decision head (readout.safetensors) 1,305,600 = 26,086,635,760
Input
text, English
Endpoint
http://127.0.0.1:19120
Started
7 Nov 2026 17:47 UTC; last answer 7 Nov 2026 17:49 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup vLLM-SR d3 (site run 259); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "vllm-sr/d3",
    "--provider",
    "vllm-sr",
    "--timeout",
    "30",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "vllm-sr/d3",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "47f5776e71db",
  "provider": "vllm-sr",
  "questions": 2163,
  "request_seconds": 121.60685011798978,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "311e26255dc2b2856dee8d35cbe20818423e8b81401a588878e32bf84eb62e52",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 259,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 481558,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19120/v1/systemone
{
 "model": "vllm-sr/d3",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · d3_en_dontknow

Name
d3
Setup
vLLM-SR d3 · decision model · with the I don't know option F
Model
vllm-sr/d3 · provider vllm-sr
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.09407 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0378 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.094. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run d3_en_dontknow · first answer 2026-11-07 17:49:54 UTC · last answer 2026-11-07 17:52:10 UTC · span 136.0 s, pauses included (not charged) · charged 136.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0941 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 26.09B dense · source https://huggingface.co/vllm-sr/d3 · read 2026-11-09 · counted from the released checkpoint: backbone 26,085,330,160 (its 11 safetensors shards) + decision head (readout.safetensors) 1,305,600 = 26,086,635,760
Input
text, English
Endpoint
http://127.0.0.1:19120
Started
7 Nov 2026 17:49 UTC; last answer 7 Nov 2026 17:52 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup vLLM-SR d3 (site run 259); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "vllm-sr/d3",
    "--provider",
    "vllm-sr",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "vllm-sr/d3",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "413d6775fc58",
  "provider": "vllm-sr",
  "questions": 2163,
  "request_seconds": 129.7481250770079,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "2d854146c0b0b41b451e78294132ce47b73db6f4e97c0bbc67afb0ba2bc46ad4",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 259,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 546448,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19120/v1/systemone
{
 "model": "vllm-sr/d3",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · d3-ultra_en_dontknow

Name
d3-ultra
Setup
vLLM-SR d3-ultra · decision model · with the I don't know option F
Model
vllm-sr/d3-ultra · provider d3-server
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.1957 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0786 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.196. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run d3-ultra_en_dontknow · first answer 2026-11-07 17:55:30 UTC · last answer 2026-11-07 18:00:13 UTC · span 283.0 s, pauses included (not charged) · charged 283.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1957 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 26.09B dense · source https://huggingface.co/vllm-sr/d3-ultra · read 2026-11-09 · counted from the released checkpoint: backbone 26,085,330,160 (its 11 safetensors shards) + decision head (readout.safetensors) 1,305,600 = 26,086,635,760
Input
text, English
Endpoint
http://127.0.0.1:19130
Started
7 Nov 2026 17:55 UTC; last answer 7 Nov 2026 18:00 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup vLLM-SR d3-ultra (site run 269); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "vllm-sr/d3-ultra",
    "--provider",
    "d3-server",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "vllm-sr/d3-ultra",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "413d6775fc58",
  "provider": "d3-server",
  "questions": 2163,
  "request_seconds": 276.0074641920164,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "861af6f2981507d1e3355bc96aa248f64ea79bb72ae91f9997feef7bb016a53d",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 269,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 750723,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19130/v1/systemone
{
 "model": "vllm-sr/d3-ultra",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · decider-v1.2-web_en_dontknow

Name
Perplexity Decider 27B v1.2 API · with web
Setup
Perplexity Decider 27B v1.2 (API, with web) · decision model · with the I don't know option F
Model
pplx-decider-v1.2 · provider perplexity
GPU
—
Cost
$2.211 for the run · provider price × reported usage
Answer time
run decider-v1.2-web_en_dontknow · first answer 2026-11-07 18:01:44 UTC · last answer 2026-11-07 18:21:49 UTC · span 1205.0 s, pauses included (not charged) · charged 1205.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/perplexity-ai/pplx-decider-v1.2 · read 2026-11-10 · the official model name: 27B (pplx-decider 27B v1.2; the card states no count of its own) (turkce #4263: the maker's stated total is shown); counted from the released files: 26.09B (counted from the released checkpoint: backbone 26,085,330,160 (its 11 safetensors shards) + decision head (readout.safetensors) 1,305,600 = 26,086,635,760; the API setups use the same count)
Input
text, English
Endpoint
https://api.perplexity.ai
Started
7 Nov 2026 18:01 UTC; last answer 7 Nov 2026 18:21 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Perplexity Decider 27B v1.2 (API, with web) (site run 640); runner ffc307cb7; web grounding: one Perplexity search per question ($0.001 each)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "pplx-decider-v1.2",
    "--provider",
    "perplexity",
    "--timeout",
    "30",
    "--perplexity-mode",
    "web_grounding",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "pplx-decider-v1.2",
  "perplexity_mode": "web_grounding",
  "price": {
    "applies_to": "billed",
    "basis": "Perplexity's Decisions API price with web grounding",
    "date": "2026-11-05",
    "extra_input_per_request": 0,
    "per_request": 0.001,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.02,
      "cache_write_input_tokens": 0.02,
      "cached_input_tokens": 0.02,
      "input_tokens": 0.02,
      "output_tokens": 0.0
    },
    "source": "https://docs.perplexity.ai/docs/getting-started/pricing",
    "text": "$0.02 input, output free per 1M tokens, plus $0.001 per request (one web search)"
  },
  "prompt_version": "413d6775fc58",
  "provider": "perplexity",
  "questions": 2163,
  "request_seconds": 1191.1350380209851,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "f00e67fc93c577cedc2f700763d370d72156a50c1c496c6abab63c11021be04a",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 640,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 2395702,
    "output_tokens": 2163
  }
}
Prompt template
POST https://api.perplexity.ai/v1/decisions
{
 "model": "pplx-decider-v1.2",
 "mode": "web_grounding",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · clef_en

Name
Clef
Setup
Clef · decision model
Model
clef · provider clef
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.1483 for the run, list · maker's list price; RTX PRO 6000 Blackwell: $0.2843 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.114 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.284. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run clef_en · first answer 2026-11-07 18:08:37 UTC · last answer 2026-11-07 18:15:28 UTC · span 411.0 s, pauses included (not charged) · charged 411.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.2843 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/Cloudflare/clef · read 2026-11-10 · the card states a 27B model (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 27.48B (counted from the checkpoint: backbone 27,356,728,560 + joint head 128,056,324; the card says a 27B model)
Input
text, English
Endpoint
http://127.0.0.1:19200
Started
7 Nov 2026 18:08 UTC; last answer 7 Nov 2026 18:15 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Clef (site run 125); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "clef",
    "--provider",
    "clef",
    "--timeout",
    "120",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "clef",
  "perplexity_mode": null,
  "price": {
    "applies_to": "maker",
    "basis": "Cloudflare's list price for the same model",
    "date": "2026-10-18",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.24,
      "cache_write_input_tokens": 0.24,
      "cached_input_tokens": 0.24,
      "input_tokens": 0.24,
      "output_tokens": 0.0
    },
    "source": "https://developers.cloudflare.com/workers-ai/models/clef/",
    "text": "$0.24 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "clef",
  "questions": 2163,
  "request_seconds": 402.72090733298955,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "ae4776017cd86ebb9d976ed97d240f7e9ccb38d79f4813159a308856a66c70ac",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 125,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 617719,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19200/v1/systemone
{
 "model": "clef",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · clef_en_dontknow

Name
Clef
Setup
Clef · decision model · with the I don't know option F
Model
clef · provider clef
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.169 for the run, list · maker's list price; RTX PRO 6000 Blackwell: $0.2781 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.112 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.278. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run clef_en_dontknow · first answer 2026-11-07 18:15:35 UTC · last answer 2026-11-07 18:22:17 UTC · span 402.0 s, pauses included (not charged) · charged 402.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.2781 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/Cloudflare/clef · read 2026-11-10 · the card states a 27B model (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 27.48B (counted from the checkpoint: backbone 27,356,728,560 + joint head 128,056,324; the card says a 27B model)
Input
text, English
Endpoint
http://127.0.0.1:19200
Started
7 Nov 2026 18:15 UTC; last answer 7 Nov 2026 18:22 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Clef (site run 125); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "clef",
    "--provider",
    "clef",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "clef",
  "perplexity_mode": null,
  "price": {
    "applies_to": "maker",
    "basis": "Cloudflare's list price for the same model",
    "date": "2026-10-18",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.24,
      "cache_write_input_tokens": 0.24,
      "cached_input_tokens": 0.24,
      "input_tokens": 0.24,
      "output_tokens": 0.0
    },
    "source": "https://developers.cloudflare.com/workers-ai/models/clef/",
    "text": "$0.24 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "clef",
  "questions": 2163,
  "request_seconds": 391.83673584496773,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "94dd8bf939e3bb7e2f6ea03b3442a77726235d9ce108a3aec6da1271f887b0c7",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 125,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 704239,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19200/v1/systemone
{
 "model": "clef",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · onejev_en

Name
OneJev-27B
Setup
OneJev 27B · decision model
Model
OmniJev/OneJev-27B · provider onejev
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.1314 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0528 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.131. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run onejev_en · first answer 2026-11-07 18:27:58 UTC · last answer 2026-11-07 18:31:08 UTC · span 190.0 s, pauses included (not charged) · charged 190.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1314 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/OmniJev/OneJev-27B · read 2026-11-10 · the card states the 27B model of OneJev (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 27.36B (counted from the checkpoint: its index total_parameters 27,356,728,560)
Input
text, English
Endpoint
http://127.0.0.1:19190
Started
7 Nov 2026 18:27 UTC; last answer 7 Nov 2026 18:31 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup OneJev 27B (site run 264); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "OmniJev/OneJev-27B",
    "--provider",
    "onejev",
    "--timeout",
    "30",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "OmniJev/OneJev-27B",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "47f5776e71db",
  "provider": "onejev",
  "questions": 2163,
  "request_seconds": 182.2895098160152,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "d7f2b6008bf1adb09b4bcc1fd0e6a381b8a3e76e295488214d80434d943d8171",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 264,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 488047,
    "output_tokens": 2163
  }
}
Prompt template
POST http://127.0.0.1:19190/v1/systemone
{
 "model": "OmniJev/OneJev-27B",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · onejev_en_dontknow

Name
OneJev-27B
Setup
OneJev 27B · decision model · with the I don't know option F
Model
OmniJev/OneJev-27B · provider onejev
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.1425 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0572 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.142. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run onejev_en_dontknow · first answer 2026-11-07 18:31:12 UTC · last answer 2026-11-07 18:34:38 UTC · span 206.0 s, pauses included (not charged) · charged 206.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1425 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/OmniJev/OneJev-27B · read 2026-11-10 · the card states the 27B model of OneJev (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 27.36B (counted from the checkpoint: its index total_parameters 27,356,728,560)
Input
text, English
Endpoint
http://127.0.0.1:19190
Started
7 Nov 2026 18:31 UTC; last answer 7 Nov 2026 18:34 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup OneJev 27B (site run 264); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "OmniJev/OneJev-27B",
    "--provider",
    "onejev",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "OmniJev/OneJev-27B",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "413d6775fc58",
  "provider": "onejev",
  "questions": 2163,
  "request_seconds": 199.55804775201614,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "9b5b31868b87143ebd2f7ec17ebbbd12a137a2925c4ebe876b233bf588408ad8",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 264,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 557263,
    "output_tokens": 2163
  }
}
Prompt template
POST http://127.0.0.1:19190/v1/systemone
{
 "model": "OmniJev/OneJev-27B",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · decider-v1.2_en

Name
Perplexity Decider 27B v1.2 API
Setup
Perplexity Decider 27B v1.2 (API) · decision model
Model
pplx-decider-v1.2 · provider perplexity
GPU
—
Cost
$0.009285 for the run · provider price × reported usage
Answer time
run decider-v1.2_en · first answer 2026-11-07 18:34:40 UTC · last answer 2026-11-07 18:44:37 UTC · span 597.0 s, pauses included (not charged) · charged 597.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/perplexity-ai/pplx-decider-v1.2 · read 2026-11-10 · the official model name: 27B (pplx-decider 27B v1.2; the card states no count of its own) (turkce #4263: the maker's stated total is shown); counted from the released files: 26.09B (counted from the released checkpoint: backbone 26,085,330,160 (its 11 safetensors shards) + decision head (readout.safetensors) 1,305,600 = 26,086,635,760; the API setups use the same count)
Input
text, English
Endpoint
https://api.perplexity.ai
Started
7 Nov 2026 18:34 UTC; last answer 7 Nov 2026 18:44 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Perplexity Decider 27B v1.2 (API) (site run 633); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "pplx-decider-v1.2",
    "--provider",
    "perplexity",
    "--timeout",
    "30",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "pplx-decider-v1.2",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Perplexity's Decisions API price",
    "date": "2026-11-05",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.02,
      "cache_write_input_tokens": 0.02,
      "cached_input_tokens": 0.02,
      "input_tokens": 0.02,
      "output_tokens": 0.0
    },
    "source": "https://docs.perplexity.ai/docs/getting-started/pricing",
    "text": "$0.02 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "perplexity",
  "questions": 2163,
  "request_seconds": 585.5249645239955,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "c1c490e7240dddb183889efe99095565b315737664fda38f2448b96de2583723",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 633,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 464254,
    "output_tokens": 2163
  }
}
Prompt template
POST https://api.perplexity.ai/v1/decisions
{
 "model": "pplx-decider-v1.2",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · elephant-1_en

Name
Elephant-1
Setup
Elephant-1 · decision model
Model
baselabs/elephant-1 · provider baselabs
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.01708 for the run, list · maker's list price; RTX PRO 6000 Blackwell: $0.1259 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.0506 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.126. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run elephant-1_en · first answer 2026-11-07 18:38:44 UTC · last answer 2026-11-07 18:41:46 UTC · span 182.0 s, pauses included (not charged) · charged 182.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1259 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/baselabs/elephant-1 · read 2026-11-10 · the card states a 9B parameter decision model (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 9.41B (counted from the checkpoint: language model 8,953,803,264 + vision encoder 456,010,480; the card says a 9B decision model)
Input
text, English
Endpoint
http://127.0.0.1:19160
Started
7 Nov 2026 18:38 UTC; last answer 7 Nov 2026 18:41 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Elephant-1 (site run 373); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "baselabs/elephant-1",
    "--provider",
    "baselabs",
    "--timeout",
    "30",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "baselabs/elephant-1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "maker",
    "basis": "Baseten's list price for the same model",
    "date": "2026-11-01",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.035,
      "cache_write_input_tokens": 0.035,
      "cached_input_tokens": 0.035,
      "input_tokens": 0.035,
      "output_tokens": 0.0
    },
    "source": "https://www.baseten.co/library/elephant-1/",
    "text": "$0.035 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "baselabs",
  "questions": 2163,
  "request_seconds": 175.8580534130042,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "f01b01578bd8fdf92832389a28b2f8a667c106182386daa816f1adbd2f2926f1",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 373,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 488047,
    "output_tokens": 2163
  }
}
Prompt template
POST http://127.0.0.1:19160/v1/systemone
{
 "model": "baselabs/elephant-1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · elephant-1_en_dontknow

Name
Elephant-1
Setup
Elephant-1 · decision model · with the I don't know option F
Model
baselabs/elephant-1 · provider baselabs
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.0195 for the run, list · maker's list price; RTX PRO 6000 Blackwell: $0.1259 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.0506 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.126. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run elephant-1_en_dontknow · first answer 2026-11-07 18:41:50 UTC · last answer 2026-11-07 18:44:52 UTC · span 182.0 s, pauses included (not charged) · charged 182.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1259 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/baselabs/elephant-1 · read 2026-11-10 · the card states a 9B parameter decision model (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 9.41B (counted from the checkpoint: language model 8,953,803,264 + vision encoder 456,010,480; the card says a 9B decision model)
Input
text, English
Endpoint
http://127.0.0.1:19160
Started
7 Nov 2026 18:41 UTC; last answer 7 Nov 2026 18:44 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Elephant-1 (site run 373); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "baselabs/elephant-1",
    "--provider",
    "baselabs",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "baselabs/elephant-1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "maker",
    "basis": "Baseten's list price for the same model",
    "date": "2026-11-01",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.035,
      "cache_write_input_tokens": 0.035,
      "cached_input_tokens": 0.035,
      "input_tokens": 0.035,
      "output_tokens": 0.0
    },
    "source": "https://www.baseten.co/library/elephant-1/",
    "text": "$0.035 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "baselabs",
  "questions": 2163,
  "request_seconds": 176.38249023901153,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "a32b666e9cb5c84ea9b14a0f5dffae53ee5a6c22872cb6321d8188e64fa5671f",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 373,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 557263,
    "output_tokens": 2163
  }
}
Prompt template
POST http://127.0.0.1:19160/v1/systemone
{
 "model": "baselabs/elephant-1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · decider-v1.2_en_dontknow

Name
Perplexity Decider 27B v1.2 API
Setup
Perplexity Decider 27B v1.2 (API) · decision model · with the I don't know option F
Model
pplx-decider-v1.2 · provider perplexity
GPU
—
Cost
$0.01058 for the run · provider price × reported usage
Answer time
run decider-v1.2_en_dontknow · first answer 2026-11-07 18:44:45 UTC · last answer 2026-11-07 18:54:58 UTC · span 613.0 s, pauses included (not charged) · charged 613.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/perplexity-ai/pplx-decider-v1.2 · read 2026-11-10 · the official model name: 27B (pplx-decider 27B v1.2; the card states no count of its own) (turkce #4263: the maker's stated total is shown); counted from the released files: 26.09B (counted from the released checkpoint: backbone 26,085,330,160 (its 11 safetensors shards) + decision head (readout.safetensors) 1,305,600 = 26,086,635,760; the API setups use the same count)
Input
text, English
Endpoint
https://api.perplexity.ai
Started
7 Nov 2026 18:44 UTC; last answer 7 Nov 2026 18:54 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Perplexity Decider 27B v1.2 (API) (site run 633); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "pplx-decider-v1.2",
    "--provider",
    "perplexity",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "pplx-decider-v1.2",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Perplexity's Decisions API price",
    "date": "2026-11-05",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.02,
      "cache_write_input_tokens": 0.02,
      "cached_input_tokens": 0.02,
      "input_tokens": 0.02,
      "output_tokens": 0.0
    },
    "source": "https://docs.perplexity.ai/docs/getting-started/pricing",
    "text": "$0.02 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "perplexity",
  "questions": 2163,
  "request_seconds": 599.4271231350249,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "430ddfcd1cd6a4a1263f79f21829581f1e062fa73633cb00e8631591e8de5a09",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 633,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 529144,
    "output_tokens": 2163
  }
}
Prompt template
POST https://api.perplexity.ai/v1/decisions
{
 "model": "pplx-decider-v1.2",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · open-jev_en

Name
Open-Jev-27B-v1.1
Setup
Open-Jev 27B v1.1 · decision model
Model
open-jev-27b-v1.1 · provider open-jev
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.3659 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.147 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.366. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run open-jev_en · first answer 2026-11-07 19:04:03 UTC · last answer 2026-11-07 19:12:52 UTC · span 529.0 s, pauses included (not charged) · charged 529.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.3659 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/ZefanCai/Open-Jev-27B-v1.1 · read 2026-11-10 · the official model name: Open-Jev 27B (turkce #4263: the maker's stated total is shown); counted from the released files: 27.80B (the model as run: Qwen3.8-27B 27,781,427,952 + its released adapter and head 15,471,617 = 27,796,899,569 (the Open-Jev repo holds only the adapter and head))
Input
text, English
Endpoint
http://127.0.0.1:19230
Started
7 Nov 2026 19:04 UTC; last answer 7 Nov 2026 19:12 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Open-Jev 27B v1.1 (site run 45); runner ffc307cb7; server --max-length 4096 tokens
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "open-jev-27b-v1.1",
    "--provider",
    "open-jev",
    "--timeout",
    "30",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "open-jev-27b-v1.1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "47f5776e71db",
  "provider": "open-jev",
  "questions": 2163,
  "request_seconds": 514.2586951239655,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "4849a0e0bd472bcce47cb7631b4c250bbfe6530b18fa1fbac7c964c4266a036d",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 45,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 1547430,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19230/v1/systemone
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · open-jev_en_dontknow

Name
Open-Jev-27B-v1.1
Setup
Open-Jev 27B v1.1 · decision model · with the I don't know option F
Model
open-jev-27b-v1.1 · provider open-jev
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.4496 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.181 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.450. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run open-jev_en_dontknow · first answer 2026-11-07 19:13:01 UTC · last answer 2026-11-07 19:23:51 UTC · span 650.0 s, pauses included (not charged) · charged 650.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.4496 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/ZefanCai/Open-Jev-27B-v1.1 · read 2026-11-10 · the official model name: Open-Jev 27B (turkce #4263: the maker's stated total is shown); counted from the released files: 27.80B (the model as run: Qwen3.8-27B 27,781,427,952 + its released adapter and head 15,471,617 = 27,796,899,569 (the Open-Jev repo holds only the adapter and head))
Input
text, English
Endpoint
http://127.0.0.1:19230
Started
7 Nov 2026 19:13 UTC; last answer 7 Nov 2026 19:23 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Open-Jev 27B v1.1 (site run 45); runner ffc307cb7; server --max-length 4096 tokens
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "open-jev-27b-v1.1",
    "--provider",
    "open-jev",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "open-jev-27b-v1.1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "413d6775fc58",
  "provider": "open-jev",
  "questions": 2163,
  "request_seconds": 641.5158790110145,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "54a4ff51ce41a1301e832ee0ee2f6fae780f9214c2166e4242629869e608b34a",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 45,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 2127533,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19230/v1/systemone
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · cygnet_en

Name
Cygnet
Setup
Cygnet · decision model
Model
cygnet · provider cygnet
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.05179 for the run, list · public host's list price (same weights); RTX PRO 6000 Blackwell: $0.05948 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.0239 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.059. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run cygnet_en · first answer 2026-11-07 19:28:53 UTC · last answer 2026-11-07 19:30:19 UTC · span 86.0 s, pauses included (not charged) · charged 86.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0595 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 11.95B dense · source https://huggingface.co/google/gemma-4-12B-it · read 2026-11-10 · the card states Gemma 4 12B IT unchanged; its card: 11.95B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 11.96B (counted from the checkpoint (11,959,730,224): Gemma 4 12B IT unchanged, whose card says 11.95B)
Input
text, English
Endpoint
http://127.0.0.1:19220
Started
7 Nov 2026 19:28 UTC; last answer 7 Nov 2026 19:30 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Cygnet (site run 127); runner ffc307cb7; vLLM 0.30.0 container (port + 5) and cygnet-recipe's shim
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "cygnet",
    "--provider",
    "cygnet",
    "--timeout",
    "180",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "cygnet",
  "perplexity_mode": null,
  "price": {
    "applies_to": "weights",
    "basis": "SiliconFlow's list price for the same weights (Gemma 4 12B IT; precision not stated, ours BF16)",
    "date": "2026-11-05",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.1,
      "cache_write_input_tokens": 0.1,
      "cached_input_tokens": 0.1,
      "input_tokens": 0.1,
      "output_tokens": 0.3
    },
    "source": "https://www.siliconflow.com/models/gemma-4-12b-it",
    "text": "$0.10 input, $0.30 output per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "cygnet",
  "questions": 2163,
  "request_seconds": 80.81341893093486,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "87919baae3619df887805282fe363e83427318f3a4b26075520b5160fe0c78e0",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 127,
  "status": "completed",
  "timeout": 180.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 511422,
    "output_tokens": 2163
  }
}
Prompt template
POST http://127.0.0.1:19220/v1/systemone
{
 "model": "cygnet",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · cygnet_en_dontknow

Name
Cygnet
Setup
Cygnet · decision model · with the I don't know option F
Model
cygnet · provider cygnet
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.0585 for the run, list · public host's list price (same weights); RTX PRO 6000 Blackwell: $0.06087 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.0244 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.061. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run cygnet_en_dontknow · first answer 2026-11-07 19:30:23 UTC · last answer 2026-11-07 19:31:51 UTC · span 88.0 s, pauses included (not charged) · charged 88.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0609 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 11.95B dense · source https://huggingface.co/google/gemma-4-12B-it · read 2026-11-10 · the card states Gemma 4 12B IT unchanged; its card: 11.95B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 11.96B (counted from the checkpoint (11,959,730,224): Gemma 4 12B IT unchanged, whose card says 11.95B)
Input
text, English
Endpoint
http://127.0.0.1:19220
Started
7 Nov 2026 19:30 UTC; last answer 7 Nov 2026 19:31 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Cygnet (site run 127); runner ffc307cb7; vLLM 0.30.0 container (port + 5) and cygnet-recipe's shim
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "cygnet",
    "--provider",
    "cygnet",
    "--timeout",
    "180",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "cygnet",
  "perplexity_mode": null,
  "price": {
    "applies_to": "weights",
    "basis": "SiliconFlow's list price for the same weights (Gemma 4 12B IT; precision not stated, ours BF16)",
    "date": "2026-11-05",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.1,
      "cache_write_input_tokens": 0.1,
      "cached_input_tokens": 0.1,
      "input_tokens": 0.1,
      "output_tokens": 0.3
    },
    "source": "https://www.siliconflow.com/models/gemma-4-12b-it",
    "text": "$0.10 input, $0.30 output per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "cygnet",
  "questions": 2163,
  "request_seconds": 83.26958296496923,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "528442b64c52197ee50a81521fbdb7a88144258b7b9275ba0b62c836405c630f",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 127,
  "status": "completed",
  "timeout": 180.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 578475,
    "output_tokens": 2163
  }
}
Prompt template
POST http://127.0.0.1:19220/v1/systemone
{
 "model": "cygnet",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · winnow_en

Name
Winnow-12B
Setup
Winnow-12B · decision model
Model
winnow-12b · provider winnow
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.083 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0333 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.083. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run winnow_en · first answer 2026-11-07 19:34:53 UTC · last answer 2026-11-07 19:36:53 UTC · span 120.0 s, pauses included (not charged) · charged 120.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0830 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 12B dense · source https://huggingface.co/EldanRing/Winnow-12B · read 2026-11-10 · the official model name: Winnow-12B (turkce #4263: the maker's stated total is shown); counted from the released files: 11.91B (counted from the Q8_0 GGUF checkpoint (11,907,350,576); a fine-tune of Gemma 4 12B IT, whose card says 11.95B)
Input
text, English
Endpoint
http://127.0.0.1:19240
Started
7 Nov 2026 19:34 UTC; last answer 7 Nov 2026 19:36 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Winnow-12B (site run 128); runner ffc307cb7; server context 8,192 tokens
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "winnow-12b",
    "--provider",
    "winnow",
    "--timeout",
    "180",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "winnow-12b",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "47f5776e71db",
  "provider": "winnow",
  "questions": 2163,
  "request_seconds": 115.08058561700636,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "e8d76f1ee2b779328b82b43ab3610f26b23a81e36ddd37eaf4c8b394912efa58",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 128,
  "status": "completed",
  "timeout": 180.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 548554,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19240/v1/systemone
{
 "model": "winnow-12b",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · winnow_en_dontknow

Name
Winnow-12B
Setup
Winnow-12B · decision model · with the I don't know option F
Model
winnow-12b · provider winnow
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.08853 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0356 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.089. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run winnow_en_dontknow · first answer 2026-11-07 19:36:57 UTC · last answer 2026-11-07 19:39:05 UTC · span 128.0 s, pauses included (not charged) · charged 128.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0885 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 12B dense · source https://huggingface.co/EldanRing/Winnow-12B · read 2026-11-10 · the official model name: Winnow-12B (turkce #4263: the maker's stated total is shown); counted from the released files: 11.91B (counted from the Q8_0 GGUF checkpoint (11,907,350,576); a fine-tune of Gemma 4 12B IT, whose card says 11.95B)
Input
text, English
Endpoint
http://127.0.0.1:19240
Started
7 Nov 2026 19:36 UTC; last answer 7 Nov 2026 19:39 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Winnow-12B (site run 128); runner ffc307cb7; server context 8,192 tokens
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "winnow-12b",
    "--provider",
    "winnow",
    "--timeout",
    "180",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "winnow-12b",
  "perplexity_mode": null,
  "price": {
    "applies_to": "gpu",
    "basis": "Runpod's on-demand price for an RTX PRO 6000 (96 GB) on Secure Cloud",
    "change": null,
    "date": "2026-11-05",
    "gpu": "RTX PRO 6000 Blackwell (96 GB)",
    "gpus": 1,
    "source": "https://www.runpod.io/pricing",
    "text": "$2.49 per GPU-hour, 1 GPU, for the run's request time",
    "usd_per_gpu_hour": 2.49
  },
  "prompt_version": "413d6775fc58",
  "provider": "winnow",
  "questions": 2163,
  "request_seconds": 123.55064278500322,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "3aba2a8f0f3910b9fe96fb0e736a016cb180b8f43fa554e9e966248a9b51ecca",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 128,
  "status": "completed",
  "timeout": 180.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 625976,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19240/v1/systemone
{
 "model": "winnow-12b",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · clef-flash_en

Name
Clef-Flash
Setup
Clef-flash · decision model
Model
clef-flash · provider clef
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.02347 for the run, list · maker's list price; RTX PRO 6000 Blackwell: $0.1556 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.0625 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.156. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run clef-flash_en · first answer 2026-11-07 19:43:58 UTC · last answer 2026-11-07 19:47:43 UTC · span 225.0 s, pauses included (not charged) · charged 225.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1556 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/Cloudflare/clef-flash · read 2026-11-10 · the card states a 9B model (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 9.53B (counted from the checkpoint: backbone 9,409,813,744 + joint head; the card says a 9B model)
Input
text, English
Endpoint
http://127.0.0.1:19210
Started
7 Nov 2026 19:43 UTC; last answer 7 Nov 2026 19:47 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Clef-flash (site run 139); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "clef-flash",
    "--provider",
    "clef",
    "--timeout",
    "60",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "clef-flash",
  "perplexity_mode": null,
  "price": {
    "applies_to": "maker",
    "basis": "Cloudflare's list price for the same model",
    "date": "2026-10-18",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.038,
      "cache_write_input_tokens": 0.038,
      "cached_input_tokens": 0.038,
      "input_tokens": 0.038,
      "output_tokens": 0.0
    },
    "source": "https://developers.cloudflare.com/workers-ai/models/clef-flash/",
    "text": "$0.038 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "clef",
  "questions": 2163,
  "request_seconds": 220.3990778950556,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "09acd031605cbf1aac613e430635473bae03202766f37995c98e65e641b29ab6",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 139,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 617719,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19210/v1/systemone
{
 "model": "clef-flash",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · clef-flash_en_dontknow

Name
Clef-Flash
Setup
Clef-flash · decision model · with the I don't know option F
Model
clef-flash · provider clef
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.02676 for the run, list · maker's list price; RTX PRO 6000 Blackwell: $0.1549 (estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h) · GPU time, at most: 0.0622 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.155. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run clef-flash_en_dontknow · first answer 2026-11-07 19:47:48 UTC · last answer 2026-11-07 19:51:32 UTC · span 224.0 s, pauses included (not charged) · charged 224.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1549 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/Cloudflare/clef-flash · read 2026-11-10 · the card states a 9B model (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 9.53B (counted from the checkpoint: backbone 9,409,813,744 + joint head; the card says a 9B model)
Input
text, English
Endpoint
http://127.0.0.1:19210
Started
7 Nov 2026 19:47 UTC; last answer 7 Nov 2026 19:51 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Clef-flash (site run 139); runner ffc307cb7
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "clef-flash",
    "--provider",
    "clef",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "clef-flash",
  "perplexity_mode": null,
  "price": {
    "applies_to": "maker",
    "basis": "Cloudflare's list price for the same model",
    "date": "2026-10-18",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.038,
      "cache_write_input_tokens": 0.038,
      "cached_input_tokens": 0.038,
      "input_tokens": 0.038,
      "output_tokens": 0.0
    },
    "source": "https://developers.cloudflare.com/workers-ai/models/clef-flash/",
    "text": "$0.038 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "clef",
  "questions": 2163,
  "request_seconds": 213.65449710295798,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "ec65c4fb5c147688ad3f6d194f0793b1fb672146457dec46ebc4336c69d45a04",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": 139,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 704239,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19210/v1/systemone
{
 "model": "clef-flash",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · qwen3.5-397b-a17b_en

Name
Qwen3.5-397B-A17B on · chat
Setup
Qwen3.5-397B-A17B · chat model
Model
qwen/qwen3.5-397b-a17b · provider openrouter
GPU
—
Cost
$0.1388 for the run · provider price × reported usage
Answer time
run qwen3.5-397b-a17b_en · first answer 2026-11-08 10:41:20 UTC · last answer 2026-11-08 10:44:35 UTC · span 195.0 s, pauses included (not charged) · charged 195.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 5.1 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 397B MoE · 17B active · from the run's notes (Tinker models table); also https://huggingface.co/Qwen/Qwen3.5-397B-A17B · read 2026-11-09
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
8 Nov 2026 10:41 UTC; last answer 8 Nov 2026 10:44 UTC
Progress
25 answered of 2,163 asked; 0 request errors; in progress
Notes
Qwen3.5-397B-A17B through OpenRouter (qwen/qwen3.5-397b-a17b), provider pinned to alibaba, fallbacks off; reasoning {"enabled": true}; temperature 1.0; max_tokens 32768; one question per request; parameters 17B / 397B active / total (Tinker models table). English translations.
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "alibaba"
    ]
  },
  "reasoning": {
    "enabled": true
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · minimax-m3.1-flash_en

Name
MiniMax-M3.1-Flash high · chat
Setup
MiniMax-M3.1-Flash · chat model
Model
minimax/minimax-m3.1-flash · provider openrouter
GPU
—
Cost
$0.2225 for the run · provider price × reported usage
Answer time
run minimax-m3.1-flash_en · first answer 2026-11-08 10:42:16 UTC · last answer 2026-11-08 16:27:20 UTC · span 20704.0 s, pauses included (not charged) · charged 1052.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 7.2 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 428B MoE · 23B active · from the run's notes (NVIDIA NIM model card); also https://huggingface.co/MiniMaxAI/MiniMax-M3.1-Flash · read 2026-11-09 · the card: approximately 428B total, 23B activated per token
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
8 Nov 2026 10:42 UTC; last answer 8 Nov 2026 16:27 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
MiniMax-M3.1-Flash through OpenRouter (minimax/minimax-m3.1-flash), provider pinned to minimax, fallbacks off; reasoning {"effort": "high"}; temperature 1.0; max_tokens 32768; one question per request; parameters 23B / 428B active / total (NVIDIA NIM model card). English translations. One reply came back broken (ending "WER: D" instead of "ANSWER: D"); its clearly readable letter D was scored by hand (Umit #3215).
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "minimax"
    ]
  },
  "reasoning": {
    "effort": "high"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · deepseek-v4.1-flash-1017_en

Name
DeepSeek-V4.1-Flash high · chat · 1017
Setup
DeepSeek-V4.1-Flash 1017 · chat model
Model
deepseek/deepseek-v4.1-flash-1017 · provider openrouter
GPU
—
Cost
$0.4919 for the run · provider price × reported usage
Answer time
run deepseek-v4.1-flash-1017_en · first answer 2026-11-08 10:44:33 UTC · last answer 2026-11-08 16:33:54 UTC · span 20961.0 s, pauses included (not charged) · charged 798.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 7.4 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 1 answer cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 552B MoE · 16B active · source https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash · read 2026-11-10 · Ümit #4254: from the V4.1 Flash model card (open weights on Hugging Face): 552B backbone total, 16B active (decode; 8B in prefill); the card also lists 196B of conditional memory, not counted in the total
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
8 Nov 2026 10:44 UTC; last answer 8 Nov 2026 16:33 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
DeepSeek-V4.1-Flash 1017 through OpenRouter (deepseek/deepseek-v4.1-flash-1017), provider pinned to deepseek, fallbacks off; reasoning {"effort": "high"}; temperature 1.0; max_tokens 32768; one question per request; parameters 16B / 552B active / total (Tinker models table). English translations. DeepSeek-V4.1-Flash 1017 also ran on our GPUs for other benchmarks (Reading, Backgammon); those are separate setups under the same model name, not verified to be identical to this API setup (Umit #3101).
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "deepseek"
    ]
  },
  "reasoning": {
    "effort": "high"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · mimo-v2.6-flash_en

Name
MiMo-V2.6-Flash on · chat
Setup
MiMo V2.6 Flash · chat model
Model
xiaomi/mimo-v2.6-flash · provider openrouter
GPU
—
Cost
$0.003493 for the run · provider price × reported usage
Answer time
run mimo-v2.6-flash_en · first answer 2026-11-08 10:52:15 UTC · last answer 2026-11-08 10:53:03 UTC · span 48.0 s, pauses included (not charged) · charged 48.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 3.9 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 309B MoE · 15B active · from the run's notes (OpenRouter model card); also https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL · read 2026-11-09
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
8 Nov 2026 10:52 UTC; last answer 8 Nov 2026 10:53 UTC
Progress
25 answered of 2,163 asked; 0 request errors; smoke test at 32,768 tokens only; the full run uses 65,536 tokens (run mimo-v2.6-flash-64k_en)
Notes
MiMo V2.6 Flash through OpenRouter (xiaomi/mimo-v2.6-flash), provider pinned to xiaomi/fp8, fallbacks off; reasoning {"enabled": true}; temperature 1.0; max_tokens 32768; one question per request; parameters 15B / 309B active / total (OpenRouter model card). English translations.
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "xiaomi/fp8"
    ]
  },
  "reasoning": {
    "enabled": true
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · glm-5.3-flash_en

Name
GLM-5.3-Flash high · chat
Setup
GLM-5.3-Flash · chat model
Model
z-ai/glm-5.3-flash · provider openrouter
GPU
—
Cost
$0.3601 for the run · provider price × reported usage
Answer time
run glm-5.3-flash_en · first answer 2026-11-08 10:52:40 UTC · last answer 2026-11-08 18:48:42 UTC · span 28562.0 s, pauses included (not charged) · charged 1966.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 6.5 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 1 answer cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 320B MoE · 18B active · from the run's notes (Tinker models table); also https://huggingface.co/zai-org/GLM-5.3-Flash · read 2026-11-09
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
8 Nov 2026 10:52 UTC; last answer 8 Nov 2026 18:48 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
GLM-5.3-Flash through OpenRouter (z-ai/glm-5.3-flash), provider pinned to z-ai/fp8, fallbacks off; reasoning {"effort": "high"}; temperature 1.0; max_tokens 32768; one question per request; parameters 18B / 320B active / total (Tinker models table). English translations.
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "z-ai/fp8"
    ]
  },
  "reasoning": {
    "effort": "high"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · inkling-small_en

Name
Inkling-Small high · chat · Jul 2026
Setup
Inkling Small · chat model
Model
thinkingmachines/inkling-small · provider openrouter
GPU
—
Cost
$0.01908 for the run · provider price × reported usage
Answer time
run inkling-small_en · first answer 2026-11-08 11:16:48 UTC · last answer 2026-11-08 11:17:52 UTC · span 64.0 s, pauses included (not charged) · charged 64.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 2.8 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 276B MoE · 12B active · source https://thinkingmachines.ai/model-card/inkling-small/ · read 2026-11-10 · the card: 276B total, 12B active (Ümit #4245: the card's count, not our checkpoint count of 265.96B by turkce, bus #3592)
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
8 Nov 2026 11:16 UTC; last answer 8 Nov 2026 11:17 UTC
Progress
25 answered of 2,163 asked; 0 request errors; smoke test at 32,768 tokens only; the full run uses 65,536 tokens (run inkling-small-64k_en)
Notes
Inkling Small through OpenRouter (thinkingmachines/inkling-small), provider pinned to deepinfra/fp8, fallbacks off; reasoning {"effort": "high"}; temperature 1.0; max_tokens 32768; one question per request; parameters 12B / 276B active / total (Thinking Machines (Tinker models table)). English translations.
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "deepinfra/fp8"
    ]
  },
  "reasoning": {
    "effort": "high"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · inkling-small-64k_en

Name
Inkling-Small high · chat · Jul 2026
Setup
Inkling Small · chat model
Model
thinkingmachines/inkling-small · provider openrouter
GPU
—
Cost
$2.063 for the run · provider price × reported usage
Answer time
run inkling-small-64k_en · first answer 2026-11-08 14:56:46 UTC · last answer 2026-11-09 02:23:29 UTC · span 41203.0 s, pauses included (not charged) · charged 1144.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 7.7 in flight (sum of request latencies / active time)
Output limit
65,536 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 276B MoE · 12B active · source https://thinkingmachines.ai/model-card/inkling-small/ · read 2026-11-10 · the card: 276B total, 12B active (Ümit #4245: the card's count, not our checkpoint count of 265.96B by turkce, bus #3592)
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
8 Nov 2026 14:56 UTC; last answer 9 Nov 2026 02:23 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Inkling Small through OpenRouter (thinkingmachines/inkling-small), provider pinned to deepinfra/fp8, fallbacks off; reasoning {"effort": "high"}; temperature 1.0; max_tokens 65536; one question per request; parameters 12B / 276B active / total (Thinking Machines (Tinker models table)). English translations.
Parameters
{
  "max_tokens": 65536,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "deepinfra/fp8"
    ]
  },
  "reasoning": {
    "effort": "high"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · mimo-v2.6-flash-64k_en

Name
MiMo-V2.6-Flash on · chat
Setup
MiMo V2.6 Flash · chat model
Model
xiaomi/mimo-v2.6-flash · provider openrouter
GPU
—
Cost
$0.4582 for the run · provider price × reported usage
Answer time
run mimo-v2.6-flash-64k_en · first answer 2026-11-08 15:18:33 UTC · last answer 2026-11-09 03:12:43 UTC · span 42850.0 s, pauses included (not charged) · charged 4084.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 7.8 in flight (sum of request latencies / active time)
Output limit
65,536 tokens · 4 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 309B MoE · 15B active · from the run's notes (OpenRouter model card); also https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL · read 2026-11-09
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
8 Nov 2026 15:18 UTC; last answer 9 Nov 2026 03:12 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
MiMo V2.6 Flash through OpenRouter (xiaomi/mimo-v2.6-flash), provider pinned to xiaomi/fp8, fallbacks off; reasoning {"enabled": true}; temperature 1.0; max_tokens 65536; one question per request; parameters 15B / 309B active / total (OpenRouter model card). English translations.
Parameters
{
  "max_tokens": 65536,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "xiaomi/fp8"
    ]
  },
  "reasoning": {
    "enabled": true
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · minimax-m3.1-flash_en_dontknow

Name
MiniMax-M3.1-Flash high · chat
Setup
MiniMax-M3.1-Flash · chat model · with the I don't know option F
Model
minimax/minimax-m3.1-flash · provider openrouter
GPU
—
Cost
$0.2081 for the run · provider price × reported usage
Answer time
run minimax-m3.1-flash_en_dontknow · first answer 2026-11-08 16:27:22 UTC · last answer 2026-11-08 16:43:39 UTC · span 977.0 s, pauses included (not charged) · charged 977.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 7.6 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 428B MoE · 23B active · from the run's notes (NVIDIA NIM model card); also https://huggingface.co/MiniMaxAI/MiniMax-M3.1-Flash · read 2026-11-09 · the card: approximately 428B total, 23B activated per token
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
8 Nov 2026 16:27 UTC; last answer 8 Nov 2026 16:43 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
MiniMax-M3.1-Flash through OpenRouter (minimax/minimax-m3.1-flash), provider pinned to minimax, fallbacks off; reasoning {"effort": "high"}; temperature 1.0; max_tokens 32768; one question per request; parameters 23B / 428B active / total (NVIDIA NIM model card). English translations. One more reply came back broken (question kk1-ondalik-sayilar-test-2-9, ending "ANS: X" instead of "ANSWER: X"); its clearly readable letter was scored by hand (Umit #3215, zeki #5349): stored before as extracted NULL, correct 0 (no letter); letter read from the reply alone, blind to the key, then graded against the key; reply sha256 5418c14f6bde28ad50761eca65d9b158a22b099fe30cb53d02c1ecd43491a330.
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "minimax"
    ]
  },
  "reasoning": {
    "effort": "high"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · deepseek-v4.1-flash-1017_en_dontknow

Name
DeepSeek-V4.1-Flash high · chat · 1017
Setup
DeepSeek-V4.1-Flash 1017 · chat model · with the I don't know option F
Model
deepseek/deepseek-v4.1-flash-1017 · provider openrouter
GPU
—
Cost
$0.4992 for the run · provider price × reported usage
Answer time
run deepseek-v4.1-flash-1017_en_dontknow · first answer 2026-11-08 16:33:56 UTC · last answer 2026-11-08 16:46:26 UTC · span 750.0 s, pauses included (not charged) · charged 750.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 8.0 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 1 answer cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 552B MoE · 16B active · source https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash · read 2026-11-10 · Ümit #4254: from the V4.1 Flash model card (open weights on Hugging Face): 552B backbone total, 16B active (decode; 8B in prefill); the card also lists 196B of conditional memory, not counted in the total
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
8 Nov 2026 16:33 UTC; last answer 8 Nov 2026 16:46 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
DeepSeek-V4.1-Flash 1017 through OpenRouter (deepseek/deepseek-v4.1-flash-1017), provider pinned to deepseek, fallbacks off; reasoning {"effort": "high"}; temperature 1.0; max_tokens 32768; one question per request; parameters 16B / 552B active / total (Tinker models table). English translations. DeepSeek-V4.1-Flash 1017 also ran on our GPUs for other benchmarks (Reading, Backgammon); those are separate setups under the same model name, not verified to be identical to this API setup (Umit #3101).
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "deepseek"
    ]
  },
  "reasoning": {
    "effort": "high"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · glm-5.3-flash_en_dontknow

Name
GLM-5.3-Flash high · chat
Setup
GLM-5.3-Flash · chat model · with the I don't know option F
Model
z-ai/glm-5.3-flash · provider openrouter
GPU
—
Cost
$0.3313 for the run · provider price × reported usage
Answer time
run glm-5.3-flash_en_dontknow · first answer 2026-11-08 18:48:44 UTC · last answer 2026-11-08 21:13:14 UTC · span 8670.0 s, pauses included (not charged) · charged 1589.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 6.7 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 1 answer cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 320B MoE · 18B active · from the run's notes (Tinker models table); also https://huggingface.co/zai-org/GLM-5.3-Flash · read 2026-11-09
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
8 Nov 2026 18:48 UTC; last answer 8 Nov 2026 21:13 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
GLM-5.3-Flash through OpenRouter (z-ai/glm-5.3-flash), provider pinned to z-ai/fp8, fallbacks off; reasoning {"effort": "high"}; temperature 1.0; max_tokens 32768; one question per request; parameters 18B / 320B active / total (Tinker models table). English translations.
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "z-ai/fp8"
    ]
  },
  "reasoning": {
    "effort": "high"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · inkling-small-64k_en_dontknow

Name
Inkling-Small high · chat · Jul 2026
Setup
Inkling Small · chat model · with the I don't know option F
Model
thinkingmachines/inkling-small · provider openrouter
GPU
—
Cost
$1.861 for the run · provider price × reported usage
Answer time
run inkling-small-64k_en_dontknow · first answer 2026-11-09 02:23:30 UTC · last answer 2026-11-09 02:39:23 UTC · span 953.0 s, pauses included (not charged) · charged 953.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 7.9 in flight (sum of request latencies / active time)
Output limit
65,536 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 276B MoE · 12B active · source https://thinkingmachines.ai/model-card/inkling-small/ · read 2026-11-10 · the card: 276B total, 12B active (Ümit #4245: the card's count, not our checkpoint count of 265.96B by turkce, bus #3592)
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
9 Nov 2026 02:23 UTC; last answer 9 Nov 2026 02:39 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Inkling Small through OpenRouter (thinkingmachines/inkling-small), provider pinned to deepinfra/fp8, fallbacks off; reasoning {"effort": "high"}; temperature 1.0; max_tokens 65536; one question per request; parameters 12B / 276B active / total (Thinking Machines (Tinker models table)). English translations.
Parameters
{
  "max_tokens": 65536,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "deepinfra/fp8"
    ]
  },
  "reasoning": {
    "effort": "high"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · mimo-v2.6-flash-64k_en_dontknow

Name
MiMo-V2.6-Flash on · chat
Setup
MiMo V2.6 Flash · chat model · with the I don't know option F
Model
xiaomi/mimo-v2.6-flash · provider openrouter
GPU
—
Cost
$0.3925 for the run · provider price × reported usage
Answer time
run mimo-v2.6-flash-64k_en_dontknow · first answer 2026-11-09 03:12:48 UTC · last answer 2026-11-09 04:12:20 UTC · span 3572.0 s, pauses included (not charged) · charged 3572.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 7.2 in flight (sum of request latencies / active time)
Output limit
65,536 tokens · 2 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 309B MoE · 15B active · from the run's notes (OpenRouter model card); also https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL · read 2026-11-09
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
9 Nov 2026 03:12 UTC; last answer 9 Nov 2026 04:12 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
MiMo V2.6 Flash through OpenRouter (xiaomi/mimo-v2.6-flash), provider pinned to xiaomi/fp8, fallbacks off; reasoning {"enabled": true}; temperature 1.0; max_tokens 65536; one question per request; parameters 15B / 309B active / total (OpenRouter model card). English translations.
Parameters
{
  "max_tokens": 65536,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "xiaomi/fp8"
    ]
  },
  "reasoning": {
    "enabled": true
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · lfm2.5-8b-a1b_en

Name
LFM2.5-8B-A1B on · chat
Setup
LFM2.5-8B-A1B (chat, thinking, on our GPU) · chat model
Model
lfm2.5-8b-a1b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.043 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.419 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.043. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-09 12:16:40–14:18:41 UTC†
GPU time
run lfm2.5-8b-a1b_en · first answer 2026-11-09 12:15:47 UTC · last answer 2026-11-09 14:42:56 UTC · span 8829.0 s, pauses included (not charged) · charged 1508.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.0430 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)†
Requests in flight
≈ 29.6 in flight (sum of request latencies / active time)†
Output limit
32,768 tokens · 5 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 8.3B MoE · 1.5B active · source https://huggingface.co/LiquidAI/LFM2.5-8B-A1B · read 2026-11-10 · the card states 8.3B total / 1.5B active (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 8.47B / 1.69B active (MoE, 4 of 32 experts; checkpoint 8,467,856,832, active 1,685,667,264 incl. tied embeddings (1.42B without); card: 8.3B total, 1.5B active)
Input
text, English
Endpoint
http://127.0.0.1:18601/v1/chat/completions
Started
9 Nov 2026 12:15 UTC; last answer 9 Nov 2026 14:42 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
LFM2.5-8B-A1B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from LiquidAI/LFM2.5-8B-A1B@dd83f5cb02aa2938b66bba9a9998d38d0517c1f3, vLLM reasoning parser qwen3, one RTX PRO 6000 on our GPU host alone; thinking always on (reasoning model; no switch, no levels); sampling temperature 0.2, top_k 80, repetition_penalty 1.05, from generation_config.json; seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 1.5B / 8.3B active / total (LiquidAI model card). English translations (DeepSeek + review). Phases: up to 8 requests at a time from the runner until 12:31:02 UTC (86 Logic answers stored; the 8 in flight then were aborted, not stored, and asked again), then up to 32 (the server, vLLM --max-num-seqs 16, runs at most 16 at once and queues the rest, so latency per request includes queue time). Same image, revision, server arguments and request settings in both phases; vLLM does not guarantee identical outputs across batch sizes (no batch-invariance flag), so the two phases are not claimed answer-equivalent. Interrupted attempts: the runner was stopped while 8 requests were in flight (not a quiescent checkpoint); those 8 attempts produced no stored answer, their questions were sent again after the restart, and their token usage is unknown (not counted as zero; it is not in the answer counts). The GPU time of this run (first to last answer per stretch) covers the wall-clock time they used, including the ~4-minute server restart. No thinking-off setup (bus #6516/#6924/#8187/#8202): the current served chat template exposes no supported switch for a separate thinking-off run; no separate off variant has been run or scored. (Checked 15 Nov in the pinned template: its only option, preserve_thinking, keeps or drops earlier turns' thinking.)
Parameters
{
  "max_tokens": 32768,
  "repetition_penalty": 1.05,
  "seed": 0,
  "temperature": 0.2,
  "top_k": 80
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · celeris-1.1_en

Name
celeris-1.1-decision
Setup
Celeris-1.1 Decision · decision model
Model
celeris-1.1-decision · provider celeris
GPU
—
Cost
$0.001968 for the run · provider price × reported usage
Answer time
run celeris-1.1_en · first answer 2026-11-09 12:24:28 UTC · last answer 2026-11-09 12:38:50 UTC · span 862.0 s, pauses included (not charged) · charged 862.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://docs.celeris.ai/decisions · read 2026-11-10
Input
text, English
Endpoint
https://inference.celeris.ai/celeris-1-decision
Started
9 Nov 2026 12:24 UTC; last answer 9 Nov 2026 12:38 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Celeris-1.1 Decision (site run None); runner ffc307cb7; probabilities come rounded to two decimals; the published runner has no price for celeris-1.1-decision, so runs carry no cost_usd (tokens × $0.004/1M)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "celeris-1.1-decision",
    "--provider",
    "celeris",
    "--timeout",
    "30",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "celeris-1.1-decision",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Celeris's decision model price",
    "date": "2026-11-09",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.004,
      "cache_write_input_tokens": 0.004,
      "cached_input_tokens": 0.004,
      "input_tokens": 0.004,
      "output_tokens": 0.0
    },
    "source": "https://docs.celeris.ai/pricing",
    "text": "$0.004 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "celeris",
  "questions": 2163,
  "request_seconds": 846.2626975429448,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "07c606eabbfebf20365c8d38ad796cb9aca61b1eece6f79ecf53368f5c58a348",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 491901,
    "output_tokens": 0
  }
}
Prompt template
POST https://inference.celeris.ai/celeris-1-decision/v1/systemone
{
 "model": "celeris-1.1-decision",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · celeris-1.1_en_dontknow

Name
celeris-1.1-decision
Setup
Celeris-1.1 Decision · decision model · with the I don't know option F
Model
celeris-1.1-decision · provider celeris
GPU
—
Cost
$0.002262 for the run · provider price × reported usage
Answer time
run celeris-1.1_en_dontknow · first answer 2026-11-09 12:39:01 UTC · last answer 2026-11-09 12:52:53 UTC · span 832.0 s, pauses included (not charged) · charged 832.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://docs.celeris.ai/decisions · read 2026-11-10
Input
text, English
Endpoint
https://inference.celeris.ai/celeris-1-decision
Started
9 Nov 2026 12:39 UTC; last answer 9 Nov 2026 12:52 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Celeris-1.1 Decision (site run None); runner ffc307cb7; probabilities come rounded to two decimals; the published runner has no price for celeris-1.1-decision, so runs carry no cost_usd (tokens × $0.004/1M)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "celeris-1.1-decision",
    "--provider",
    "celeris",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "celeris-1.1-decision",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Celeris's decision model price",
    "date": "2026-11-09",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.004,
      "cache_write_input_tokens": 0.004,
      "cached_input_tokens": 0.004,
      "input_tokens": 0.004,
      "output_tokens": 0.0
    },
    "source": "https://docs.celeris.ai/pricing",
    "text": "$0.004 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "celeris",
  "questions": 2163,
  "request_seconds": 818.4304124059418,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "54f6fc41ff689e17d661af43213cc2a0b2d29e38f89a04da13298696dc62d28c",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 565443,
    "output_tokens": 0
  }
}
Prompt template
POST https://inference.celeris.ai/celeris-1-decision/v1/systemone
{
 "model": "celeris-1.1-decision",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · lfm2.5-8b-a1b_en_dontknow

Name
LFM2.5-8B-A1B on · chat
Setup
LFM2.5-8B-A1B (chat, thinking, on our GPU) · chat model · with the I don't know option F
Model
lfm2.5-8b-a1b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.7989 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.321 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.799. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run lfm2.5-8b-a1b_en_dontknow · first answer 2026-11-09 16:00:54 UTC · last answer 2026-11-09 16:20:09 UTC · span 1155.0 s, pauses included (not charged) · charged 1155.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.7989 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.4 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 1 answer cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 8.3B MoE · 1.5B active · source https://huggingface.co/LiquidAI/LFM2.5-8B-A1B · read 2026-11-10 · the card states 8.3B total / 1.5B active (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 8.47B / 1.69B active (MoE, 4 of 32 experts; checkpoint 8,467,856,832, active 1,685,667,264 incl. tied embeddings (1.42B without); card: 8.3B total, 1.5B active)
Input
text, English
Endpoint
http://127.0.0.1:18601/v1/chat/completions
Started
9 Nov 2026 16:00 UTC; last answer 9 Nov 2026 16:20 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
LFM2.5-8B-A1B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from LiquidAI/LFM2.5-8B-A1B@dd83f5cb02aa2938b66bba9a9998d38d0517c1f3, vLLM reasoning parser qwen3, one RTX PRO 6000 on our GPU host alone; thinking always on (reasoning model; no switch, no levels); sampling temperature 0.2, top_k 80, repetition_penalty 1.05, from generation_config.json; seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 1.69B / 8.47B active / total (MoE, 4 of 32 experts; checkpoint 8,467,856,832, active 1,685,667,264 incl. tied embeddings, 1.42B without; card: 8.3B total, 1.5B active). English translations (DeepSeek + review). No thinking-off setup (bus #6516/#6924/#8187/#8202): the current served chat template exposes no supported switch for a separate thinking-off run; no separate off variant has been run or scored. (Checked 15 Nov in the pinned template: its only option, preserve_thinking, keeps or drops earlier turns' thinking.)
Parameters
{
  "max_tokens": 32768,
  "repetition_penalty": 1.05,
  "seed": 0,
  "temperature": 0.2,
  "top_k": 80
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · gemma-4-e4b-it_en

Name
Gemma 4 E4B chat
Setup
Gemma 4 E4B (chat, on our GPU) · chat model
Model
gemma-4-e4b-it · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.5865 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.236 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.587. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-09 16:40:27–17:02:38 UTC
GPU time
run gemma-4-e4b-it_en · first answer 2026-11-09 16:40:06 UTC · last answer 2026-11-09 17:16:25 UTC · span 2179.0 s, pauses included (not charged) · charged 848.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.5865 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 30.9 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 8B dense · 4.5B effective · source https://huggingface.co/google/gemma-4-E4B-it · read 2026-11-10 · the card states 4.5B effective (8B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 8.00B (counted from the checkpoint (7,996,157,418: language model 7.52B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 8B with embeddings, 4.5B effective; not a mixture of experts)
Input
text, English
Endpoint
http://127.0.0.1:18602/v1/chat/completions
Started
9 Nov 2026 16:40 UTC; last answer 9 Nov 2026 17:16 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Gemma 4 E4B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from google/gemma-4-E4B-it@4ecc2ba63bcac23ba4a767fc87025b1450f84c9c, vLLM reasoning parser gemma4, one RTX PRO 6000 on our GPU host alone; thinking off (template default, as the Gemma 4 31B run); sampling temperature 0, as the Gemma 4 31B chat run; no seed (as the Gemma 4 31B run); max_tokens 8,192; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 8.00B / 8.00B active / total (dense; checkpoint 7,996,157,418 incl. vision 0.17B and audio 0.31B towers; card: 8B with per-layer embeddings, 4.5B effective). English translations (DeepSeek + review).
Parameters
{
  "max_tokens": 8192,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · gemma-4-e4b-it_en_dontknow

Name
Gemma 4 E4B chat
Setup
Gemma 4 E4B (chat, on our GPU) · chat model · with the I don't know option F
Model
gemma-4-e4b-it · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.5734 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.23 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.573. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gemma-4-e4b-it_en_dontknow · first answer 2026-11-09 17:38:14 UTC · last answer 2026-11-09 17:52:03 UTC · span 829.0 s, pauses included (not charged) · charged 829.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.5734 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.9 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 1 answer cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 8B dense · 4.5B effective · source https://huggingface.co/google/gemma-4-E4B-it · read 2026-11-10 · the card states 4.5B effective (8B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 8.00B (counted from the checkpoint (7,996,157,418: language model 7.52B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 8B with embeddings, 4.5B effective; not a mixture of experts)
Input
text, English
Endpoint
http://127.0.0.1:18602/v1/chat/completions
Started
9 Nov 2026 17:38 UTC; last answer 9 Nov 2026 17:52 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Gemma 4 E4B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from google/gemma-4-E4B-it@4ecc2ba63bcac23ba4a767fc87025b1450f84c9c, vLLM reasoning parser gemma4, one RTX PRO 6000 on our GPU host alone; thinking off (template default, as the Gemma 4 31B run); sampling temperature 0, as the Gemma 4 31B chat run; no seed (as the Gemma 4 31B run); max_tokens 8,192; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 8.00B / 8.00B active / total (dense; checkpoint 7,996,157,418 incl. vision 0.17B and audio 0.31B towers; card: 8B with per-layer embeddings, 4.5B effective). English translations (DeepSeek + review).
Parameters
{
  "max_tokens": 8192,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · lfm3-1.7b_en

Name
LFM3-1.7B on · chat
Setup
LFM3-1.7B (chat, thinking, on our GPU) · chat model
Model
lfm3-1.7b · provider llama.cpp → vLLM
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.3133 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.126 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.313. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-09 18:19:55–18:03:31 UTC†
GPU time
run lfm3-1.7b_en · first answer 2026-11-09 18:18:53 UTC · last answer 2026-11-12 18:10:02 UTC · span 258669.0 s, pauses included (not charged) · charged 453.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.3133 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)†
Requests in flight
≈ 29.1 in flight (sum of request latencies / active time)†
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 1.73B dense · from the run's notes (dense; BF16 GGUF checkpoint 1,728,801,600, text only; card: 1.7B); also https://huggingface.co/LiquidAI/LFM3-1.7B-GGUF · read 2026-11-09 · the released checkpoint's tensors without registered buffers: 1,728,801,536 (the served BF16 GGUF also stores a 64-element RoPE buffer, 1,728,801,600); text only; card: 1.7B
Input
text, English
Endpoint
http://127.0.0.1:18605/v1/chat/completions
Started
9 Nov 2026 18:18 UTC; last answer 12 Nov 2026 18:10 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete, but replaced by run lfm3-1.7b-vllm_en (see its notes for why)
Notes
Engine: llama.cpp b12020 for answers 1–25, then vLLM nightly be47189f (from 12 Nov 18:03 UTC); same checkpoint, template and sampling. LFM3-1.7B, served with llama.cpp server b12020 (commit 0ec4027, CUDA build), official BF16 GGUF LiquidAI/LFM3-1.7B-GGUF@cd0c74c575b5c416f3011b5b4c298077f26ca8c2 LFM3-1.7B-BF16.gguf (vLLM v0.30.0 has no Lfm3ForCausalLM), chat template from LiquidAI/LFM3-1.7B@0dc4268b561499c648c7ba7c9466bf3d8d01517a, llama.cpp --reasoning-format deepseek, one RTX PRO 6000 on our GPU host alone; thinking on (enable_thinking true, the template default; on/off only); sampling temperature 0.1, top_k 50, repeat_penalty 1.1, repeat_last_n 69632, top_p 1, min_p 0, from generation_config.json; repetition_penalty as llama.cpp repeat_penalty over the whole context, llama.cpp's own top_p/min_p defaults switched off; seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 1.73B / 1.73B active / total (dense; BF16 GGUF checkpoint 1,728,801,600, text only; card: 1.7B). English translations (DeepSeek + review). Was paused (Umit #3883, 2026-11-10T00:38Z) until 12 Nov: skipped for now, stopped at 25 answers; kept, not ranked. Cause: llama.cpp main thread CPU-bound (repeat_penalty over the whole context for every token, 32 slots: ~118 tokens/s in total). Resumed 2026-11-12T11:31Z from answer 207 (Logic) / 26 (Math): 4 llama-server instances × 8 slots (same build, GGUF, flags, sampling, per-slot context 69,632, 32 requests in flight as before) behind a forwarder on the same port, for throughput only (zeki #5247/#5265); its latency and throughput figures mix the two topologies (1 × 32 slots before, 4 × 8 after). PAUSED 2026-11-12T17:17:39Z by logic under Umit's $30 cumulative cap for LFM3-1.7B (via zeki #5540): GPU-time cost by the site's rule (first to last answer less gaps >10 min, × $2.49/h) $27.62 at the pause; new dispatch stopped; the 32 requests then in flight were dropped (no reply stored, not scored); stored answers kept. Resume only after a verified engine check and a human OK. Cap removed by Can #5612 (2026-11-12): the $30 cap above no longer applies; the run continues on the vLLM nightly (ENGINE BOUNDARY below), target: finished within a day. ENGINE BOUNDARY (2026-11-12T18:03:30+00:00): Engine switched from llama.cpp b12020 (commit 0ec4027, official BF16 GGUF LiquidAI/LFM3-1.7B-GGUF@cd0c74c575b5c416f3011b5b4c298077f26ca8c2, 4 x 8 slots) to vLLM 0.34.0rc1.dev302+gbe47189f3 (nightly, commit be47189f37f6, CUDA 13.5 image; docker vllm/vllm-openai@sha256:7f0618da95bac2f1cc97fcd1791380d7c1a06d24a4998da45aeb7fbe6c49179e), BF16 serving LiquidAI/LFM3-1.7B@0dc4268b561499c648c7ba7c9466bf3d8d01517a (BF16 safetensors, tensors byte-identical to the GGUF), reasoning parser qwen3, 32 sequences at once, on GPU 0 of our GPU host alone. Unchanged: checkpoint, precision, chat template (sha256 f45005c5…, thinking on; prompt token ids identical), sampling (temperature 0.1, top_k 50, top_p 1, min_p 0, repetition penalty 1.1 over prompt and output in both engines: vLLM repetition_penalty in place of llama.cpp repeat_penalty with repeat_last_n over the whole context), seed 0, max_tokens 32,768, prompt and scoring. Differs: the random stream (vLLM Gumbel-max vs llama.cpp mt19937, so answers are not reproducible across the boundary), GPU kernels, and the reasoning splitter (qwen3 vs deepseek; the letter is read from the reply only). Why: llama.cpp was too slow (CPU-bound repeat penalty) for the one-day target; permission Umit via zeki #5540, cap removed by Can #5612; qualification work/lfm3_vllm_qual.md. Latency and throughput figures of this run mix the two engines. The 25 answers stored before it (last at 2026-11-09T18:19:55+00:00) came from endpoint http://127.0.0.1:18605/v1/chat/completions with parameters {"chat_template_kwargs": {"enable_thinking": true}, "max_tokens": 32768, "min_p": 0.0, "repeat_last_n": 69632, "repeat_penalty": 1.1, "seed": 0, "temperature": 0.1, "top_k": 50, "top_p": 1.0}; every answer stored after it (answer 26 onward, including re-asks of the 0 failed requests) comes from endpoint http://127.0.0.1:18605/v1/chat/completions with parameters {"chat_template_kwargs": {"enable_thinking": true}, "max_tokens": 32768, "min_p": 0.0, "repetition_penalty": 1.1, "seed": 0, "temperature": 0.1, "top_k": 50, "top_p": 1.0}. SUPERSEDED by lfm3-1.7b-vllm_en (engine mix llama.cpp + vLLM; cost/latency not comparable), Umit #6250.
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "repetition_penalty": 1.1,
  "seed": 0,
  "temperature": 0.1,
  "top_k": 50,
  "top_p": 1.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · gemma-4-e2b-it_en

Name
Gemma 4 E2B chat
Setup
Gemma 4 E2B (chat, on our GPU) · chat model
Model
gemma-4-e2b-it · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.4503 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.181 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.450. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-10 00:41:51–00:55:52 UTC
GPU time
run gemma-4-e2b-it_en · first answer 2026-11-10 00:41:40 UTC · last answer 2026-11-10 01:06:32 UTC · span 1492.0 s, pauses included (not charged) · charged 651.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.4503 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.3 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 16 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 5.1B dense · 2.3B effective · source https://huggingface.co/google/gemma-4-E2B-it · read 2026-11-10 · the card states 2.3B effective (5.1B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 5.12B (counted from the checkpoint (5,123,178,979: language model 4.65B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 5.1B with embeddings, 2.3B effective; not a mixture of experts)
Input
text, English
Endpoint
http://127.0.0.1:18606/v1/chat/completions
Started
10 Nov 2026 00:41 UTC; last answer 10 Nov 2026 01:06 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Gemma 4 E2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from google/gemma-4-E2B-it@4adf15748de9e1a0097b6274cee9c15836c41c37, vLLM reasoning parser gemma4, one RTX PRO 6000 on our GPU host alone; thinking off (template default, as the Gemma 4 31B run); sampling temperature 0, as the Gemma 4 31B chat run; no seed (as the Gemma 4 31B run); max_tokens 8,192; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 5.12B / 5.12B active / total (dense; checkpoint 5,123,178,979 incl. vision 0.17B and audio 0.31B towers; card: 5.1B with per-layer embeddings, 2.3B effective). English translations (DeepSeek + review).
Parameters
{
  "max_tokens": 8192,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · gemma-4-e2b-it_en_dontknow

Name
Gemma 4 E2B chat
Setup
Gemma 4 E2B (chat, on our GPU) · chat model · with the I don't know option F
Model
gemma-4-e2b-it · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.4565 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.183 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.457. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gemma-4-e2b-it_en_dontknow · first answer 2026-11-10 01:22:06 UTC · last answer 2026-11-10 01:33:06 UTC · span 660.0 s, pauses included (not charged) · charged 660.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.4565 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.5 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 16 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 5.1B dense · 2.3B effective · source https://huggingface.co/google/gemma-4-E2B-it · read 2026-11-10 · the card states 2.3B effective (5.1B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 5.12B (counted from the checkpoint (5,123,178,979: language model 4.65B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 5.1B with embeddings, 2.3B effective; not a mixture of experts)
Input
text, English
Endpoint
http://127.0.0.1:18606/v1/chat/completions
Started
10 Nov 2026 01:22 UTC; last answer 10 Nov 2026 01:33 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Gemma 4 E2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from google/gemma-4-E2B-it@4adf15748de9e1a0097b6274cee9c15836c41c37, vLLM reasoning parser gemma4, one RTX PRO 6000 on our GPU host alone; thinking off (template default, as the Gemma 4 31B run); sampling temperature 0, as the Gemma 4 31B chat run; no seed (as the Gemma 4 31B run); max_tokens 8,192; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 5.12B / 5.12B active / total (dense; checkpoint 5,123,178,979 incl. vision 0.17B and audio 0.31B towers; card: 5.1B with per-layer embeddings, 2.3B effective). English translations (DeepSeek + review).
Parameters
{
  "max_tokens": 8192,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · gemma-4-12b-it_en

Name
Gemma 4 12B Unified chat
Setup
Gemma 4 12B (chat, on our GPU) · chat model
Model
gemma-4-12b-it · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.011 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.406 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.011. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-10 01:43:36–02:16:18 UTC
GPU time
run gemma-4-12b-it_en · first answer 2026-11-10 01:43:14 UTC · last answer 2026-11-10 02:40:17 UTC · span 3423.0 s, pauses included (not charged) · charged 1461.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.0105 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.6 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 11 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 11.95B dense · source https://huggingface.co/google/gemma-4-12B-it · read 2026-11-10 · the card states Total Parameters 11.95B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 11.96B (counted from the checkpoint (11,959,730,224: language model 11.91B, vision embedder 0.05B, audio embedder 0.002B); the card: 11.95B)
Input
text, English
Endpoint
http://127.0.0.1:18607/v1/chat/completions
Started
10 Nov 2026 01:43 UTC; last answer 10 Nov 2026 02:40 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Gemma 4 12B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from google/gemma-4-12B-it@43c9d014b231474fa7212dbb48a762566c2b255d, vLLM reasoning parser gemma4, one RTX PRO 6000 on our GPU host alone; thinking off (template default, as the Gemma 4 31B run); sampling temperature 0, as the Gemma 4 31B chat run; no seed (as the Gemma 4 31B run); max_tokens 8,192; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 11.96B / 11.96B active / total (dense; checkpoint 11,959,730,224 incl. vision embedder 0.05B and audio embedder 0.002B; card: 11.95B). English translations (DeepSeek + review).
Parameters
{
  "max_tokens": 8192,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · gemma-4-12b-it_en_dontknow

Name
Gemma 4 12B Unified chat
Setup
Gemma 4 12B (chat, on our GPU) · chat model · with the I don't know option F
Model
gemma-4-12b-it · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.9649 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.388 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.965. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gemma-4-12b-it_en_dontknow · first answer 2026-11-10 03:14:56 UTC · last answer 2026-11-10 03:38:11 UTC · span 1395.0 s, pauses included (not charged) · charged 1395.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.9649 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 32.0 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 6 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 11.95B dense · source https://huggingface.co/google/gemma-4-12B-it · read 2026-11-10 · the card states Total Parameters 11.95B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 11.96B (counted from the checkpoint (11,959,730,224: language model 11.91B, vision embedder 0.05B, audio embedder 0.002B); the card: 11.95B)
Input
text, English
Endpoint
http://127.0.0.1:18607/v1/chat/completions
Started
10 Nov 2026 03:14 UTC; last answer 10 Nov 2026 03:38 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Gemma 4 12B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from google/gemma-4-12B-it@43c9d014b231474fa7212dbb48a762566c2b255d, vLLM reasoning parser gemma4, one RTX PRO 6000 on our GPU host alone; thinking off (template default, as the Gemma 4 31B run); sampling temperature 0, as the Gemma 4 31B chat run; no seed (as the Gemma 4 31B run); max_tokens 8,192; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 11.96B / 11.96B active / total (dense; checkpoint 11,959,730,224 incl. vision embedder 0.05B and audio embedder 0.002B; card: 11.95B). English translations (DeepSeek + review).
Parameters
{
  "max_tokens": 8192,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · glide-27b_en

Name
GLiDE 27B
Setup
GLiDE 27B (open) · decision model
Model
glide-27b · provider glide
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.05657 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.266 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.057 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run glide-27b_en · first answer 2026-11-10 09:03:34 UTC · last answer 2026-11-10 09:19:31 UTC · span 957.0 s, pauses included (not charged) · charged 957.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0566 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/fastino/GLiDE-27B · read 2026-11-10 · the card states a 27B model (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 26.09B (counted from the checkpoint (26,086,635,760: language model 25,624,600,064, vision encoder 460,730,096, readout 1,305,600; the same count as GLiDE v1.2, changed weights); recurrent depth: layers 24–47 run in 2 passes, their parameters counted once; the card says a 27B model)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
10 Nov 2026 09:03 UTC; last answer 10 Nov 2026 09:19 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup GLiDE 27B (open) (site run None); runner ffc307cb7; fastino/GLiDE-27B@4d32743ae96b74f0116f050d37742bbd43c04e7d (Fastino Community License); recurrent depth (decision_config midloop): layers 24–47 run in 2 passes (glide/recurrent.py), parameters counted once; on a shared Spark host (DGX Spark, GB10), served by the glide package the repository ships, one request at a time (GLIDE_MAX_BATCH=1): systemd-run --user --unit=logic-glide-27b --working-directory=~/glide-family/models/GLiDE-27B@4d32743ae96b74f0116f050d37742bbd43c04e7d -E GLIDE_MAX_BATCH=1 -E PYTHONUNBUFFERED=1 -E HF_HUB_OFFLINE=1 ~/envs/glide-family/bin/python -m glide.server --model ~/glide-family/models/GLiDE-27B@4d32743ae96b74f0116f050d37742bbd43c04e7d --host <a shared Spark host> --port 19310; env ~/envs/glide-family (CPython 3.12.13, torch 2.14.0+cu130, triton 3.8.0, transformers 5.17.0, flash-linear-attention 0.5.2); no price: no maker price for this checkpoint and no GPU-time rate for a DGX Spark in the runner
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "glide-27b",
    "--provider",
    "glide",
    "--timeout",
    "60",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "glide-27b",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "glide",
  "questions": 2163,
  "request_seconds": 940.7599747205677,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "36abcc2dca6673a6b4ccf3194eed6b72b0940098c12e953f48682381c1018fd3",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 408016,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19310/v1/systemone
{
 "model": "glide-27b",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · glide-27b_en_dontknow

Name
GLiDE 27B
Setup
GLiDE 27B (open) · decision model · with the I don't know option F
Model
glide-27b · provider glide
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.05918 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.278 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.059 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run glide-27b_en_dontknow · first answer 2026-11-10 09:31:04 UTC · last answer 2026-11-10 09:47:45 UTC · span 1001.0 s, pauses included (not charged) · charged 1001.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0592 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/fastino/GLiDE-27B · read 2026-11-10 · the card states a 27B model (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 26.09B (counted from the checkpoint (26,086,635,760: language model 25,624,600,064, vision encoder 460,730,096, readout 1,305,600; the same count as GLiDE v1.2, changed weights); recurrent depth: layers 24–47 run in 2 passes, their parameters counted once; the card says a 27B model)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
10 Nov 2026 09:31 UTC; last answer 10 Nov 2026 09:47 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup GLiDE 27B (open) (site run None); runner ffc307cb7; fastino/GLiDE-27B@4d32743ae96b74f0116f050d37742bbd43c04e7d (Fastino Community License); recurrent depth (decision_config midloop): layers 24–47 run in 2 passes (glide/recurrent.py), parameters counted once; on a shared Spark host (DGX Spark, GB10), served by the glide package the repository ships, one request at a time (GLIDE_MAX_BATCH=1): systemd-run --user --unit=logic-glide-27b --working-directory=~/glide-family/models/GLiDE-27B@4d32743ae96b74f0116f050d37742bbd43c04e7d -E GLIDE_MAX_BATCH=1 -E PYTHONUNBUFFERED=1 -E HF_HUB_OFFLINE=1 ~/envs/glide-family/bin/python -m glide.server --model ~/glide-family/models/GLiDE-27B@4d32743ae96b74f0116f050d37742bbd43c04e7d --host <a shared Spark host> --port 19310; env ~/envs/glide-family (CPython 3.12.13, torch 2.14.0+cu130, triton 3.8.0, transformers 5.17.0, flash-linear-attention 0.5.2); no price: no maker price for this checkpoint and no GPU-time rate for a DGX Spark in the runner
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "glide-27b",
    "--provider",
    "glide",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "d34eca45758fb0f34790435742d5c28028131623",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "glide-27b",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "glide",
  "questions": 2163,
  "request_seconds": 985.379409737885,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "5726f2aa27cfe6a4d8da21d15161e4a145ccd0ee14d04c7c433e97565d8745b1",
  "runner": "benchmark_ollama.py",
  "runner_commit": "ffc307cb723143e7a64ecd4cd1865f8eaa2e6e12",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 472906,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19310/v1/systemone
{
 "model": "glide-27b",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · glide-9b_en

Name
GLiDE 9B
Setup
GLiDE 9B (open) · decision model
Model
glide-9b · provider glide
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.0227 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.107 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.023 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run glide-9b_en · first answer 2026-11-10 11:49:56 UTC · last answer 2026-11-10 11:56:20 UTC · span 384.0 s, pauses included (not charged) · charged 384.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0227 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 7.94B dense · source https://huggingface.co/fastino/GLiDE-9B/tree/ab54df5acba0eebf7df45560ffdaacd64e32df3b · read 2026-11-10 · counted from the checkpoint (7,937,729,024: language model 7,936,684,544, readout 1,044,480; text only); recurrent depth: layers 12–23 run in 2 passes, their parameters counted once; the card: 7.94B (1.04M decision readout)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
10 Nov 2026 11:49 UTC; last answer 10 Nov 2026 11:56 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup GLiDE 9B (open) (site run None); runner 9ba1b409d; fastino/GLiDE-9B@ab54df5acba0eebf7df45560ffdaacd64e32df3b (Fastino Community License); recurrent depth (decision_config midloop): layers 12–23 run in 2 passes (glide/recurrent.py), parameters counted once; on a shared Spark host (DGX Spark, GB10), served by the glide package the repository ships, one request at a time (GLIDE_MAX_BATCH=1): systemd-run --user --unit=logic-glide-9b --working-directory=~/glide-family/models/GLiDE-9B@ab54df5acba0eebf7df45560ffdaacd64e32df3b -E GLIDE_MAX_BATCH=1 -E PYTHONUNBUFFERED=1 -E HF_HUB_OFFLINE=1 ~/envs/glide-family/bin/python -m glide.server --model ~/glide-family/models/GLiDE-9B@ab54df5acba0eebf7df45560ffdaacd64e32df3b --host <a shared Spark host> --port 19320; env ~/envs/glide-family (CPython 3.12.13, torch 2.14.0+cu130, triton 3.8.0, transformers 5.17.0, flash-linear-attention 0.5.2); no price: no maker price for this checkpoint and no GPU-time rate for a DGX Spark in the runner
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "glide-9b",
    "--provider",
    "glide",
    "--timeout",
    "60",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "3b14502cefed852b2c6ac3a9139c7574549374b6",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "glide-9b",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "glide",
  "questions": 2163,
  "request_seconds": 374.3174605209206,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "fc08858c43861204b5d1356f74cf69d9612f9c715852b22d8a2fefb43a64a459",
  "runner": "benchmark_ollama.py",
  "runner_commit": "9ba1b409d68af1a8b3c846f54993e438b9db212a",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 408016,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19320/v1/systemone
{
 "model": "glide-9b",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · glide-9b_en_dontknow

Name
GLiDE 9B
Setup
GLiDE 9B (open) · decision model · with the I don't know option F
Model
glide-9b · provider glide
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.02388 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.112 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.024 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run glide-9b_en_dontknow · first answer 2026-11-10 12:00:53 UTC · last answer 2026-11-10 12:07:37 UTC · span 404.0 s, pauses included (not charged) · charged 404.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0239 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 7.94B dense · source https://huggingface.co/fastino/GLiDE-9B/tree/ab54df5acba0eebf7df45560ffdaacd64e32df3b · read 2026-11-10 · counted from the checkpoint (7,937,729,024: language model 7,936,684,544, readout 1,044,480; text only); recurrent depth: layers 12–23 run in 2 passes, their parameters counted once; the card: 7.94B (1.04M decision readout)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
10 Nov 2026 12:00 UTC; last answer 10 Nov 2026 12:07 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup GLiDE 9B (open) (site run None); runner 9ba1b409d; fastino/GLiDE-9B@ab54df5acba0eebf7df45560ffdaacd64e32df3b (Fastino Community License); recurrent depth (decision_config midloop): layers 12–23 run in 2 passes (glide/recurrent.py), parameters counted once; on a shared Spark host (DGX Spark, GB10), served by the glide package the repository ships, one request at a time (GLIDE_MAX_BATCH=1): systemd-run --user --unit=logic-glide-9b --working-directory=~/glide-family/models/GLiDE-9B@ab54df5acba0eebf7df45560ffdaacd64e32df3b -E GLIDE_MAX_BATCH=1 -E PYTHONUNBUFFERED=1 -E HF_HUB_OFFLINE=1 ~/envs/glide-family/bin/python -m glide.server --model ~/glide-family/models/GLiDE-9B@ab54df5acba0eebf7df45560ffdaacd64e32df3b --host <a shared Spark host> --port 19320; env ~/envs/glide-family (CPython 3.12.13, torch 2.14.0+cu130, triton 3.8.0, transformers 5.17.0, flash-linear-attention 0.5.2); no price: no maker price for this checkpoint and no GPU-time rate for a DGX Spark in the runner
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "glide-9b",
    "--provider",
    "glide",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "3b14502cefed852b2c6ac3a9139c7574549374b6",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "glide-9b",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "glide",
  "questions": 2163,
  "request_seconds": 390.85128419226385,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "80eff531ba5813f3e670eaf03bce87fa3481a7dc9e2a799f9f0b5b437048c4ee",
  "runner": "benchmark_ollama.py",
  "runner_commit": "9ba1b409d68af1a8b3c846f54993e438b9db212a",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 472906,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19320/v1/systemone
{
 "model": "glide-9b",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · glide-2b_en

Name
GLiDE 2B
Setup
GLiDE 2B (open) · decision model
Model
glide-2b · provider glide
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.003843 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0181 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.004 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run glide-2b_en · first answer 2026-11-10 12:41:45 UTC · last answer 2026-11-10 12:42:50 UTC · span 65.0 s, pauses included (not charged) · charged 65.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0038 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 1.88B dense · source https://huggingface.co/fastino/GLiDE-2B/tree/00f122c120cdeed82284106cb77d9bab5a50ad16 · read 2026-11-10 · counted from the checkpoint (1,882,347,328: language model 1,881,825,088, readout 522,240; text only); the card: 1.88B
Input
text, English
Endpoint
internal address on a shared Spark host
Started
10 Nov 2026 12:41 UTC; last answer 10 Nov 2026 12:42 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup GLiDE 2B (open) (site run None); runner 9ba1b409d; fastino/GLiDE-2B@00f122c120cdeed82284106cb77d9bab5a50ad16 (Fastino Community License) on a shared Spark host (DGX Spark, GB10), served by the glide package the repository ships, one request at a time (GLIDE_MAX_BATCH=1): systemd-run --user --unit=logic-glide-2b --working-directory=~/glide-family/models/GLiDE-2B@00f122c120cdeed82284106cb77d9bab5a50ad16 -E GLIDE_MAX_BATCH=1 -E PYTHONUNBUFFERED=1 -E HF_HUB_OFFLINE=1 ~/envs/glide-family/bin/python -m glide.server --model ~/glide-family/models/GLiDE-2B@00f122c120cdeed82284106cb77d9bab5a50ad16 --host <a shared Spark host> --port 19330; env ~/envs/glide-family (CPython 3.12.13, torch 2.14.0+cu130, triton 3.8.0, transformers 5.17.0, flash-linear-attention 0.5.2); no price: no maker price for this checkpoint and no GPU-time rate for a DGX Spark in the runner
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "glide-2b",
    "--provider",
    "glide",
    "--timeout",
    "60",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "3b14502cefed852b2c6ac3a9139c7574549374b6",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "glide-2b",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "glide",
  "questions": 2163,
  "request_seconds": 60.26507895544637,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "f01fc9f779db80c997eca885c5cc960f8269422fb63dd0507d6d9f5f63e74e09",
  "runner": "benchmark_ollama.py",
  "runner_commit": "9ba1b409d68af1a8b3c846f54993e438b9db212a",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 408016,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19330/v1/systemone
{
 "model": "glide-2b",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · glide-2b_en_dontknow

Name
GLiDE 2B
Setup
GLiDE 2B (open) · decision model · with the I don't know option F
Model
glide-2b · provider glide
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.00402 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0189 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.004 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run glide-2b_en_dontknow · first answer 2026-11-10 12:43:39 UTC · last answer 2026-11-10 12:44:47 UTC · span 68.0 s, pauses included (not charged) · charged 68.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0040 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 1.88B dense · source https://huggingface.co/fastino/GLiDE-2B/tree/00f122c120cdeed82284106cb77d9bab5a50ad16 · read 2026-11-10 · counted from the checkpoint (1,882,347,328: language model 1,881,825,088, readout 522,240; text only); the card: 1.88B
Input
text, English
Endpoint
internal address on a shared Spark host
Started
10 Nov 2026 12:43 UTC; last answer 10 Nov 2026 12:44 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup GLiDE 2B (open) (site run None); runner 9ba1b409d; fastino/GLiDE-2B@00f122c120cdeed82284106cb77d9bab5a50ad16 (Fastino Community License) on a shared Spark host (DGX Spark, GB10), served by the glide package the repository ships, one request at a time (GLIDE_MAX_BATCH=1): systemd-run --user --unit=logic-glide-2b --working-directory=~/glide-family/models/GLiDE-2B@00f122c120cdeed82284106cb77d9bab5a50ad16 -E GLIDE_MAX_BATCH=1 -E PYTHONUNBUFFERED=1 -E HF_HUB_OFFLINE=1 ~/envs/glide-family/bin/python -m glide.server --model ~/glide-family/models/GLiDE-2B@00f122c120cdeed82284106cb77d9bab5a50ad16 --host <a shared Spark host> --port 19330; env ~/envs/glide-family (CPython 3.12.13, torch 2.14.0+cu130, triton 3.8.0, transformers 5.17.0, flash-linear-attention 0.5.2); no price: no maker price for this checkpoint and no GPU-time rate for a DGX Spark in the runner
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "glide-2b",
    "--provider",
    "glide",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "3b14502cefed852b2c6ac3a9139c7574549374b6",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "glide-2b",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "glide",
  "questions": 2163,
  "request_seconds": 62.90074737084797,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "a899dd657b349a7b724f331e7b60a58b5102651b669e22ee7fdc164cff55cdca",
  "runner": "benchmark_ollama.py",
  "runner_commit": "9ba1b409d68af1a8b3c846f54993e438b9db212a",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 472906,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19330/v1/systemone
{
 "model": "glide-2b",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · glide-0.8b_en

Name
GLiDE 0.8B
Setup
GLiDE 0.8B (open) · decision model
Model
glide-0.8b · provider glide
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.002778 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0131 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.003 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run glide-0.8b_en · first answer 2026-11-10 12:52:44 UTC · last answer 2026-11-10 12:53:31 UTC · span 47.0 s, pauses included (not charged) · charged 47.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0028 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 0.75B dense · source https://huggingface.co/fastino/GLiDE-0.8B/tree/9eacdeec79fa48ec52a8d6f54b331376b4bbfaf0 · read 2026-11-10 · counted from the checkpoint (752,654,144: language model 752,393,024, readout 261,120; text only); the card: 0.75B
Input
text, English
Endpoint
internal address on a shared Spark host
Started
10 Nov 2026 12:52 UTC; last answer 10 Nov 2026 12:53 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup GLiDE 0.8B (open) (site run None); runner 9ba1b409d; fastino/GLiDE-0.8B@9eacdeec79fa48ec52a8d6f54b331376b4bbfaf0 (Fastino Community License) on a shared Spark host (DGX Spark, GB10), served by the glide package the repository ships, one request at a time (GLIDE_MAX_BATCH=1): systemd-run --user --unit=logic-glide-0.8b --working-directory=~/glide-family/models/GLiDE-0.8B@9eacdeec79fa48ec52a8d6f54b331376b4bbfaf0 -E GLIDE_MAX_BATCH=1 -E PYTHONUNBUFFERED=1 -E HF_HUB_OFFLINE=1 ~/envs/glide-family/bin/python -m glide.server --model ~/glide-family/models/GLiDE-0.8B@9eacdeec79fa48ec52a8d6f54b331376b4bbfaf0 --host <a shared Spark host> --port 19340; env ~/envs/glide-family (CPython 3.12.13, torch 2.14.0+cu130, triton 3.8.0, transformers 5.17.0, flash-linear-attention 0.5.2); no price: no maker price for this checkpoint and no GPU-time rate for a DGX Spark in the runner
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "glide-0.8b",
    "--provider",
    "glide",
    "--timeout",
    "60",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "3b14502cefed852b2c6ac3a9139c7574549374b6",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "glide-0.8b",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "glide",
  "questions": 2163,
  "request_seconds": 42.90148255604436,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "31a8d4aad633ecf6c513e6bc1881254a10e4127e0334945652013cd0efda024b",
  "runner": "benchmark_ollama.py",
  "runner_commit": "9ba1b409d68af1a8b3c846f54993e438b9db212a",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 408016,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19340/v1/systemone
{
 "model": "glide-0.8b",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · glide-0.8b_en_dontknow

Name
GLiDE 0.8B
Setup
GLiDE 0.8B (open) · decision model · with the I don't know option F
Model
glide-0.8b · provider glide
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.00266 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0125 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.003 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run glide-0.8b_en_dontknow · first answer 2026-11-10 12:54:06 UTC · last answer 2026-11-10 12:54:51 UTC · span 45.0 s, pauses included (not charged) · charged 45.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0027 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 0.75B dense · source https://huggingface.co/fastino/GLiDE-0.8B/tree/9eacdeec79fa48ec52a8d6f54b331376b4bbfaf0 · read 2026-11-10 · counted from the checkpoint (752,654,144: language model 752,393,024, readout 261,120; text only); the card: 0.75B
Input
text, English
Endpoint
internal address on a shared Spark host
Started
10 Nov 2026 12:54 UTC; last answer 10 Nov 2026 12:54 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup GLiDE 0.8B (open) (site run None); runner 9ba1b409d; fastino/GLiDE-0.8B@9eacdeec79fa48ec52a8d6f54b331376b4bbfaf0 (Fastino Community License) on a shared Spark host (DGX Spark, GB10), served by the glide package the repository ships, one request at a time (GLIDE_MAX_BATCH=1): systemd-run --user --unit=logic-glide-0.8b --working-directory=~/glide-family/models/GLiDE-0.8B@9eacdeec79fa48ec52a8d6f54b331376b4bbfaf0 -E GLIDE_MAX_BATCH=1 -E PYTHONUNBUFFERED=1 -E HF_HUB_OFFLINE=1 ~/envs/glide-family/bin/python -m glide.server --model ~/glide-family/models/GLiDE-0.8B@9eacdeec79fa48ec52a8d6f54b331376b4bbfaf0 --host <a shared Spark host> --port 19340; env ~/envs/glide-family (CPython 3.12.13, torch 2.14.0+cu130, triton 3.8.0, transformers 5.17.0, flash-linear-attention 0.5.2); no price: no maker price for this checkpoint and no GPU-time rate for a DGX Spark in the runner
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "glide-0.8b",
    "--provider",
    "glide",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "3b14502cefed852b2c6ac3a9139c7574549374b6",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "glide-0.8b",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "glide",
  "questions": 2163,
  "request_seconds": 40.223021584475646,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "383b904c14319a4832981d32f25755668807a6ac7f05fd37c75849ff3e537288",
  "runner": "benchmark_ollama.py",
  "runner_commit": "9ba1b409d68af1a8b3c846f54993e438b9db212a",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 472906,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19340/v1/systemone
{
 "model": "glide-0.8b",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · glide-0.4b_en

Name
GLiDE 0.4B
Setup
GLiDE 0.4B (open) · decision model
Model
glide-0.4b · provider glide
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.001655 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.00778 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.002 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run glide-0.4b_en · first answer 2026-11-10 13:01:00 UTC · last answer 2026-11-10 13:01:28 UTC · span 28.0 s, pauses included (not charged) · charged 28.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0017 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.8 in flight (sum of request latencies / active time)
Model size
Parameters: 0.40B dense · source https://huggingface.co/fastino/GLiDE-0.4B/tree/6bda7ab91aeb36dcad6d27828a8871e548c7fc71 · read 2026-11-10 · counted from the checkpoint (395,042,816: encoder 394,781,696, readout 261,120; an encoder, ModernBERT architecture); the card: 0.40B
Input
text, English
Endpoint
internal address on a shared Spark host
Started
10 Nov 2026 13:01 UTC; last answer 10 Nov 2026 13:01 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup GLiDE 0.4B (open) (site run None); runner 9ba1b409d; fastino/GLiDE-0.4B@6bda7ab91aeb36dcad6d27828a8871e548c7fc71 (Fastino Community License) on a shared Spark host (DGX Spark, GB10), served by the glide package the repository ships, one request at a time (GLIDE_MAX_BATCH=1): systemd-run --user --unit=logic-glide-0.4b --working-directory=~/glide-family/models/GLiDE-0.4B@6bda7ab91aeb36dcad6d27828a8871e548c7fc71 -E GLIDE_MAX_BATCH=1 -E PYTHONUNBUFFERED=1 -E HF_HUB_OFFLINE=1 ~/envs/glide-family/bin/python -m glide.server --model ~/glide-family/models/GLiDE-0.4B@6bda7ab91aeb36dcad6d27828a8871e548c7fc71 --host <a shared Spark host> --port 19350; env ~/envs/glide-family (CPython 3.12.13, torch 2.14.0+cu130, triton 3.8.0, transformers 5.17.0, flash-linear-attention 0.5.2); no price: no maker price for this checkpoint and no GPU-time rate for a DGX Spark in the runner
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "glide-0.4b",
    "--provider",
    "glide",
    "--timeout",
    "60",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "3b14502cefed852b2c6ac3a9139c7574549374b6",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "glide-0.4b",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "glide",
  "questions": 2163,
  "request_seconds": 23.382233941316372,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "436bb79641abdc545af38d8182c8c3ca8ad6aef98dd7cdbc85916773b9a21bba",
  "runner": "benchmark_ollama.py",
  "runner_commit": "9ba1b409d68af1a8b3c846f54993e438b9db212a",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 448290,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19350/v1/systemone
{
 "model": "glide-0.4b",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · glide-0.4b_en_dontknow

Name
GLiDE 0.4B
Setup
GLiDE 0.4B (open) · decision model · with the I don't know option F
Model
glide-0.4b · provider glide
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.001655 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.00778 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.002 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run glide-0.4b_en_dontknow · first answer 2026-11-10 13:01:54 UTC · last answer 2026-11-10 13:02:22 UTC · span 28.0 s, pauses included (not charged) · charged 28.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0017 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.8 in flight (sum of request latencies / active time)
Model size
Parameters: 0.40B dense · source https://huggingface.co/fastino/GLiDE-0.4B/tree/6bda7ab91aeb36dcad6d27828a8871e548c7fc71 · read 2026-11-10 · counted from the checkpoint (395,042,816: encoder 394,781,696, readout 261,120; an encoder, ModernBERT architecture); the card: 0.40B
Input
text, English
Endpoint
internal address on a shared Spark host
Started
10 Nov 2026 13:01 UTC; last answer 10 Nov 2026 13:02 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup GLiDE 0.4B (open) (site run None); runner 9ba1b409d; fastino/GLiDE-0.4B@6bda7ab91aeb36dcad6d27828a8871e548c7fc71 (Fastino Community License) on a shared Spark host (DGX Spark, GB10), served by the glide package the repository ships, one request at a time (GLIDE_MAX_BATCH=1): systemd-run --user --unit=logic-glide-0.4b --working-directory=~/glide-family/models/GLiDE-0.4B@6bda7ab91aeb36dcad6d27828a8871e548c7fc71 -E GLIDE_MAX_BATCH=1 -E PYTHONUNBUFFERED=1 -E HF_HUB_OFFLINE=1 ~/envs/glide-family/bin/python -m glide.server --model ~/glide-family/models/GLiDE-0.4B@6bda7ab91aeb36dcad6d27828a8871e548c7fc71 --host <a shared Spark host> --port 19350; env ~/envs/glide-family (CPython 3.12.13, torch 2.14.0+cu130, triton 3.8.0, transformers 5.17.0, flash-linear-attention 0.5.2); no price: no maker price for this checkpoint and no GPU-time rate for a DGX Spark in the runner
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "glide-0.4b",
    "--provider",
    "glide",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "3b14502cefed852b2c6ac3a9139c7574549374b6",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "glide-0.4b",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "glide",
  "questions": 2163,
  "request_seconds": 23.36070511757862,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "4e8b10d55e943d6316796ca2c8362075fd2789f13be2dfb74815e083ce7abcdc",
  "runner": "benchmark_ollama.py",
  "runner_commit": "9ba1b409d68af1a8b3c846f54993e438b9db212a",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 515343,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19350/v1/systemone
{
 "model": "glide-0.4b",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · gpt-6-luna_en

Name
GPT-6 Luna max · chat
Setup
GPT-6 Luna · chat model
Model
openai/gpt-6-luna · provider openrouter
GPU
—
Cost
$0.03644 for the run · provider price × reported usage
Answer time
run gpt-6-luna_en · first answer 2026-11-10 14:04:24 UTC · last answer 2026-11-10 14:07:34 UTC · span 190.0 s, pauses included (not charged) · charged 190.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 7.8 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: not disclosed by maker (closed weights) · source https://openrouter.ai/openai/gpt-6-luna · read 2026-11-10
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
10 Nov 2026 14:04 UTC; last answer 10 Nov 2026 14:07 UTC
Progress
200 answered of 2,163 asked; 0 request errors; paused for now at 200 of 2,163 (zeki #4129, 2026-11-10T14:13Z)
Notes
GPT-6 Luna through OpenRouter (openai/gpt-6-luna), provider pinned to openai, fallbacks off; reasoning {"effort": "max"}; temperature 1.0; max_tokens 32768; one question per request; parameters not disclosed (closed weights). English translations. PAUSED (zeki #4129, 2026-11-10T14:13Z): skipped for now, stopped at 200 answers; paid smoke stopped, no scoped budget yet; kept, not ranked
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "openai"
    ]
  },
  "reasoning": {
    "effort": "max"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · muse-ember_en

Name
Muse Ember max · chat
Setup
Muse Ember · chat model
Model
meta/muse-ember · provider openrouter
GPU
—
Cost
$0.7886 for the run · provider price × reported usage
Answer time
run muse-ember_en · first answer 2026-11-10 14:09:10 UTC · last answer 2026-11-10 15:15:15 UTC · span 3965.0 s, pauses included (not charged) · charged 467.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 8.0 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 2 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: not disclosed by maker (closed weights) · source https://openrouter.ai/meta/muse-ember · read 2026-11-10
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
10 Nov 2026 14:09 UTC; last answer 10 Nov 2026 15:15 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Muse Ember through OpenRouter (meta/muse-ember), provider pinned to meta, fallbacks off; reasoning {"effort": "max"}; temperature 1.0; max_tokens 32768; one question per request; parameters not disclosed (closed weights). English translations.
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "meta"
    ]
  },
  "reasoning": {
    "effort": "max"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · muse-ember_en_dontknow

Name
Muse Ember max · chat
Setup
Muse Ember · chat model · with the I don't know option F
Model
meta/muse-ember · provider openrouter
GPU
—
Cost
$0.7878 for the run · provider price × reported usage
Answer time
run muse-ember_en_dontknow · first answer 2026-11-10 15:15:16 UTC · last answer 2026-11-10 15:23:02 UTC · span 466.0 s, pauses included (not charged) · charged 466.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 8.0 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 2 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: not disclosed by maker (closed weights) · source https://openrouter.ai/meta/muse-ember · read 2026-11-10
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
10 Nov 2026 15:15 UTC; last answer 10 Nov 2026 15:23 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Muse Ember through OpenRouter (meta/muse-ember), provider pinned to meta, fallbacks off; reasoning {"effort": "max"}; temperature 1.0; max_tokens 32768; one question per request; parameters not disclosed (closed weights). English translations.
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "meta"
    ]
  },
  "reasoning": {
    "effort": "max"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · knowline-31b_en

Name
KnowLine-31B-Gen1.5
Setup
KnowLine 31B Gen1.5 · decision model
Model
PelaAI/KnowLine-31B-Gen1.5 · provider knowline
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.1014 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.476 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.101 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run knowline-31b_en · first answer 2026-11-10 15:46:33 UTC · last answer 2026-11-10 16:15:08 UTC · span 1715.0 s, pauses included (not charged) · charged 1715.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1014 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 31B dense · source https://huggingface.co/PelaAI/KnowLine-31B-Gen1.5 · read 2026-11-10 · the card states about 31B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 31.27B (counted from the checkpoint (31,273,088,876: language model 30,697,345,340, vision tower 575,743,536); a fine-tune of Gemma 4 31B IT; the card: about 31B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
10 Nov 2026 15:46 UTC; last answer 10 Nov 2026 16:15 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup KnowLine 31B Gen1.5 (site run None); runner 9ba1b409d; PelaAI/KnowLine-31B-Gen1.5@8e56aa1b236a6dd1064dcfd22ba1e5e360e4c140 (Apache-2.0 weights, MIT code; Gemma 4 31B IT fine-tune, readout temperature folded into the weights) behind the card's serve_knowline.sh: SGLang 0.5.21 (BF16, --enable-fp32-lm-head) on port+1000 and the repository's knowline_server.py front end (read twice for prompts up to 3,000 tokens): cd ~/decision-models/KnowLine-31B-Gen1.5@8e56aa1b236a6dd1064dcfd22ba1e5e360e4c140 && PATH=~/envs/knowline-sglang/bin:$PATH HOST=0.0.0.0 bash serve_knowline.sh ~/decision-models/KnowLine-31B-Gen1.5@8e56aa1b236a6dd1064dcfd22ba1e5e360e4c140 0 19360 (MEM default 0.85), env ~/envs/knowline-sglang (sglang==0.5.21, transformers==5.12.1, torch 2.13.0+cu130); vision on (the server's default); bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'knowline' (System One at /v1/systemone; model field ignored)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "PelaAI/KnowLine-31B-Gen1.5",
    "--provider",
    "knowline",
    "--timeout",
    "120",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "3b14502cefed852b2c6ac3a9139c7574549374b6",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "PelaAI/KnowLine-31B-Gen1.5",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "knowline",
  "questions": 2163,
  "request_seconds": 1699.2243719821563,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "c83fc738c39a896640b9d41ffebed23dfc3f64c847168c68b247ddf2368d5ff5",
  "runner": "benchmark_ollama.py",
  "runner_commit": "9ba1b409d68af1a8b3c846f54993e438b9db212a",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 925509,
    "output_tokens": 2163
  }
}
Prompt template
POST http://spark2:19360/v1/systemone
{
 "model": "PelaAI/KnowLine-31B-Gen1.5",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · knowline-31b_en_dontknow

Name
KnowLine-31B-Gen1.5
Setup
KnowLine 31B Gen1.5 · decision model · with the I don't know option F
Model
PelaAI/KnowLine-31B-Gen1.5 · provider knowline
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.1092 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.513 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.109 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run knowline-31b_en_dontknow · first answer 2026-11-10 16:36:52 UTC · last answer 2026-11-10 17:07:39 UTC · span 1847.0 s, pauses included (not charged) · charged 1847.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1092 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 31B dense · source https://huggingface.co/PelaAI/KnowLine-31B-Gen1.5 · read 2026-11-10 · the card states about 31B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 31.27B (counted from the checkpoint (31,273,088,876: language model 30,697,345,340, vision tower 575,743,536); a fine-tune of Gemma 4 31B IT; the card: about 31B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
10 Nov 2026 16:36 UTC; last answer 10 Nov 2026 17:07 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup KnowLine 31B Gen1.5 (site run None); runner 9ba1b409d; PelaAI/KnowLine-31B-Gen1.5@8e56aa1b236a6dd1064dcfd22ba1e5e360e4c140 (Apache-2.0 weights, MIT code; Gemma 4 31B IT fine-tune, readout temperature folded into the weights) behind the card's serve_knowline.sh: SGLang 0.5.21 (BF16, --enable-fp32-lm-head) on port+1000 and the repository's knowline_server.py front end (read twice for prompts up to 3,000 tokens): cd ~/decision-models/KnowLine-31B-Gen1.5@8e56aa1b236a6dd1064dcfd22ba1e5e360e4c140 && PATH=~/envs/knowline-sglang/bin:$PATH HOST=0.0.0.0 bash serve_knowline.sh ~/decision-models/KnowLine-31B-Gen1.5@8e56aa1b236a6dd1064dcfd22ba1e5e360e4c140 0 19360 (MEM default 0.85), env ~/envs/knowline-sglang (sglang==0.5.21, transformers==5.12.1, torch 2.13.0+cu130); vision on (the server's default); bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'knowline' (System One at /v1/systemone; model field ignored)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "PelaAI/KnowLine-31B-Gen1.5",
    "--provider",
    "knowline",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "3b14502cefed852b2c6ac3a9139c7574549374b6",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "PelaAI/KnowLine-31B-Gen1.5",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "knowline",
  "questions": 2163,
  "request_seconds": 1831.6187651634682,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "a65ada6f4fc771409f2c4eae96635b3ae32d3861c3a8bf8433f5d2c7eed47ffe",
  "runner": "benchmark_ollama.py",
  "runner_commit": "9ba1b409d68af1a8b3c846f54993e438b9db212a",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 1068267,
    "output_tokens": 2163
  }
}
Prompt template
POST http://spark2:19360/v1/systemone
{
 "model": "PelaAI/KnowLine-31B-Gen1.5",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · knowline-4b_en

Name
KnowLine 4B Gen5.5
Setup
KnowLine 4B Gen5.5 · decision model
Model
PelaAI/KnowLine-4B-Gen5.5 · provider knowline
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.009872 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0464 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.010 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run knowline-4b_en · first answer 2026-11-11 11:05:21 UTC · last answer 2026-11-11 11:08:08 UTC · span 167.0 s, pauses included (not charged) · charged 167.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0099 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 4B dense · source https://huggingface.co/PelaAI/KnowLine-4B-Gen5.5 · read 2026-11-10 · the card states 4B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 4.66B (counted from the checkpoint (4,659,865,088: language model 4,205,751,296, MTP layer 120,599,552 (model-extra-from-base.safetensors), vision tower 333,514,240); from Qwen3.5-4B; the card: 4B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
11 Nov 2026 11:05 UTC; last answer 11 Nov 2026 11:08 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup KnowLine 4B Gen5.5 (site run None); runner 5fb51788d; PelaAI/KnowLine-4B-Gen5.5@779944d2cc8ffa19cef386aee83c83bcb967ade2 (Apache-2.0 weights, MIT code; Qwen3.5-4B LoRA average, temperature folded into the weights) behind the card's serve_knowline.sh: SGLang 0.5.21 (FP8 at load) on port+1000 and knowline_server.py (read twice up to 3,000 tokens): cd ~/decision-models/KnowLine-4B-Gen5.5@779944d2cc8ffa19cef386aee83c83bcb967ade2 && PATH=~/envs/knowline-sglang/bin:$PATH HOST=0.0.0.0 bash serve_knowline.sh ~/decision-models/KnowLine-4B-Gen5.5@779944d2cc8ffa19cef386aee83c83bcb967ade2 0 19400 (MEM default 0.72), env ~/envs/knowline-sglang; vision on (the server's default); bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'knowline'
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "PelaAI/KnowLine-4B-Gen5.5",
    "--provider",
    "knowline",
    "--timeout",
    "60",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "b55c22654bc72e59b7a430a92a37731267ddb6de",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "PelaAI/KnowLine-4B-Gen5.5",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "knowline",
  "questions": 2163,
  "request_seconds": 162.4493859471113,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "50ad728c898b0dc825b77eb21671b66e1059ca1fd44ae1f21193a25015a62b14",
  "runner": "benchmark_ollama.py",
  "runner_commit": "5fb51788d3992285ed8f0cf76fd6e8716353b0d9",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 924182,
    "output_tokens": 2163
  }
}
Prompt template
POST http://spark2:19400/v1/systemone
{
 "model": "PelaAI/KnowLine-4B-Gen5.5",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · knowline-4b_en_dontknow

Name
KnowLine 4B Gen5.5
Setup
KnowLine 4B Gen5.5 · decision model · with the I don't know option F
Model
PelaAI/KnowLine-4B-Gen5.5 · provider knowline
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.01046 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0492 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.010 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run knowline-4b_en_dontknow · first answer 2026-11-11 11:10:11 UTC · last answer 2026-11-11 11:13:08 UTC · span 177.0 s, pauses included (not charged) · charged 177.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0105 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 4B dense · source https://huggingface.co/PelaAI/KnowLine-4B-Gen5.5 · read 2026-11-10 · the card states 4B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 4.66B (counted from the checkpoint (4,659,865,088: language model 4,205,751,296, MTP layer 120,599,552 (model-extra-from-base.safetensors), vision tower 333,514,240); from Qwen3.5-4B; the card: 4B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
11 Nov 2026 11:10 UTC; last answer 11 Nov 2026 11:13 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup KnowLine 4B Gen5.5 (site run None); runner 5fb51788d; PelaAI/KnowLine-4B-Gen5.5@779944d2cc8ffa19cef386aee83c83bcb967ade2 (Apache-2.0 weights, MIT code; Qwen3.5-4B LoRA average, temperature folded into the weights) behind the card's serve_knowline.sh: SGLang 0.5.21 (FP8 at load) on port+1000 and knowline_server.py (read twice up to 3,000 tokens): cd ~/decision-models/KnowLine-4B-Gen5.5@779944d2cc8ffa19cef386aee83c83bcb967ade2 && PATH=~/envs/knowline-sglang/bin:$PATH HOST=0.0.0.0 bash serve_knowline.sh ~/decision-models/KnowLine-4B-Gen5.5@779944d2cc8ffa19cef386aee83c83bcb967ade2 0 19400 (MEM default 0.72), env ~/envs/knowline-sglang; vision on (the server's default); bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'knowline'
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "PelaAI/KnowLine-4B-Gen5.5",
    "--provider",
    "knowline",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "b55c22654bc72e59b7a430a92a37731267ddb6de",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "PelaAI/KnowLine-4B-Gen5.5",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "knowline",
  "questions": 2163,
  "request_seconds": 171.4305778639682,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "cf2e2ed9f71fd817f2ae6eec1263bbce1b4a4e2433bcb4532b7c26625a38fecb",
  "runner": "benchmark_ollama.py",
  "runner_commit": "5fb51788d3992285ed8f0cf76fd6e8716353b0d9",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 1053962,
    "output_tokens": 2163
  }
}
Prompt template
POST http://spark2:19400/v1/systemone
{
 "model": "PelaAI/KnowLine-4B-Gen5.5",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · gemini-4-carbon_en

Name
Gemini 4 Carbon high · chat
Setup
Gemini 4 Carbon · chat model
Model
google/gemini-4-carbon · provider openrouter
GPU
—
Cost
not measured
Answer time
run gemini-4-carbon_en · first answer 2026-11-11 11:49:11 UTC · last answer 2026-11-11 11:51:20 UTC · span 129.0 s, pauses included (not charged) · charged 129.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 7.9 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: not disclosed by maker (closed weights) · source https://ai.google.dev/gemini-api/docs/models · read 2026-11-11
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
11 Nov 2026 11:48 UTC; last answer 11 Nov 2026 11:51 UTC
Progress
54 answered of 2,163 asked; 0 request errors; paused for now at 54 of 2,163 (zeki #4681, 2026-11-11T11:53Z)
Notes
Gemini 4 Carbon through OpenRouter (google/gemini-4-carbon), provider pinned to google-ai-studio, fallbacks off; reasoning {"effort": "high"}; temperature 1.0; max_tokens 32768; one question per request; parameters not disclosed (closed weights). English translations. PAUSED (zeki #4681, 2026-11-11T11:53Z): skipped for now, stopped at 54 answers; no scoped human budget for paid calls yet; kept, not ranked
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "google-ai-studio"
    ]
  },
  "reasoning": {
    "effort": "high"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · claude-fable-5.2-sub_en

Name
Claude Fable 5.2 chat
Setup
Claude Fable 5.2 · chat model
Model
claude-fable-5-2 · provider claude-subscription
GPU
—
Cost
$0 for the run · Claude subscription
Answer time
run claude-fable-5.2-sub_en · first answer 2026-11-11 12:02:08 UTC · last answer 2026-11-11 12:02:22 UTC · span 14.0 s, pauses included (not charged) · charged 14.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.3 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker (closed weights) · source https://platform.claude.com/docs/en/about-claude/models · read 2026-11-13
Input
text, English
Endpoint
claude-code-subscription
Started
11 Nov 2026 12:02 UTC; last answer 11 Nov 2026 12:02 UTC
Progress
5 answered of 2,163 asked; 0 request errors; in progress
Notes
Claude Fable 5.2 on the Claude subscription through the Claude Code CLI (`claude --print`, as turkce's runs 968/971), model claude-fable-5-2, --effort max (the highest of low/medium/high/xhigh/max; adaptive thinking always on); CLAUDE_CODE_MAX_OUTPUT_TOKENS 32768; no temperature sent (Anthropic's default 1.0); Claude Code's agent prompt replaced by a neutral one-line system prompt, tools, settings, MCP servers and session saving off, API-key variables removed; one question per request, one request at a time; cost $0 (Claude subscription); api_equivalent_usd is Claude Code's API-list-price estimate, not a cost. English translations. 12 Nov 2026: 6 Logic replies stored as request errors (logic benchmark.db rows 238666, 238694, 241498, 242157, 242300, 243470) re-scored in place as the model's outcome, no letter, counted wrong, the CLI's stored record kept as the reply: 5 API refusals (finish_reason 'refusal') and 1 reply cut at the 32,768-token cap (finish_reason 'length'); run_eval.py now stores such Claude Code replies the same way and never retries or re-asks them (zeki #5263).
Parameters
{
  "effort": "max",
  "max_output_tokens": 32768,
  "route": "claude-code-cli --print",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message."
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · startlux-2b_en

Name
StartLux-Decision-v1.1-2B-Instant
Setup
StartLux-Decision v1.1 2B · decision model
Model
startlux-models/StartLux-Decision-v1.1-2B-Instant · provider startlux
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.004907 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0231 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.005 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run startlux-2b_en · first answer 2026-11-11 12:24:23 UTC · last answer 2026-11-11 12:25:46 UTC · span 83.0 s, pauses included (not charged) · charged 83.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0049 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 2B dense · source https://huggingface.co/startlux-models/StartLux-Decision-v1.1-2B-Instant · read 2026-11-10 · the card states 2B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 2.27B (counted from the checkpoint (2,274,069,824: language model 1,881,825,088, MTP layer 60,828,160, vision tower 331,416,576; dense, no experts); from Qwen3.5-2B; the card: 2B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
11 Nov 2026 12:24 UTC; last answer 11 Nov 2026 12:25 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup StartLux-Decision v1.1 2B (site run None); runner 5fb51788d; startlux-models/StartLux-Decision-v1.1-2B-Instant@f1e6cd147dda82c9ffa3e8e725d5c1f175566fb1 (weights CC BY-NC 4.0, non-commercial: run for this non-commercial benchmark; code Apache-2.0; Qwen3.5-2B, per-type temperatures from decision_config.json (choice 1.2937), prefix-bidirectional vLLM plugin) behind the card's server (github.com/StartLuxLabs/StartLux-Decision at a99977d): ~/envs/startlux/bin/startlux-decide serve --model ~/decision-models/StartLux-Decision-v1.1-2B-Instant@f1e6cd147dda82c9ffa3e8e725d5c1f175566fb1 --launch-vllm --gpus 0 --host 0.0.0.0 --port 19440 (env ~/envs/startlux: vllm==0.31.0, torch 2.13.0+cu130, the repository at a99977d with [vllm]; vLLM 0.31.0 because 0.19.1, the version of the card's published scores, has no aarch64 build; the plugin is validated on both); vision on (the server's default); bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'startlux' (System One at /v1/systemone, model ignored)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "startlux-models/StartLux-Decision-v1.1-2B-Instant",
    "--provider",
    "startlux",
    "--timeout",
    "60",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "b55c22654bc72e59b7a430a92a37731267ddb6de",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "startlux-models/StartLux-Decision-v1.1-2B-Instant",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "startlux",
  "questions": 2163,
  "request_seconds": 77.88105855556205,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "c6c2fca6629ed7529395d34ac1a81d12807c77a4cf96a343722903b9e2c004b8",
  "runner": "benchmark_ollama.py",
  "runner_commit": "5fb51788d3992285ed8f0cf76fd6e8716353b0d9",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 416668,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19440/v1/systemone
{
 "model": "startlux-models/StartLux-Decision-v1.1-2B-Instant",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · startlux-2b_en_dontknow

Name
StartLux-Decision-v1.1-2B-Instant
Setup
StartLux-Decision v1.1 2B · decision model · with the I don't know option F
Model
startlux-models/StartLux-Decision-v1.1-2B-Instant · provider startlux
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.004966 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0233 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.005 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run startlux-2b_en_dontknow · first answer 2026-11-11 12:26:41 UTC · last answer 2026-11-11 12:28:05 UTC · span 84.0 s, pauses included (not charged) · charged 84.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0050 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 2B dense · source https://huggingface.co/startlux-models/StartLux-Decision-v1.1-2B-Instant · read 2026-11-10 · the card states 2B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 2.27B (counted from the checkpoint (2,274,069,824: language model 1,881,825,088, MTP layer 60,828,160, vision tower 331,416,576; dense, no experts); from Qwen3.5-2B; the card: 2B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
11 Nov 2026 12:26 UTC; last answer 11 Nov 2026 12:28 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup StartLux-Decision v1.1 2B (site run None); runner 5fb51788d; startlux-models/StartLux-Decision-v1.1-2B-Instant@f1e6cd147dda82c9ffa3e8e725d5c1f175566fb1 (weights CC BY-NC 4.0, non-commercial: run for this non-commercial benchmark; code Apache-2.0; Qwen3.5-2B, per-type temperatures from decision_config.json (choice 1.2937), prefix-bidirectional vLLM plugin) behind the card's server (github.com/StartLuxLabs/StartLux-Decision at a99977d): ~/envs/startlux/bin/startlux-decide serve --model ~/decision-models/StartLux-Decision-v1.1-2B-Instant@f1e6cd147dda82c9ffa3e8e725d5c1f175566fb1 --launch-vllm --gpus 0 --host 0.0.0.0 --port 19440 (env ~/envs/startlux: vllm==0.31.0, torch 2.13.0+cu130, the repository at a99977d with [vllm]; vLLM 0.31.0 because 0.19.1, the version of the card's published scores, has no aarch64 build; the plugin is validated on both); vision on (the server's default); bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'startlux' (System One at /v1/systemone, model ignored)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "startlux-models/StartLux-Decision-v1.1-2B-Instant",
    "--provider",
    "startlux",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "b55c22654bc72e59b7a430a92a37731267ddb6de",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "startlux-models/StartLux-Decision-v1.1-2B-Instant",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "startlux",
  "questions": 2163,
  "request_seconds": 78.83271049533505,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "7d5090cab6e382936f2fb1f7bfc95599c59bf0ed9f67ef651c17afdb6eba6813",
  "runner": "benchmark_ollama.py",
  "runner_commit": "5fb51788d3992285ed8f0cf76fd6e8716353b0d9",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 477232,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19440/v1/systemone
{
 "model": "startlux-models/StartLux-Decision-v1.1-2B-Instant",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · gemini-4-carbon-sub_en

Name
Gemini 4 Carbon chat
Setup
Gemini 4 Carbon (Antigravity subscription) · chat model
Model
gemini-4-carbon · provider Antigravity (Gemini subscription)
GPU
—
Cost
not measured
Answer time
run gemini-4-carbon-sub_en · first answer 2026-11-11 13:26:58 UTC · last answer 2026-11-15 10:00:34 UTC · span 333216.0 s, pauses included (not charged) · charged 22803.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 2.0 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 3 empty replies asked once more
Model size
Parameters: not disclosed by maker (closed weights) · source https://ai.google.dev/gemini-api/docs/models · read 2026-11-11
Input
text, English
Endpoint
http://lan-host:8899/v1/chat/completions
Started
11 Nov 2026 13:26 UTC; last answer 15 Nov 2026 10:00 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Pilot (Can via personal-assistant #4757): Gemini 4 Carbon through Can's Antigravity subscription (antigravity-benchmark #4762; model gemini-4-carbon = gemini-4-carbon-high, tools denied, ~11.2k tokens of Antigravity's own context per request, one request at a time). Same 54 Math questions and prompt as the paid API run gemini-4-carbon_en; not ranked, not published. 12 Nov 19:02 UTC: label changed from 'Gemini 4 Carbon (Antigravity pilot)' to the full run's label: the 54 pilot answers are the start of the approved subscription run (Umit via zeki #5133), not a separate setup. 2026-11-14T01:20:09Z: the Antigravity (Gemini subscription) bridge now answers up to 4 requests at once (antigravity-benchmark #6946); this run still sends one at a time; Carbon, Hydrogen and turkce's Reading arms share the bridge, so a request time can include the bridge's queue. 2026-11-14T06:49:26Z: two requests at a time from here on (turkce freed one of her bridge lanes, #7088); before, one at a time. Accuracy unaffected; request times not comparable across the switch. Bridge address changed 14 Nov 17:44 UTC from lan-host to lan-host (the Mac moved to its wired adapter after a network change; same Antigravity (Gemini subscription) service, same settings; antigravity-benchmark #7808); answers after that came over the new address.
Parameters
{
  "max_tokens": 32768,
  "temperature": 1.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · grok-4.7-sub_en

Name
Grok 4.7 chat
Setup
Grok 4.7 · chat model
Model
grok-4.7 · provider xai-subscription
GPU
—
Cost
$26.09 for the run, API list-price equivalent · xAI subscription
Answer time
run grok-4.7-sub_en · first answer 2026-11-11 15:17:40 UTC · last answer 2026-11-12 02:19:07 UTC · span 39687.0 s, pauses included (not charged) · charged 37319.0 s in 3 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.9 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker (closed weights) · source https://x.ai · read 2026-11-11
Input
text, English
Endpoint
https://api.x.ai/v1/responses
Started
11 Nov 2026 15:17 UTC; last answer 12 Nov 2026 02:19 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Grok 4.7 on the SuperGrok subscription through xAI's Responses API with OMP's xai-oauth login (as turkce's ask_xai and jevgammon's xai-subscription route), model grok-4.7, reasoning effort high (Reading's setting); max_output_tokens 32768; no temperature sent (xAI's default) and no tools; a neutral one-line system prompt; one question per request, one request at a time; a usage-limit reply stops the run without retrying it (resumable); cost $0 (xAI subscription); api_equivalent_usd is the tokens at xAI's API list price {"input": 2.0, "cached_input": 0.5, "output": 6.0} per 1M tokens, not a cost. English translations.
Parameters
{
  "effort": "high",
  "list_price_per_mtok": {
    "cached_input": 0.5,
    "input": 2.0,
    "output": 6.0
  },
  "max_output_tokens": 32768,
  "route": "xai responses api (OMP xai-oauth)",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message.",
  "tools": "none"
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · qwen3-14b_en

Name
Qwen3-14B off · chat
Setup
Qwen3-14B (chat, reasoning off, on our GPU) · chat model
Model
qwen3-14b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.7678 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.308 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.768. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-11 17:10:54–18:12:50 UTC
GPU time
run qwen3-14b_en · first answer 2026-11-11 17:10:26 UTC · last answer 2026-11-11 18:30:52 UTC · span 4826.0 s, pauses included (not charged) · charged 1110.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.7678 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 27.9 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 40 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 14.8B dense · source https://huggingface.co/Qwen/Qwen3-14B · read 2026-11-11 · dense; the maker's stated figure (14.8B, the card) is shown, as for every model (turkce #4263); checkpoint count 14.77B
Input
text, English
Endpoint
http://127.0.0.1:19500/v1/chat/completions
Started
11 Nov 2026 17:10 UTC; last answer 11 Nov 2026 18:30 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Qwen3-14B as Reading's setup: Qwen/Qwen3-14B@40c069824f4251a91eefaf281ebe4c544efd3e18 BF16, served on our GPU host (one RTX PRO 6000) via docker vLLM 0.30.0 (vllm/vllm-openai:v0.30.0) from the HF cache as Reading's runs, max-model-len 16,384, gpu-memory-utilization 0.85; reasoning off as Reading's requests (chat_template_kwargs enable_thinking=false, thinking=false), temperature 0, seed 0; max_tokens 8192; one question per request (32 requests in flight; 8 for the first 25 answers), line prompt 'ANSWER: X', no system message and no JSON schema (Reading: system prompt, JSON schema, 16 questions per request). English translations.
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "max_tokens": 8192,
  "seed": 0,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · knowline-0.8b_en

Name
KnowLine-0.8B-Gen1
Setup
KnowLine-0.8B-Gen1 · decision model
Model
PelaAI/KnowLine-0.8B-Gen1 · provider knowline
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.003665 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0172 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.004 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run knowline-0.8b_en · first answer 2026-11-11 18:10:31 UTC · last answer 2026-11-11 18:11:33 UTC · span 62.0 s, pauses included (not charged) · charged 62.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0037 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 0.8B dense · source https://huggingface.co/PelaAI/KnowLine-0.8B-Gen1 · read 2026-11-11 · the card states a 0.8B decision model; checkpoint count 0.87B (counted from the checkpoint (873,438,784: language model 752,393,024, MTP layer 20,452,864 (model-extra-from-base.safetensors), vision tower 100,592,896); a LoRA fine-tune of Qwen3.5-0.8B merged into the weights; dense; the card: 0.8B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
11 Nov 2026 18:10 UTC; last answer 11 Nov 2026 18:11 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup KnowLine-0.8B-Gen1 (site run None); runner 5fb51788d; PelaAI/KnowLine-0.8B-Gen1@6b27831f7b783c03278bd946d4d8f4f7a98a3798 (Apache-2.0 weights and code; one-epoch LoRA SFT of Qwen3.5-0.8B merged into BF16, readout temperature 1.54 folded into the weights, score questions divided by temperatures.json) behind the card's serve_knowline.sh: SGLang 0.5.21 in BF16 (--enable-fp32-lm-head --mamba-radix-cache-strategy extra_buffer, MEM default 0.85) on port+1000 (20450) and knowline_server.py (read twice up to 3,000 tokens): cd ~/decision-models/KnowLine-0.8B-Gen1@6b27831f7b783c03278bd946d4d8f4f7a98a3798 && PATH=~/envs/knowline-sglang/bin:$PATH HOST=0.0.0.0 bash serve_knowline.sh ~/decision-models/KnowLine-0.8B-Gen1@6b27831f7b783c03278bd946d4d8f4f7a98a3798 0 19450, env ~/envs/knowline-sglang (sglang 0.5.21, torch 2.13.0+cu130, transformers 5.12.1); vram_mib is MEM 0.85 of the unified memory (SGLang sizes its KV cache to it), so it cannot run beside another large server; vision on (the server's default); bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'knowline'
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "PelaAI/KnowLine-0.8B-Gen1",
    "--provider",
    "knowline",
    "--timeout",
    "60",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "b55c22654bc72e59b7a430a92a37731267ddb6de",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "PelaAI/KnowLine-0.8B-Gen1",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "knowline",
  "questions": 2163,
  "request_seconds": 55.21783578576287,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "2ff4cd08d8f28c915db388bb7527c0ccb77896413b6e0b4085df98c59b94aceb",
  "runner": "benchmark_ollama.py",
  "runner_commit": "5fb51788d3992285ed8f0cf76fd6e8716353b0d9",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 924182,
    "output_tokens": 2163
  }
}
Prompt template
POST http://spark2:19450/v1/systemone
{
 "model": "PelaAI/KnowLine-0.8B-Gen1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · knowline-0.8b_en_dontknow

Name
KnowLine-0.8B-Gen1
Setup
KnowLine-0.8B-Gen1 · decision model · with the I don't know option F
Model
PelaAI/KnowLine-0.8B-Gen1 · provider knowline
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.003665 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0172 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.004 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run knowline-0.8b_en_dontknow · first answer 2026-11-11 18:12:19 UTC · last answer 2026-11-11 18:13:21 UTC · span 62.0 s, pauses included (not charged) · charged 62.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0037 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 0.8B dense · source https://huggingface.co/PelaAI/KnowLine-0.8B-Gen1 · read 2026-11-11 · the card states a 0.8B decision model; checkpoint count 0.87B (counted from the checkpoint (873,438,784: language model 752,393,024, MTP layer 20,452,864 (model-extra-from-base.safetensors), vision tower 100,592,896); a LoRA fine-tune of Qwen3.5-0.8B merged into the weights; dense; the card: 0.8B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
11 Nov 2026 18:12 UTC; last answer 11 Nov 2026 18:13 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup KnowLine-0.8B-Gen1 (site run None); runner 5fb51788d; PelaAI/KnowLine-0.8B-Gen1@6b27831f7b783c03278bd946d4d8f4f7a98a3798 (Apache-2.0 weights and code; one-epoch LoRA SFT of Qwen3.5-0.8B merged into BF16, readout temperature 1.54 folded into the weights, score questions divided by temperatures.json) behind the card's serve_knowline.sh: SGLang 0.5.21 in BF16 (--enable-fp32-lm-head --mamba-radix-cache-strategy extra_buffer, MEM default 0.85) on port+1000 (20450) and knowline_server.py (read twice up to 3,000 tokens): cd ~/decision-models/KnowLine-0.8B-Gen1@6b27831f7b783c03278bd946d4d8f4f7a98a3798 && PATH=~/envs/knowline-sglang/bin:$PATH HOST=0.0.0.0 bash serve_knowline.sh ~/decision-models/KnowLine-0.8B-Gen1@6b27831f7b783c03278bd946d4d8f4f7a98a3798 0 19450, env ~/envs/knowline-sglang (sglang 0.5.21, torch 2.13.0+cu130, transformers 5.12.1); vram_mib is MEM 0.85 of the unified memory (SGLang sizes its KV cache to it), so it cannot run beside another large server; vision on (the server's default); bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'knowline'
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "PelaAI/KnowLine-0.8B-Gen1",
    "--provider",
    "knowline",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "b55c22654bc72e59b7a430a92a37731267ddb6de",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "PelaAI/KnowLine-0.8B-Gen1",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "knowline",
  "questions": 2163,
  "request_seconds": 58.24401554316864,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "ffe5016eded29e0aed43f8b47f9399af638d01098c0d472ed32b9d3d0d112b4f",
  "runner": "benchmark_ollama.py",
  "runner_commit": "5fb51788d3992285ed8f0cf76fd6e8716353b0d9",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 1053962,
    "output_tokens": 2163
  }
}
Prompt template
POST http://spark2:19450/v1/systemone
{
 "model": "PelaAI/KnowLine-0.8B-Gen1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · qwen3-14b_en_dontknow

Name
Qwen3-14B off · chat
Setup
Qwen3-14B (chat, reasoning off, on our GPU) · chat model · with the I don't know option F
Model
qwen3-14b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.6744 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.271 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.674. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run qwen3-14b_en_dontknow · first answer 2026-11-11 18:30:52 UTC · last answer 2026-11-11 18:47:07 UTC · span 975.0 s, pauses included (not charged) · charged 975.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.6744 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 27.2 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 29 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 14.8B dense · source https://huggingface.co/Qwen/Qwen3-14B · read 2026-11-11 · dense; the maker's stated figure (14.8B, the card) is shown, as for every model (turkce #4263); checkpoint count 14.77B
Input
text, English
Endpoint
http://127.0.0.1:19500/v1/chat/completions
Started
11 Nov 2026 18:30 UTC; last answer 11 Nov 2026 18:47 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Qwen3-14B as Reading's setup: Qwen/Qwen3-14B@40c069824f4251a91eefaf281ebe4c544efd3e18 BF16, served on our GPU host (one RTX PRO 6000) via docker vLLM 0.30.0 (vllm/vllm-openai:v0.30.0) from the HF cache as Reading's runs, max-model-len 16,384, gpu-memory-utilization 0.85; reasoning off as Reading's requests (chat_template_kwargs enable_thinking=false, thinking=false), temperature 0, seed 0; max_tokens 8192; one question per request (32 in flight), line prompt 'ANSWER: X', no system message and no JSON schema (Reading: system prompt, JSON schema, 16 questions per request). English translations.
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "max_tokens": 8192,
  "seed": 0,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · erk-14b_en

Name
Erk-14B off · chat
Setup
Erk-14B (chat, reasoning off, on our GPU) · chat model
Model
erk-14b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.5755 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.231 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.575. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-11 18:52:50–19:11:38 UTC
GPU time
run erk-14b_en · first answer 2026-11-11 18:50:10 UTC · last answer 2026-11-11 19:22:50 UTC · span 1960.0 s, pauses included (not charged) · charged 832.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.5755 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 25.8 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 11 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 14.77B dense · source https://huggingface.co/ecloudtech/Erk-14B · read 2026-11-11 · dense; counted from the released checkpoint
Input
text, English
Endpoint
http://127.0.0.1:19501/v1/chat/completions
Started
11 Nov 2026 18:50 UTC; last answer 11 Nov 2026 19:22 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Erk-14B as Reading's setup: ecloudtech/Erk-14B@8584b14a829d5cc803bc227f44e9012e36e8ad04 BF16, served on our GPU host (one RTX PRO 6000) via docker vLLM 0.30.0 (vllm/vllm-openai:v0.30.0) from the HF cache as Reading's runs, max-model-len 16,384, gpu-memory-utilization 0.85; reasoning off as Reading's requests (chat_template_kwargs enable_thinking=false, thinking=false), temperature 0, seed 0; max_tokens 8192; one question per request (32 in flight), line prompt 'ANSWER: X', no system message and no JSON schema (Reading: system prompt, JSON schema, 16 questions per request); with no system message Erk-14B's chat template inserts its own default (Turkish identity) system prompt. English translations.
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "max_tokens": 8192,
  "seed": 0,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · erk-14b_en_dontknow

Name
Erk-14B off · chat
Setup
Erk-14B (chat, reasoning off, on our GPU) · chat model · with the I don't know option F
Model
erk-14b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.5782 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.232 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.578. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run erk-14b_en_dontknow · first answer 2026-11-11 19:22:52 UTC · last answer 2026-11-11 19:36:48 UTC · span 836.0 s, pauses included (not charged) · charged 836.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.5782 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 26.5 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 18 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 14.77B dense · source https://huggingface.co/ecloudtech/Erk-14B · read 2026-11-11 · dense; counted from the released checkpoint
Input
text, English
Endpoint
http://127.0.0.1:19501/v1/chat/completions
Started
11 Nov 2026 19:22 UTC; last answer 11 Nov 2026 19:36 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Erk-14B as Reading's setup: ecloudtech/Erk-14B@8584b14a829d5cc803bc227f44e9012e36e8ad04 BF16, served on our GPU host (one RTX PRO 6000) via docker vLLM 0.30.0 (vllm/vllm-openai:v0.30.0) from the HF cache as Reading's runs, max-model-len 16,384, gpu-memory-utilization 0.85; reasoning off as Reading's requests (chat_template_kwargs enable_thinking=false, thinking=false), temperature 0, seed 0; max_tokens 8192; one question per request (32 in flight), line prompt 'ANSWER: X', no system message and no JSON schema (Reading: system prompt, JSON schema, 16 questions per request); with no system message Erk-14B's chat template inserts its own default (Turkish identity) system prompt. English translations.
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "thinking": false
  },
  "max_tokens": 8192,
  "seed": 0,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · claude-opus-5.5-sub_en

Name
Claude Opus 5.5 chat
Setup
Claude Opus 5.5 · chat model
Model
claude-opus-5-5 · provider claude-subscription
GPU
—
Cost
$21.35 for the run, API list-price equivalent · Claude subscription
Answer time
run claude-opus-5.5-sub_en · first answer 2026-11-11 19:31:58 UTC · last answer 2026-11-11 21:37:27 UTC · span 7529.0 s, pauses included (not charged) · charged 7529.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker (closed weights) · source https://platform.claude.com/docs/en/about-claude/models · read 2026-11-13
Input
text, English
Endpoint
claude-code-subscription
Started
11 Nov 2026 19:31 UTC; last answer 11 Nov 2026 21:37 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Claude Opus 5.5 on the Claude subscription through the Claude Code CLI (`claude --print`, as turkce's runs 968/971), model claude-opus-5-5, --effort high (the highest of low/medium/high/xhigh/max; adaptive thinking always on); CLAUDE_CODE_MAX_OUTPUT_TOKENS 32768; no temperature sent (Anthropic's default 1.0); Claude Code's agent prompt replaced by a neutral one-line system prompt, tools, settings, MCP servers and session saving off, API-key variables removed; one question per request, one request at a time; cost $0 (Claude subscription); api_equivalent_usd is Claude Code's API-list-price estimate, not a cost. English translations.
Parameters
{
  "effort": "high",
  "max_output_tokens": 32768,
  "route": "claude-code-cli --print",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message."
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · startlux-0.8b_en

Name
StartLux-Decision-v1.1-0.8B-Instant
Setup
StartLux-Decision-v1.1-0.8B-Instant · decision model
Model
startlux-models/StartLux-Decision-v1.1-0.8B-Instant · provider startlux
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.00331 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0156 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.003 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run startlux-0.8b_en · first answer 2026-11-11 19:33:21 UTC · last answer 2026-11-11 19:34:17 UTC · span 56.0 s, pauses included (not charged) · charged 56.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0033 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 0.8B dense · source https://huggingface.co/startlux-models/StartLux-Decision-v1.1-0.8B-Instant · read 2026-11-11 · the card states a 0.8B decision model; checkpoint count 0.87B (counted from the checkpoint (873,438,784: language model incl. MTP layer 772,845,888, vision tower 100,592,896); from Qwen3.5-0.8B; dense; the card: 0.8B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
11 Nov 2026 19:33 UTC; last answer 11 Nov 2026 19:34 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup StartLux-Decision-v1.1-0.8B-Instant (site run None); runner 5fb51788d; startlux-models/StartLux-Decision-v1.1-0.8B-Instant@6496ae641cc71b149ce9dad69bd97f2a44e7bf11 (weights CC BY-NC 4.0, non-commercial: run for this non-commercial benchmark, as StartLux 4B and 2B; code Apache-2.0; Qwen3.5-0.8B, per-type temperatures from decision_config.json (choice 1.4364), prefix-bidirectional vLLM plugin) behind the card's server (github.com/StartLuxLabs/StartLux-Decision at a99977d): ~/envs/startlux/bin/startlux-decide serve --model ~/decision-models/StartLux-Decision-v1.1-0.8B-Instant@6496ae641cc71b149ce9dad69bd97f2a44e7bf11 --launch-vllm --gpus 0 --host 0.0.0.0 --port 19460 (env ~/envs/startlux: vllm==0.31.0, torch 2.13.0+cu130, the repository at a99977d with [vllm]; vLLM 0.31.0 because 0.19.1, the version of the card's published scores, has no aarch64 build; the card supports 0.31); vram_mib is vLLM's --gpu-memory-utilization default 0.88 of the unified memory, so it cannot run beside another large server; vision on (the server's default); bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'startlux' (System One at /v1/systemone, model ignored)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "startlux-models/StartLux-Decision-v1.1-0.8B-Instant",
    "--provider",
    "startlux",
    "--timeout",
    "60",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "b55c22654bc72e59b7a430a92a37731267ddb6de",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "startlux-models/StartLux-Decision-v1.1-0.8B-Instant",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "startlux",
  "questions": 2163,
  "request_seconds": 51.167206322570564,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "55d9817a5924cb3c884527a1474385d2beaa033ae1f9d6beee56a38aff0a6631",
  "runner": "benchmark_ollama.py",
  "runner_commit": "5fb51788d3992285ed8f0cf76fd6e8716353b0d9",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 416668,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19460/v1/systemone
{
 "model": "startlux-models/StartLux-Decision-v1.1-0.8B-Instant",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · startlux-0.8b_en_dontknow

Name
StartLux-Decision-v1.1-0.8B-Instant
Setup
StartLux-Decision-v1.1-0.8B-Instant · decision model · with the I don't know option F
Model
startlux-models/StartLux-Decision-v1.1-0.8B-Instant · provider startlux
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.003429 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0161 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.003 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run startlux-0.8b_en_dontknow · first answer 2026-11-11 19:35:00 UTC · last answer 2026-11-11 19:35:58 UTC · span 58.0 s, pauses included (not charged) · charged 58.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0034 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 0.8B dense · source https://huggingface.co/startlux-models/StartLux-Decision-v1.1-0.8B-Instant · read 2026-11-11 · the card states a 0.8B decision model; checkpoint count 0.87B (counted from the checkpoint (873,438,784: language model incl. MTP layer 772,845,888, vision tower 100,592,896); from Qwen3.5-0.8B; dense; the card: 0.8B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
11 Nov 2026 19:35 UTC; last answer 11 Nov 2026 19:35 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup StartLux-Decision-v1.1-0.8B-Instant (site run None); runner 5fb51788d; startlux-models/StartLux-Decision-v1.1-0.8B-Instant@6496ae641cc71b149ce9dad69bd97f2a44e7bf11 (weights CC BY-NC 4.0, non-commercial: run for this non-commercial benchmark, as StartLux 4B and 2B; code Apache-2.0; Qwen3.5-0.8B, per-type temperatures from decision_config.json (choice 1.4364), prefix-bidirectional vLLM plugin) behind the card's server (github.com/StartLuxLabs/StartLux-Decision at a99977d): ~/envs/startlux/bin/startlux-decide serve --model ~/decision-models/StartLux-Decision-v1.1-0.8B-Instant@6496ae641cc71b149ce9dad69bd97f2a44e7bf11 --launch-vllm --gpus 0 --host 0.0.0.0 --port 19460 (env ~/envs/startlux: vllm==0.31.0, torch 2.13.0+cu130, the repository at a99977d with [vllm]; vLLM 0.31.0 because 0.19.1, the version of the card's published scores, has no aarch64 build; the card supports 0.31); vram_mib is vLLM's --gpu-memory-utilization default 0.88 of the unified memory, so it cannot run beside another large server; vision on (the server's default); bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'startlux' (System One at /v1/systemone, model ignored)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "startlux-models/StartLux-Decision-v1.1-0.8B-Instant",
    "--provider",
    "startlux",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "b55c22654bc72e59b7a430a92a37731267ddb6de",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "startlux-models/StartLux-Decision-v1.1-0.8B-Instant",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "startlux",
  "questions": 2163,
  "request_seconds": 52.60540234367363,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "9dd867b97bed5e3831910c81aac227bea9ca58c20a51c98209bf7bedbda2c8e0",
  "runner": "benchmark_ollama.py",
  "runner_commit": "5fb51788d3992285ed8f0cf76fd6e8716353b0d9",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 477232,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19460/v1/systemone
{
 "model": "startlux-models/StartLux-Decision-v1.1-0.8B-Instant",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · gemini-decision-1_en

Name
Gemini Decision 1 20261111
Setup
Gemini Decision 1 · decision model
Model
google/gemini-decision-1 · provider openrouter
GPU
—
Cost
$0.04764 for the run · provider price × reported usage
Answer time
run gemini-decision-1_en · first answer 2026-11-11 19:46:07 UTC · last answer 2026-11-11 20:07:59 UTC · span 1312.0 s, pauses included (not charged) · charged 1312.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 0.5 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker (closed weights) · source https://blog.google/innovation-and-ai/technology/developers-tools/gemini-decisions-api/ · read 2026-11-11
Input
text, English
Endpoint
https://openrouter.ai/api
Started
11 Nov 2026 19:46 UTC; last answer 11 Nov 2026 20:07 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Gemini Decision 1 (site run None); runner 7b6fdc115; via OpenRouter's System One route (Google AI Studio serves it; replies name google/gemini-decision-1-20261111); latency includes OpenRouter's time; text only on Logic & Math. Launched 11 Nov 2026; Umit via Can #4966.
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "google/gemini-decision-1",
    "--provider",
    "openrouter",
    "--timeout",
    "60",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "b55c22654bc72e59b7a430a92a37731267ddb6de",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "google/gemini-decision-1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "OpenRouter's Gemini Decision 1 price (Google AI Studio)",
    "date": "2026-11-11",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.1,
      "cache_write_input_tokens": 0.1,
      "cached_input_tokens": 0.1,
      "input_tokens": 0.1,
      "output_tokens": 0.0
    },
    "source": "https://openrouter.ai/google/gemini-decision-1",
    "text": "$0.10 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "openrouter",
  "questions": 2163,
  "request_seconds": 712.9165812758729,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "26b84078887d801980f85e6ed94f0c8d98e971bc7beb15d5796671ad89f3571a",
  "runner": "benchmark_ollama.py",
  "runner_commit": "7b6fdc11507bb13f2aa97d91cc180214a3031447",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 476393,
    "output_tokens": 0
  }
}
Prompt template
POST https://openrouter.ai/api/v1/systemone
{
 "model": "google/gemini-decision-1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · gemini-decision-1_en_dontknow

Name
Gemini Decision 1 20261111
Setup
Gemini Decision 1 · decision model · with the I don't know option F
Model
google/gemini-decision-1 · provider openrouter
GPU
—
Cost
$0.05478 for the run · provider price × reported usage
Answer time
run gemini-decision-1_en_dontknow · first answer 2026-11-11 19:49:08 UTC · last answer 2026-11-11 20:17:47 UTC · span 1719.0 s, pauses included (not charged) · charged 590.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker (closed weights) · source https://blog.google/innovation-and-ai/technology/developers-tools/gemini-decisions-api/ · read 2026-11-11
Input
text, English
Endpoint
https://openrouter.ai/api
Started
11 Nov 2026 19:49 UTC; last answer 11 Nov 2026 20:17 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Gemini Decision 1 (site run None); runner 7b6fdc115; via OpenRouter's System One route (Google AI Studio serves it; replies name google/gemini-decision-1-20261111); latency includes OpenRouter's time; text only on Logic & Math. Launched 11 Nov 2026; Umit via Can #4966.
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "google/gemini-decision-1",
    "--provider",
    "openrouter",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "b55c22654bc72e59b7a430a92a37731267ddb6de",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "google/gemini-decision-1",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "OpenRouter's Gemini Decision 1 price (Google AI Studio)",
    "date": "2026-11-11",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.1,
      "cache_write_input_tokens": 0.1,
      "cached_input_tokens": 0.1,
      "input_tokens": 0.1,
      "output_tokens": 0.0
    },
    "source": "https://openrouter.ai/google/gemini-decision-1",
    "text": "$0.10 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "openrouter",
  "questions": 2163,
  "request_seconds": 579.9854490382713,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "0b7e60f3b2dcd724182d9765c1d03a2708ff6af939de3bb369b0c117d522172d",
  "runner": "benchmark_ollama.py",
  "runner_commit": "7b6fdc11507bb13f2aa97d91cc180214a3031447",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 547772,
    "output_tokens": 0
  }
}
Prompt template
POST https://openrouter.ai/api/v1/systemone
{
 "model": "google/gemini-decision-1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · onejev1.6-0.8b_en

Name
OneJev1.6-0.8B
Setup
OneJev1.6-0.8B · decision model
Model
OmniJev/OneJev1.6-0.8B · provider onejev
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.002365 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0111 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.002 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run onejev1.6-0.8b_en · first answer 2026-11-11 20:54:31 UTC · last answer 2026-11-11 20:55:11 UTC · span 40.0 s, pauses included (not charged) · charged 40.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0024 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.8 in flight (sum of request latencies / active time)
Model size
Parameters: 0.8B dense · source https://huggingface.co/OmniJev/OneJev1.6-0.8B · read 2026-11-10 · the official model name: OneJev1.6-0.8B (turkce #4263: the maker's stated total is shown); counted from the released files: 0.85B (counted from the checkpoint (852,985,920: language model 752,393,024, vision tower 100,592,896); an average of three Qwen3.5-0.8B fine-tunes; the card's chart: 0.9B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
11 Nov 2026 20:54 UTC; last answer 11 Nov 2026 20:55 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup OneJev1.6-0.8B (site run None); runner 7b6fdc115; OmniJev/OneJev1.6-0.8B@bcd88a382f314b26cb7b1b7f369c1ed3502ee7a0 (Apache-2.0; average of three Qwen3.5-0.8B fine-tunes; no calibration.json shipped, so temperature 1); main moved from c89933c2 to bcd88a38 on 10 Nov (README only: 'Link the Flide paper'); all other files byte-identical, verified 2026-11-11. Behind the card's server, Flide (flide pinned at 0.1.11, the current PyPI release; the card names no version): ~/envs/onejev16/bin/flide serve --model ~/decision-models/OneJev1.6-0.8B@bcd88a382f314b26cb7b1b7f369c1ed3502ee7a0 --name OmniJev/OneJev1.6-0.8B --host 0.0.0.0 --port 19420 (env: glide-family's stack + flide==0.1.11, accelerate, torchvision 0.29.1+cu130); vision on (the server's default); bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider onejev as is (model field = --name; probabilities rounded to 6 places)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "OmniJev/OneJev1.6-0.8B",
    "--provider",
    "onejev",
    "--timeout",
    "30",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "b49f0758f6a5aa13377063fd0c34ff02dd9deaa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "OmniJev/OneJev1.6-0.8B",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "onejev",
  "questions": 2163,
  "request_seconds": 31.670519392442657,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "baf401ec778c29c6413ed0d23cb2b712ddb99b3ddd8f7846b5d594a5371881b0",
  "runner": "benchmark_ollama.py",
  "runner_commit": "7b6fdc11507bb13f2aa97d91cc180214a3031447",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 488047,
    "output_tokens": 2163
  }
}
Prompt template
POST http://spark2:19420/v1/systemone
{
 "model": "OmniJev/OneJev1.6-0.8B",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · onejev1.6-0.8b_en_dontknow

Name
OneJev1.6-0.8B
Setup
OneJev1.6-0.8B · decision model · with the I don't know option F
Model
OmniJev/OneJev1.6-0.8B · provider onejev
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.002365 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0111 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.002 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run onejev1.6-0.8b_en_dontknow · first answer 2026-11-11 20:55:42 UTC · last answer 2026-11-11 20:56:22 UTC · span 40.0 s, pauses included (not charged) · charged 40.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0024 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.8 in flight (sum of request latencies / active time)
Model size
Parameters: 0.8B dense · source https://huggingface.co/OmniJev/OneJev1.6-0.8B · read 2026-11-10 · the official model name: OneJev1.6-0.8B (turkce #4263: the maker's stated total is shown); counted from the released files: 0.85B (counted from the checkpoint (852,985,920: language model 752,393,024, vision tower 100,592,896); an average of three Qwen3.5-0.8B fine-tunes; the card's chart: 0.9B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
11 Nov 2026 20:55 UTC; last answer 11 Nov 2026 20:56 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup OneJev1.6-0.8B (site run None); runner 7b6fdc115; OmniJev/OneJev1.6-0.8B@bcd88a382f314b26cb7b1b7f369c1ed3502ee7a0 (Apache-2.0; average of three Qwen3.5-0.8B fine-tunes; no calibration.json shipped, so temperature 1); main moved from c89933c2 to bcd88a38 on 10 Nov (README only: 'Link the Flide paper'); all other files byte-identical, verified 2026-11-11. Behind the card's server, Flide (flide pinned at 0.1.11, the current PyPI release; the card names no version): ~/envs/onejev16/bin/flide serve --model ~/decision-models/OneJev1.6-0.8B@bcd88a382f314b26cb7b1b7f369c1ed3502ee7a0 --name OmniJev/OneJev1.6-0.8B --host 0.0.0.0 --port 19420 (env: glide-family's stack + flide==0.1.11, accelerate, torchvision 0.29.1+cu130); vision on (the server's default); bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider onejev as is (model field = --name; probabilities rounded to 6 places)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "OmniJev/OneJev1.6-0.8B",
    "--provider",
    "onejev",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "b49f0758f6a5aa13377063fd0c34ff02dd9deaa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "OmniJev/OneJev1.6-0.8B",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "onejev",
  "questions": 2163,
  "request_seconds": 32.34961328122881,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "c4d551365668b59b47e723fcfa3dd2e595a3c4ff06a5bce1740a6a58b6df5c25",
  "runner": "benchmark_ollama.py",
  "runner_commit": "7b6fdc11507bb13f2aa97d91cc180214a3031447",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 557263,
    "output_tokens": 2163
  }
}
Prompt template
POST http://spark2:19420/v1/systemone
{
 "model": "OmniJev/OneJev1.6-0.8B",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · claude-opus-5.5-sub_en_dontknow

Name
Claude Opus 5.5 chat
Setup
Claude Opus 5.5 · chat model · with the I don't know option F
Model
claude-opus-5-5 · provider claude-subscription
GPU
—
Cost
$21.94 for the run, API list-price equivalent · Claude subscription
Answer time
run claude-opus-5.5-sub_en_dontknow · first answer 2026-11-11 21:37:34 UTC · last answer 2026-11-12 01:24:25 UTC · span 13611.0 s, pauses included (not charged) · charged 7418.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker (closed weights) · source https://platform.claude.com/docs/en/about-claude/models · read 2026-11-13
Input
text, English
Endpoint
claude-code-subscription
Started
11 Nov 2026 21:37 UTC; last answer 12 Nov 2026 01:24 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Claude Opus 5.5 on the Claude subscription through the Claude Code CLI (`claude --print`, as turkce's runs 968/971), model claude-opus-5-5, --effort high (the highest of low/medium/high/xhigh/max; adaptive thinking always on); CLAUDE_CODE_MAX_OUTPUT_TOKENS 32768; no temperature sent (Anthropic's default 1.0); Claude Code's agent prompt replaced by a neutral one-line system prompt, tools, settings, MCP servers and session saving off, API-key variables removed; one question per request, one request at a time; cost $0 (Claude subscription); api_equivalent_usd is Claude Code's API-list-price estimate, not a cost. English translations.
Parameters
{
  "effort": "high",
  "max_output_tokens": 32768,
  "route": "claude-code-cli --print",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message."
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · lfm2.5-2.6b_en

Name
LFM2.5-2.6B on · chat
Setup
LFM2.5-2.6B (chat, thinking, on our GPU) · chat model
Model
lfm2.5-2.6b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.6488 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.261 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.649. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-12 01:15:13–02:20:41 UTC
GPU time
run lfm2.5-2.6b_en · first answer 2026-11-12 01:14:34 UTC · last answer 2026-11-12 02:35:40 UTC · span 4866.0 s, pauses included (not charged) · charged 938.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.6488 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 30.0 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 2 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 2.69B dense · source https://huggingface.co/LiquidAI/LFM2.5-2.6B · read 2026-11-12 · the card states 2.69B total parameters (the maker's figure is shown); checkpoint count 2.70B (2,697,198,592 incl. tied embeddings, text only; dense)
Input
text, English
Endpoint
http://127.0.0.1:18613/v1/chat/completions
Started
12 Nov 2026 01:14 UTC; last answer 12 Nov 2026 02:35 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
LFM2.5-2.6B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from LiquidAI/LFM2.5-2.6B@7141b56cd4650af95c7c7b86983443aed050c96a, vLLM reasoning parser qwen3, one RTX PRO 6000 on our GPU host alone; thinking always on (the template opens <think> for every answer; no switch, no levels); sampling temperature 0.1, top_k 50, repetition_penalty 1.1, from generation_config.json; seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 2.70B / 2.70B active / total (dense; checkpoint 2,697,198,592 incl. tied embeddings, text only; card: 2.69B). English translations (DeepSeek + review). No thinking-off setup (bus #6516/#6924/#8187/#8202): the current served chat template exposes no supported switch for a separate thinking-off run; no separate off variant has been run or scored. (Checked 15 Nov in the pinned template: its only option, preserve_thinking, keeps or drops earlier turns' thinking.)
Parameters
{
  "max_tokens": 32768,
  "repetition_penalty": 1.1,
  "seed": 0,
  "temperature": 0.1,
  "top_k": 50
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · grok-4.7-sub_en_dontknow

Name
Grok 4.7 chat
Setup
Grok 4.7 · chat model · with the I don't know option F
Model
grok-4.7 · provider xai-subscription
GPU
—
Cost
$21.28 for the run, API list-price equivalent · xAI subscription
Answer time
run grok-4.7-sub_en_dontknow · first answer 2026-11-12 02:19:27 UTC · last answer 2026-11-12 10:16:24 UTC · span 28617.0 s, pauses included (not charged) · charged 27546.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 2.1 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker (closed weights) · source https://x.ai · read 2026-11-11
Input
text, English
Endpoint
https://api.x.ai/v1/responses
Started
12 Nov 2026 02:19 UTC; last answer 12 Nov 2026 10:16 UTC
Progress
1,987 answered of 2,163 asked; 0 request errors; paused for now at 1,987 of 2,163 (Ümit #8603, zeki #8637, 2026-11-15T11:37Z)
Notes
Grok 4.7 on the SuperGrok subscription through xAI's Responses API with OMP's xai-oauth login (as turkce's ask_xai and jevgammon's xai-subscription route), model grok-4.7, reasoning effort high (Reading's setting); max_output_tokens 32768; no temperature sent (xAI's default) and no tools; a neutral one-line system prompt; one question per request, one request at a time; a usage-limit reply stops the run without retrying it (resumable); cost $0 (xAI subscription); api_equivalent_usd is the tokens at xAI's API list price {"input": 2.0, "cached_input": 0.5, "output": 6.0} per 1M tokens, not a cost. English translations. PAUSED (Umit #8603, zeki #8637, 2026-11-15T11:37Z): skipped for now, stopped at 1987 answers; large models are left for now (in-flight requests at the stop were not stored); resumable from the stored answers at the same settings if asked.
Parameters
{
  "effort": "high",
  "list_price_per_mtok": {
    "cached_input": 0.5,
    "input": 2.0,
    "output": 6.0
  },
  "max_output_tokens": 32768,
  "route": "xai responses api (OMP xai-oauth)",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message.",
  "tools": "none"
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · lfm2.5-2.6b_en_dontknow

Name
LFM2.5-2.6B on · chat
Setup
LFM2.5-2.6B (chat, thinking, on our GPU) · chat model · with the I don't know option F
Model
lfm2.5-2.6b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.6197 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.249 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.620. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run lfm2.5-2.6b_en_dontknow · first answer 2026-11-12 03:40:09 UTC · last answer 2026-11-12 03:55:05 UTC · span 896.0 s, pauses included (not charged) · charged 896.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.6197 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.2 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 2 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 2.69B dense · source https://huggingface.co/LiquidAI/LFM2.5-2.6B · read 2026-11-12 · the card states 2.69B total parameters (the maker's figure is shown); checkpoint count 2.70B (2,697,198,592 incl. tied embeddings, text only; dense)
Input
text, English
Endpoint
http://127.0.0.1:18613/v1/chat/completions
Started
12 Nov 2026 03:40 UTC; last answer 12 Nov 2026 03:55 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
LFM2.5-2.6B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from LiquidAI/LFM2.5-2.6B@7141b56cd4650af95c7c7b86983443aed050c96a, vLLM reasoning parser qwen3, one RTX PRO 6000 on our GPU host alone; thinking always on (the template opens <think> for every answer; no switch, no levels); sampling temperature 0.1, top_k 50, repetition_penalty 1.1, from generation_config.json; seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 2.70B / 2.70B active / total (dense; checkpoint 2,697,198,592 incl. tied embeddings, text only; card: 2.69B). English translations (DeepSeek + review). No thinking-off setup (bus #6516/#6924/#8187/#8202): the current served chat template exposes no supported switch for a separate thinking-off run; no separate off variant has been run or scored. (Checked 15 Nov in the pinned template: its only option, preserve_thinking, keeps or drops earlier turns' thinking.)
Parameters
{
  "max_tokens": 32768,
  "repetition_penalty": 1.1,
  "seed": 0,
  "temperature": 0.1,
  "top_k": 50
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · gliclass-v4-edge_en

Name
GLiClass v4 Edge
Setup
GLiClass v4 Edge · decision model
Model
knowledgator/gliclass-v4-edge · provider gliclass
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.001833 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.00861 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.002 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gliclass-v4-edge_en · first answer 2026-11-12 08:32:59 UTC · last answer 2026-11-12 08:33:30 UTC · span 31.0 s, pauses included (not charged) · charged 31.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0018 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 0.4B dense · source https://huggingface.co/knowledgator/gliclass-v4-edge · read 2026-11-11 · the card states a 0.4B-parameter encoder (435M in its comparison table); checkpoint count 0.44B (counted from the checkpoint (439,790,596: DeBERTa-v3-large encoder 433,913,856, text and class projectors 4,198,400, scorer 557,569, calibrator 1,120,770); dense, text only; the card: 0.4B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
12 Nov 2026 08:32 UTC; last answer 12 Nov 2026 08:33 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup GLiClass v4 Edge (site run None); runner df9ae2734; knowledgator/gliclass-v4-edge@885af1af8a07edf4ed34df83ca0d995ed8201283 (Apache-2.0; a 0.44B GLiClass uni-encoder on DeBERTa-v3-large with a calibrator; text only) behind turkce's runner/servers/gliclass_server.py (sha256 aa39e5f4…, copied unchanged to ~/decision-src/gliclass_server.py on a shared Spark host), which applies the card's own System One mapping (its jev_api.py: one gliclass pass per choice question, the state as text, the instructions as the prompt, each option's description as a label; probabilities the softmax over the calibrated label logits) with gliclass 0.1.22 in BF16 and refuses an input over 16,384 tokens with 422, never cut: HF_HUB_OFFLINE=1 ~/envs/gliclass/bin/python ~/decision-src/gliclass_server.py --model ~/decision-models/gliclass-v4-edge@885af1af8a07edf4ed34df83ca0d995ed8201283 --device cuda:0 --host 0.0.0.0 --port 19480 (env ~/envs/gliclass: gliclass 0.1.22 (the card's pip install gliclass -U on 11 November 2026), torch 2.14.0+cu130, transformers 5.22.2); the server has no health route (health null: check with a request); model field absent or knowledgator/gliclass-v4-edge; vram_mib a weights-plus-activations estimate; bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'gliclass' (tb-next)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "knowledgator/gliclass-v4-edge",
    "--provider",
    "gliclass",
    "--timeout",
    "60",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "6ad0eac51a4abb950bde74b24687209aec628aa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "knowledgator/gliclass-v4-edge",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "gliclass",
  "questions": 2163,
  "request_seconds": 26.535286557336804,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "004c98d0e87bd776ada3928bea60d76c4f5c07983deb8aa6261fa2baf4b1b019",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 271040,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19480/v1/systemone
{
 "model": "knowledgator/gliclass-v4-edge",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · gliclass-v4-edge_en_dontknow

Name
GLiClass v4 Edge
Setup
GLiClass v4 Edge · decision model · with the I don't know option F
Model
knowledgator/gliclass-v4-edge · provider gliclass
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.001833 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.00861 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.002 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gliclass-v4-edge_en_dontknow · first answer 2026-11-12 08:34:03 UTC · last answer 2026-11-12 08:34:34 UTC · span 31.0 s, pauses included (not charged) · charged 31.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0018 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 0.4B dense · source https://huggingface.co/knowledgator/gliclass-v4-edge · read 2026-11-11 · the card states a 0.4B-parameter encoder (435M in its comparison table); checkpoint count 0.44B (counted from the checkpoint (439,790,596: DeBERTa-v3-large encoder 433,913,856, text and class projectors 4,198,400, scorer 557,569, calibrator 1,120,770); dense, text only; the card: 0.4B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
12 Nov 2026 08:34 UTC; last answer 12 Nov 2026 08:34 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup GLiClass v4 Edge (site run None); runner df9ae2734; knowledgator/gliclass-v4-edge@885af1af8a07edf4ed34df83ca0d995ed8201283 (Apache-2.0; a 0.44B GLiClass uni-encoder on DeBERTa-v3-large with a calibrator; text only) behind turkce's runner/servers/gliclass_server.py (sha256 aa39e5f4…, copied unchanged to ~/decision-src/gliclass_server.py on a shared Spark host), which applies the card's own System One mapping (its jev_api.py: one gliclass pass per choice question, the state as text, the instructions as the prompt, each option's description as a label; probabilities the softmax over the calibrated label logits) with gliclass 0.1.22 in BF16 and refuses an input over 16,384 tokens with 422, never cut: HF_HUB_OFFLINE=1 ~/envs/gliclass/bin/python ~/decision-src/gliclass_server.py --model ~/decision-models/gliclass-v4-edge@885af1af8a07edf4ed34df83ca0d995ed8201283 --device cuda:0 --host 0.0.0.0 --port 19480 (env ~/envs/gliclass: gliclass 0.1.22 (the card's pip install gliclass -U on 11 November 2026), torch 2.14.0+cu130, transformers 5.22.2); the server has no health route (health null: check with a request); model field absent or knowledgator/gliclass-v4-edge; vram_mib a weights-plus-activations estimate; bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'gliclass' (tb-next)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "knowledgator/gliclass-v4-edge",
    "--provider",
    "gliclass",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "6ad0eac51a4abb950bde74b24687209aec628aa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "knowledgator/gliclass-v4-edge",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "gliclass",
  "questions": 2163,
  "request_seconds": 27.11270485102432,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "60774792e3a629270a67d0254fbaf6ee826ddcf15dd2ecb0b0e7538df18d93e9",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 331604,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19480/v1/systemone
{
 "model": "knowledgator/gliclass-v4-edge",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · gemma-4-31b-it_en_t0_dontknow

Name
Gemma 4 31B chat
Setup
Gemma 4 31B (chat, on our GPU) · chat model · with the I don't know option F
Model
gemma-4-31B-it · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.109 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.446 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.109. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gemma-4-31b-it_en_t0_dontknow · first answer 2026-11-12 08:58:01 UTC · last answer 2026-11-12 09:24:45 UTC · span 1604.0 s, pauses included (not charged) · charged 1604.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.1094 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 16.0 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 2 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 30.7B dense · source https://huggingface.co/google/gemma-4-31B-it · read 2026-11-10 · the card states Total Parameters 30.7B (plus a vision encoder of about 550M) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 31.27B (counted from the checkpoint (31,273,088,876, the vision encoder included); the card gives 30.7B plus a vision encoder of about 550M)
Input
text, English
Endpoint
http://127.0.0.1:8021/v1/chat/completions
Started
12 Nov 2026 08:57 UTC; last answer 12 Nov 2026 09:24 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Gemma 4 31B-it FP8 weights + FP8 KV (vLLM v0.30.0, GPU 0 of our GPU host), thinking off (server default), temperature 0, English translations (DeepSeek + review); dont_know run (sixth option F "I don't know"), otherwise the same server and request settings as gemma-4-31b-it_en_t0
Parameters
{
  "max_tokens": 8192,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · minicpm5-2b_en

Name
MiniCPM5-2B on · chat
Setup
MiniCPM5-2B (chat, thinking, on our GPU) · chat model
Model
minicpm5-2b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.8978 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.361 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.898. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-12 10:06:31–11:33:26 UTC
GPU time
run minicpm5-2b_en · first answer 2026-11-12 10:05:28 UTC · last answer 2026-11-12 11:54:01 UTC · span 6513.0 s, pauses included (not charged) · charged 1298.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.8978 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 29.4 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 2 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 2.52B dense · from the run's notes (dense; checkpoint 2,516,756,480 incl. untied lm_head, text only; card: 2,516,756,480); also https://huggingface.co/openbmb/MiniCPM5-2B · read 2026-11-12 · counted from the checkpoint (2,516,756,480 incl. untied lm_head, text only; dense); the card: 2,516,756,480
Input
text, English
Endpoint
http://127.0.0.1:18614/v1/chat/completions
Started
12 Nov 2026 10:05 UTC; last answer 12 Nov 2026 11:54 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
MiniCPM5-2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from openbmb/MiniCPM5-2B@42433e90dad95123ab87729fd7011e7110629556, vLLM reasoning parser qwen3, one RTX PRO 6000 on our GPU host alone; thinking on (enable_thinking true; on/off only, no levels); sampling temperature 1, top_p 0.95, from generation_config.json; seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 2.52B / 2.52B active / total (dense; checkpoint 2,516,756,480 incl. untied lm_head, text only; card: 2,516,756,480). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "seed": 0,
  "temperature": 1.0,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · qwen3.8-27b-bf16_en_low_dontknow

Name
Qwen3.8-27B low · chat
Setup
Qwen3.8-27B (chat, reasoning low, on our GPU) · chat model · with the I don't know option F
Model
qwen38-27b-bf16 · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.421 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.571 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.421. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run qwen3.8-27b-bf16_en_low_dontknow · first answer 2026-11-12 10:44:24 UTC · last answer 2026-11-12 11:18:38 UTC · span 2054.0 s, pauses included (not charged) · charged 2054.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.4207 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 12.1 in flight (sum of request latencies / active time)
Output limit
60,000 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 27B dense · source https://huggingface.co/Qwen/Qwen3.8-27B · read 2026-11-10 · the card states Number of Parameters: 27B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 27.78B (counted from the checkpoint (27,781,427,952, the vision encoder included); the card says 27B)
Input
text, English
Endpoint
http://127.0.0.1:8022/v1/chat/completions
Started
12 Nov 2026 10:44 UTC; last answer 12 Nov 2026 11:18 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Qwen3.8-27B BF16 (vLLM v0.30.0, MTP, GPU 1 of our GPU host), thinking on at reasoning_effort low, model generation config sampling (1.0/0.95/20, seed 0), JSON answer schema; English translations (DeepSeek + review); dont_know run in the line answer format: the standard run used the JSON answer schema (A–E only), which the runner doesn't define for option F; otherwise the same server and request settings
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "reasoning_effort": "low"
  },
  "max_tokens": 60000,
  "seed": 0,
  "temperature": 1.0,
  "top_k": 20,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · minicpm5-2b_en_dontknow

Name
MiniCPM5-2B on · chat
Setup
MiniCPM5-2B (chat, thinking, on our GPU) · chat model · with the I don't know option F
Model
minicpm5-2b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.6958 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.279 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.696. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run minicpm5-2b_en_dontknow · first answer 2026-11-12 13:05:41 UTC · last answer 2026-11-12 13:22:27 UTC · span 1006.0 s, pauses included (not charged) · charged 1006.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.6958 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 30.9 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 1 answer cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 2.52B dense · from the run's notes (dense; checkpoint 2,516,756,480 incl. untied lm_head, text only; card: 2,516,756,480); also https://huggingface.co/openbmb/MiniCPM5-2B · read 2026-11-12 · counted from the checkpoint (2,516,756,480 incl. untied lm_head, text only; dense); the card: 2,516,756,480
Input
text, English
Endpoint
http://127.0.0.1:18614/v1/chat/completions
Started
12 Nov 2026 13:05 UTC; last answer 12 Nov 2026 13:22 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
MiniCPM5-2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from openbmb/MiniCPM5-2B@42433e90dad95123ab87729fd7011e7110629556, vLLM reasoning parser qwen3, one RTX PRO 6000 on our GPU host alone; thinking on (enable_thinking true; on/off only, no levels); sampling temperature 1, top_p 0.95, from generation_config.json; seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 2.52B / 2.52B active / total (dense; checkpoint 2,516,756,480 incl. untied lm_head, text only; card: 2,516,756,480). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "seed": 0,
  "temperature": 1.0,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · claude-sonnet-5.5-sub_en

Name
Claude Sonnet 5.5 chat
Setup
Claude Sonnet 5.5 · chat model
Model
claude-sonnet-5-5 · provider claude-subscription
GPU
—
Cost
$10.24 for the run, API list-price equivalent · Claude subscription
Answer time
run claude-sonnet-5.5-sub_en · first answer 2026-11-12 16:03:58 UTC · last answer 2026-11-12 18:28:13 UTC · span 8655.0 s, pauses included (not charged) · charged 8655.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker (closed weights) · source https://platform.claude.com/docs/en/about-claude/models · read 2026-11-13
Input
text, English
Endpoint
claude-code-subscription
Started
12 Nov 2026 16:03 UTC; last answer 12 Nov 2026 18:28 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Claude Sonnet 5.5 on the Claude subscription through the Claude Code CLI (`claude --print`, as turkce's runs 968/971), model claude-sonnet-5-5, --effort high (the highest of low/medium/high/xhigh/max; adaptive thinking always on); CLAUDE_CODE_MAX_OUTPUT_TOKENS 32768; no temperature sent (Anthropic's default 1.0); Claude Code's agent prompt replaced by a neutral one-line system prompt, tools, settings, MCP servers and session saving off, API-key variables removed; one question per request, one request at a time; cost $0 (Claude subscription); api_equivalent_usd is Claude Code's API-list-price estimate, not a cost. English translations.
Parameters
{
  "effort": "high",
  "max_output_tokens": 32768,
  "route": "claude-code-cli --print",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message."
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · lfm2.5-24b-a2b_en

Name
LFM2.5-24B-A2B on · chat
Setup
LFM2.5-24B-A2B (chat, thinking, on our GPU) · chat model
Model
lfm2.5-24b-a2b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$2.031 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.816 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $2.031. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-12 16:24:07–19:41:31 UTC
GPU time
run lfm2.5-24b-a2b_en · first answer 2026-11-12 16:23:11 UTC · last answer 2026-11-12 20:29:32 UTC · span 14781.0 s, pauses included (not charged) · charged 2937.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $2.0314 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 29.6 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 5 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 24.0B MoE · 2.45B active · source https://huggingface.co/LiquidAI/LFM2.5-24B-A2B · read 2026-11-12 · the card states 24.0B total / 2.45B active (the maker's figure is shown); checkpoint count 23.97B / 2.45B active (MoE, 4 of 64 experts; checkpoint 23,971,587,712, active 2,454,808,192 incl. tied embeddings)
Input
text, English
Endpoint
http://127.0.0.1:18618/v1/chat/completions
Started
12 Nov 2026 16:23 UTC; last answer 12 Nov 2026 20:29 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
LFM2.5-24B-A2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from LiquidAI/LFM2.5-24B-A2B@1769516ac588c4c558089c94f27d96b0dc67b183, vLLM reasoning parser qwen3 (the revision backgammon served (jevgammon tools/v3/servers/lfm25-24b.sh), not today's main 3dc4cd09; parser qwen3 as our other LFM2.5 runs, where backgammon's server used deepseek_r1), one RTX PRO 6000 on our GPU host alone; thinking always on (the template opens <think> for every answer; no switch, no levels); sampling temperature 0.1, top_k 50, repetition_penalty 1.05, from generation_config.json; seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 2.45B / 23.97B active / total (MoE, 4 of 64 experts; checkpoint 23,971,587,712, active 2,454,808,192 incl. tied embeddings; card: 24.0B total, 2.45B active). English translations (DeepSeek + review). No thinking-off setup (bus #6516/#6924/#8187/#8202): the current served chat template exposes no supported switch for a separate thinking-off run; no separate off variant has been run or scored. (Checked 15 Nov in the pinned template: its only option, preserve_thinking, keeps or drops earlier turns' thinking.)
Parameters
{
  "max_tokens": 32768,
  "repetition_penalty": 1.05,
  "seed": 0,
  "temperature": 0.1,
  "top_k": 50
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · knowline-2b_en

Name
KnowLine-2B-Gen1
Setup
KnowLine-2B-Gen1 · decision model
Model
PelaAI/KnowLine-2B-Gen1 · provider knowline
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.006917 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0325 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.007 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run knowline-2b_en · first answer 2026-11-12 16:35:10 UTC · last answer 2026-11-12 16:37:07 UTC · span 117.0 s, pauses included (not charged) · charged 117.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0069 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 2B dense · source https://huggingface.co/PelaAI/KnowLine-2B-Gen1 · read 2026-11-11 · the card states a 2B decision model; checkpoint count 2.27B (counted from the checkpoint (2,274,069,824: language model 1,881,825,088, MTP layer 60,828,160 (model-extra-from-base.safetensors), vision tower 331,416,576); a LoRA fine-tune of Qwen3.5-2B merged into the weights; dense; the card: 2B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
12 Nov 2026 16:35 UTC; last answer 12 Nov 2026 16:37 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup KnowLine-2B-Gen1 (site run None); runner df9ae2734; PelaAI/KnowLine-2B-Gen1@53245d470a4ce316e32fd0d18c24e142c697f40b (Apache-2.0 weights and code; one-epoch LoRA SFT of Qwen3.5-2B merged into BF16, readout temperature 1.32 folded into the weights, score questions divided by temperatures.json) behind the card's serve_knowline.sh: SGLang 0.5.21 in BF16 (--enable-fp32-lm-head --mamba-radix-cache-strategy extra_buffer, MEM default 0.85) on port+1000 (20490) and knowline_server.py (read twice up to 3,000 tokens): cd ~/decision-models/KnowLine-2B-Gen1@53245d470a4ce316e32fd0d18c24e142c697f40b && PATH=~/envs/knowline-sglang/bin:$PATH HOST=0.0.0.0 bash serve_knowline.sh ~/decision-models/KnowLine-2B-Gen1@53245d470a4ce316e32fd0d18c24e142c697f40b 0 19490, env ~/envs/knowline-sglang (sglang 0.5.21, torch 2.13.0+cu130, transformers 5.12.1); vram_mib is MEM 0.85 of the unified memory (SGLang sizes its KV cache to it), so it cannot run beside another large server; vision on (the server's default); bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'knowline'
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "PelaAI/KnowLine-2B-Gen1",
    "--provider",
    "knowline",
    "--timeout",
    "60",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "6ad0eac51a4abb950bde74b24687209aec628aa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "PelaAI/KnowLine-2B-Gen1",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "knowline",
  "questions": 2163,
  "request_seconds": 108.46224289055681,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "8492686409a788ab2646d77e9349fcc7e2ece1ae75be6c7305ceb088185753c6",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 924182,
    "output_tokens": 2163
  }
}
Prompt template
POST http://spark2:19490/v1/systemone
{
 "model": "PelaAI/KnowLine-2B-Gen1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · knowline-2b_en_dontknow

Name
KnowLine-2B-Gen1
Setup
KnowLine-2B-Gen1 · decision model · with the I don't know option F
Model
PelaAI/KnowLine-2B-Gen1 · provider knowline
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.007153 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0336 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.007 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run knowline-2b_en_dontknow · first answer 2026-11-12 16:38:29 UTC · last answer 2026-11-12 16:40:30 UTC · span 121.0 s, pauses included (not charged) · charged 121.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0072 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 2B dense · source https://huggingface.co/PelaAI/KnowLine-2B-Gen1 · read 2026-11-11 · the card states a 2B decision model; checkpoint count 2.27B (counted from the checkpoint (2,274,069,824: language model 1,881,825,088, MTP layer 60,828,160 (model-extra-from-base.safetensors), vision tower 331,416,576); a LoRA fine-tune of Qwen3.5-2B merged into the weights; dense; the card: 2B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
12 Nov 2026 16:38 UTC; last answer 12 Nov 2026 16:40 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup KnowLine-2B-Gen1 (site run None); runner df9ae2734; PelaAI/KnowLine-2B-Gen1@53245d470a4ce316e32fd0d18c24e142c697f40b (Apache-2.0 weights and code; one-epoch LoRA SFT of Qwen3.5-2B merged into BF16, readout temperature 1.32 folded into the weights, score questions divided by temperatures.json) behind the card's serve_knowline.sh: SGLang 0.5.21 in BF16 (--enable-fp32-lm-head --mamba-radix-cache-strategy extra_buffer, MEM default 0.85) on port+1000 (20490) and knowline_server.py (read twice up to 3,000 tokens): cd ~/decision-models/KnowLine-2B-Gen1@53245d470a4ce316e32fd0d18c24e142c697f40b && PATH=~/envs/knowline-sglang/bin:$PATH HOST=0.0.0.0 bash serve_knowline.sh ~/decision-models/KnowLine-2B-Gen1@53245d470a4ce316e32fd0d18c24e142c697f40b 0 19490, env ~/envs/knowline-sglang (sglang 0.5.21, torch 2.13.0+cu130, transformers 5.12.1); vram_mib is MEM 0.85 of the unified memory (SGLang sizes its KV cache to it), so it cannot run beside another large server; vision on (the server's default); bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'knowline'
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "PelaAI/KnowLine-2B-Gen1",
    "--provider",
    "knowline",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "6ad0eac51a4abb950bde74b24687209aec628aa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "PelaAI/KnowLine-2B-Gen1",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "knowline",
  "questions": 2163,
  "request_seconds": 114.51774726813892,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "f45fef41ea38c7fc24b0b5ea6b18e75e9be054c45c6f979a4f6cc4eec5c35c5e",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 1053962,
    "output_tokens": 2163
  }
}
Prompt template
POST http://spark2:19490/v1/systemone
{
 "model": "PelaAI/KnowLine-2B-Gen1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · sieve-31b_en

Name
Sieve 31B v3
Setup
Sieve 31B v3 · decision model
Model
sthanika-ai/Sieve-31B-v3 · provider sieve
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.08465 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.398 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.085 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run sieve-31b_en · first answer 2026-11-12 17:48:09 UTC · last answer 2026-11-12 18:12:01 UTC · span 1432.0 s, pauses included (not charged) · charged 1432.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0847 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 31B dense · source https://huggingface.co/sthanika-ai/Sieve-31B-v3 · read 2026-11-10 · the official model name: Sieve 31B v3 (turkce #4263: the maker's stated total is shown); counted from the released files: 31.27B (counted from the checkpoint (31,273,088,876: language model 30,697,345,340, vision tower 575,743,536); a full fine-tune of Gemma 4 31B IT; the card: 29.29B trained parameters (the decoder projections))
Input
text, English
Endpoint
internal address on a shared Spark host
Started
12 Nov 2026 17:48 UTC; last answer 12 Nov 2026 18:12 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Sieve 31B v3 (site run None); runner df9ae2734; sthanika-ai/Sieve-31B-v3@8fcee858f7f20a50bfa3c8f984c22adb3cbb46a2 (Apache-2.0; Gemma 4 31B IT full fine-tune, letters readout from head.pt, each question read in 2 option orders) behind the card's sieve-serve (github.com/sthanika-ai/Sieve main at 1e64657, v0.5.1; the card asks for sieve >= 0.4.0): ~/envs/sieve/bin/sieve-serve --model ~/decision-models/Sieve-31B-v3@8fcee858f7f20a50bfa3c8f984c22adb3cbb46a2 --host 0.0.0.0 --port 19370 (env ~/envs/sieve: the repository's requirements.txt exactly (transformers 5.17.0, peft 0.21.0, accelerate 1.15.0, fastapi 0.141.1, uvicorn 0.53.0, pydantic 2.13.5, ...) except torch 2.14.0+cu130 instead of the tested 2.8.0, plus sieve-decisions 0.5.1 from the checkout at 1e64657); text only; bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'sieve' (System One body at /v1/decide, no model field; probabilities rounded to 4 places, no usage.input_tokens)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "sthanika-ai/Sieve-31B-v3",
    "--provider",
    "sieve",
    "--timeout",
    "120",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "6ad0eac51a4abb950bde74b24687209aec628aa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "sthanika-ai/Sieve-31B-v3",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "sieve",
  "questions": 2163,
  "request_seconds": 1418.8521080814535,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "ceaaac7d13f18cabdd4d24750de7330137cf23b08a96e368746f5012088e1952",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 0,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19370/v1/decide
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · claude-sonnet-5.5-sub_en_dontknow

Name
Claude Sonnet 5.5 chat
Setup
Claude Sonnet 5.5 · chat model · with the I don't know option F
Model
claude-sonnet-5-5 · provider claude-subscription
GPU
—
Cost
$10.49 for the run, API list-price equivalent · Claude subscription
Answer time
run claude-sonnet-5.5-sub_en_dontknow · first answer 2026-11-12 18:28:17 UTC · last answer 2026-11-12 20:50:01 UTC · span 8504.0 s, pauses included (not charged) · charged 8504.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker (closed weights) · source https://platform.claude.com/docs/en/about-claude/models · read 2026-11-13
Input
text, English
Endpoint
claude-code-subscription
Started
12 Nov 2026 18:28 UTC; last answer 12 Nov 2026 20:50 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Claude Sonnet 5.5 on the Claude subscription through the Claude Code CLI (`claude --print`, as turkce's runs 968/971), model claude-sonnet-5-5, --effort high (the highest of low/medium/high/xhigh/max; adaptive thinking always on); CLAUDE_CODE_MAX_OUTPUT_TOKENS 32768; no temperature sent (Anthropic's default 1.0); Claude Code's agent prompt replaced by a neutral one-line system prompt, tools, settings, MCP servers and session saving off, API-key variables removed; one question per request, one request at a time; cost $0 (Claude subscription); api_equivalent_usd is Claude Code's API-list-price estimate, not a cost. English translations.
Parameters
{
  "effort": "high",
  "max_output_tokens": 32768,
  "route": "claude-code-cli --print",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message."
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · sieve-31b_en_dontknow

Name
Sieve 31B v3
Setup
Sieve 31B v3 · decision model · with the I don't know option F
Model
sthanika-ai/Sieve-31B-v3 · provider sieve
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.09725 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.457 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.097 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run sieve-31b_en_dontknow · first answer 2026-11-12 18:35:17 UTC · last answer 2026-11-12 19:02:42 UTC · span 1645.0 s, pauses included (not charged) · charged 1645.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0972 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 31B dense · source https://huggingface.co/sthanika-ai/Sieve-31B-v3 · read 2026-11-10 · the official model name: Sieve 31B v3 (turkce #4263: the maker's stated total is shown); counted from the released files: 31.27B (counted from the checkpoint (31,273,088,876: language model 30,697,345,340, vision tower 575,743,536); a full fine-tune of Gemma 4 31B IT; the card: 29.29B trained parameters (the decoder projections))
Input
text, English
Endpoint
internal address on a shared Spark host
Started
12 Nov 2026 18:35 UTC; last answer 12 Nov 2026 19:02 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Sieve 31B v3 (site run None); runner df9ae2734; sthanika-ai/Sieve-31B-v3@8fcee858f7f20a50bfa3c8f984c22adb3cbb46a2 (Apache-2.0; Gemma 4 31B IT full fine-tune, letters readout from head.pt, each question read in 2 option orders) behind the card's sieve-serve (github.com/sthanika-ai/Sieve main at 1e64657, v0.5.1; the card asks for sieve >= 0.4.0): ~/envs/sieve/bin/sieve-serve --model ~/decision-models/Sieve-31B-v3@8fcee858f7f20a50bfa3c8f984c22adb3cbb46a2 --host 0.0.0.0 --port 19370 (env ~/envs/sieve: the repository's requirements.txt exactly (transformers 5.17.0, peft 0.21.0, accelerate 1.15.0, fastapi 0.141.1, uvicorn 0.53.0, pydantic 2.13.5, ...) except torch 2.14.0+cu130 instead of the tested 2.8.0, plus sieve-decisions 0.5.1 from the checkout at 1e64657); text only; bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'sieve' (System One body at /v1/decide, no model field; probabilities rounded to 4 places, no usage.input_tokens)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "sthanika-ai/Sieve-31B-v3",
    "--provider",
    "sieve",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "6ad0eac51a4abb950bde74b24687209aec628aa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "sthanika-ai/Sieve-31B-v3",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "sieve",
  "questions": 2163,
  "request_seconds": 1630.2771369977854,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "7db2afc7ae0c4d22a09defbfb4e176d0bb9fbfd7f26f5d730fae6fa1121681db",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 0,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19370/v1/decide
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · lfm3-1.7b_en_dontknow

Name
LFM3-1.7B on · chat
Setup
LFM3-1.7B (chat, thinking, on our GPU) · chat model · with the I don't know option F
Model
lfm3-1.7b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.3306 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.133 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.331. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run lfm3-1.7b_en_dontknow · first answer 2026-11-12 18:41:09 UTC · last answer 2026-11-12 18:49:07 UTC · span 478.0 s, pauses included (not charged) · charged 478.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.3306 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 27.1 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 2 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 1.73B dense · from the run's notes (dense; BF16 safetensors 1,728,801,536, text only; the GGUF of the llama.cpp answers: 1,728,801,600 incl. a 64-value rope_freqs table; card: 1.7B); also https://huggingface.co/LiquidAI/LFM3-1.7B-GGUF · read 2026-11-09 · the released checkpoint's tensors without registered buffers: 1,728,801,536 (the served BF16 GGUF also stores a 64-element RoPE buffer, 1,728,801,600); text only; card: 1.7B
Input
text, English
Endpoint
http://127.0.0.1:18605/v1/chat/completions
Started
12 Nov 2026 18:41 UTC; last answer 12 Nov 2026 18:49 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
LFM3-1.7B, served with vLLM 0.34.0rc1.dev302+gbe47189f3 (nightly, commit be47189f37f6, CUDA 13.5 image; docker vllm/vllm-openai@sha256:7f0618da95bac2f1cc97fcd1791380d7c1a06d24a4998da45aeb7fbe6c49179e), BF16 from LiquidAI/LFM3-1.7B@0dc4268b561499c648c7ba7c9466bf3d8d01517a, vLLM reasoning parser qwen3 (vLLM nightly because no vLLM release has Lfm3ForCausalLM; the same weights as the llama.cpp setup's official BF16 GGUF LiquidAI/LFM3-1.7B-GGUF@cd0c74c5 (tensors byte-identical), the same chat template and prompt token ids; the dont_know runs ran entirely on this engine, the standard runs lfm3-1.7b_en began on llama.cpp b12020 and switched part-way (ENGINE BOUNDARY in their notes); qualification work/lfm3_vllm_qual.md), one RTX PRO 6000 on our GPU host alone; thinking on (enable_thinking true, the template default; on/off only); sampling temperature 0.1, top_k 50, repetition_penalty 1.1, top_p 1, min_p 0, from generation_config.json, passed 1:1 as vLLM sampling params (repetition_penalty over prompt and output, as the llama.cpp setup's repeat_penalty over the whole context); seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 1.73B / 1.73B active / total (dense; BF16 safetensors 1,728,801,536, text only; the GGUF of the llama.cpp answers: 1,728,801,600 incl. a 64-value rope_freqs table; card: 1.7B). English translations (DeepSeek + review). All answers: 4 llama-server instances × 8 slots (same build, GGUF, flags, sampling, per-slot context 69,632, 32 requests in flight as in lfm3-1.7b_en) behind a forwarder on the same port, for throughput only (zeki #5247/#5265).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "repetition_penalty": 1.1,
  "seed": 0,
  "temperature": 0.1,
  "top_k": 50,
  "top_p": 1.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · gemma-4-e2b-it-think_en

Name
Gemma 4 E2B on · chat
Setup
Gemma 4 E2B (chat, thinking on, on our GPU) · chat model
Model
gemma-4-e2b-it-think · provider vllm
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.2986 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 1.4 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.299 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-12 20:58:50–22:48:49 UTC
GPU time
run gemma-4-e2b-it-think_en · first answer 2026-11-12 20:55:58 UTC · last answer 2026-11-13 00:10:08 UTC · span 11650.0 s, pauses included (not charged) · charged 5051.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.2986 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 31.0 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 16 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 5.1B dense · 2.3B effective · source https://huggingface.co/google/gemma-4-E2B-it · read 2026-11-10 · the card states 2.3B effective (5.1B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 5.12B (counted from the checkpoint (5,123,178,979: language model 4.65B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 5.1B with embeddings, 2.3B effective; not a mixture of experts)
Input
text, English
Endpoint
http://lan-host:18624/v1/chat/completions
Started
12 Nov 2026 20:55 UTC; last answer 13 Nov 2026 00:10 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Gemma 4 E2B, served with vLLM v0.30.0 (docker vllm/vllm-openai@sha256:8a69ffad015f138d7170c4ddc429e230a3bc1c1719f67e14324749df200a4b90, the v0.30.0 release's index digest, linux/arm64 build sha256:4864d46625cb…; vLLM commit ced6857afa0e), BF16 from google/gemma-4-E2B-it@4adf15748de9e1a0097b6274cee9c15836c41c37, vLLM reasoning parser gemma4, one DGX Spark (a shared Spark host, NVIDIA GB10, 121 GB unified memory) alone; thinking on (chat_template_kwargs enable_thinking true, the template's own switch: it opens a system turn with <|think|>; the template default is off; on/off only, no levels); sampling temperature 0, as the Gemma 4 31B chat run; no seed (as the Gemma 4 31B run); max_tokens 8,192; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 5.12B / 5.12B active / total (dense; checkpoint 5,123,178,979 incl. vision 0.17B and audio 0.31B towers; card: 5.1B with per-layer embeddings, 2.3B effective). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 8192,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · lfm2.5-24b-a2b_en_dontknow

Name
LFM2.5-24B-A2B on · chat
Setup
LFM2.5-24B-A2B (chat, thinking, on our GPU) · chat model · with the I don't know option F
Model
lfm2.5-24b-a2b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.881 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.756 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.881. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run lfm2.5-24b-a2b_en_dontknow · first answer 2026-11-12 23:38:57 UTC · last answer 2026-11-13 00:24:17 UTC · span 2720.0 s, pauses included (not charged) · charged 2720.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.8813 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.2 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 2 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 24.0B MoE · 2.45B active · source https://huggingface.co/LiquidAI/LFM2.5-24B-A2B · read 2026-11-12 · the card states 24.0B total / 2.45B active (the maker's figure is shown); checkpoint count 23.97B / 2.45B active (MoE, 4 of 64 experts; checkpoint 23,971,587,712, active 2,454,808,192 incl. tied embeddings)
Input
text, English
Endpoint
http://127.0.0.1:18618/v1/chat/completions
Started
12 Nov 2026 23:38 UTC; last answer 13 Nov 2026 00:24 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
LFM2.5-24B-A2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from LiquidAI/LFM2.5-24B-A2B@1769516ac588c4c558089c94f27d96b0dc67b183, vLLM reasoning parser qwen3 (the revision backgammon served (jevgammon tools/v3/servers/lfm25-24b.sh), not today's main 3dc4cd09; parser qwen3 as our other LFM2.5 runs, where backgammon's server used deepseek_r1), one RTX PRO 6000 on our GPU host alone; thinking always on (the template opens <think> for every answer; no switch, no levels); sampling temperature 0.1, top_k 50, repetition_penalty 1.05, from generation_config.json; seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 2.45B / 23.97B active / total (MoE, 4 of 64 experts; checkpoint 23,971,587,712, active 2,454,808,192 incl. tied embeddings; card: 24.0B total, 2.45B active). English translations (DeepSeek + review). No thinking-off setup (bus #6516/#6924/#8187/#8202): the current served chat template exposes no supported switch for a separate thinking-off run; no separate off variant has been run or scored. (Checked 15 Nov in the pinned template: its only option, preserve_thinking, keeps or drops earlier turns' thinking.)
Parameters
{
  "max_tokens": 32768,
  "repetition_penalty": 1.05,
  "seed": 0,
  "temperature": 0.1,
  "top_k": 50
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · gemma-4-e2b-it-think_en_dontknow

Name
Gemma 4 E2B on · chat
Setup
Gemma 4 E2B (chat, thinking on, on our GPU) · chat model · with the I don't know option F
Model
gemma-4-e2b-it-think · provider vllm
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.2888 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 1.36 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.289 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gemma-4-e2b-it-think_en_dontknow · first answer 2026-11-13 02:03:58 UTC · last answer 2026-11-13 03:25:24 UTC · span 4886.0 s, pauses included (not charged) · charged 4886.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.2888 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 31.7 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 14 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 5.1B dense · 2.3B effective · source https://huggingface.co/google/gemma-4-E2B-it · read 2026-11-10 · the card states 2.3B effective (5.1B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 5.12B (counted from the checkpoint (5,123,178,979: language model 4.65B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 5.1B with embeddings, 2.3B effective; not a mixture of experts)
Input
text, English
Endpoint
http://lan-host:18624/v1/chat/completions
Started
13 Nov 2026 02:03 UTC; last answer 13 Nov 2026 03:25 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Gemma 4 E2B, served with vLLM v0.30.0 (docker vllm/vllm-openai@sha256:8a69ffad015f138d7170c4ddc429e230a3bc1c1719f67e14324749df200a4b90, the v0.30.0 release's index digest, linux/arm64 build sha256:4864d46625cb…; vLLM commit ced6857afa0e), BF16 from google/gemma-4-E2B-it@4adf15748de9e1a0097b6274cee9c15836c41c37, vLLM reasoning parser gemma4, one DGX Spark (a shared Spark host, NVIDIA GB10, 121 GB unified memory) alone; thinking on (chat_template_kwargs enable_thinking true, the template's own switch: it opens a system turn with <|think|>; the template default is off; on/off only, no levels); sampling temperature 0, as the Gemma 4 31B chat run; no seed (as the Gemma 4 31B run); max_tokens 8,192; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 5.12B / 5.12B active / total (dense; checkpoint 5,123,178,979 incl. vision 0.17B and audio 0.31B towers; card: 5.1B with per-layer embeddings, 2.3B effective). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 8192,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · qwen3.5-4b_en

Name
Qwen3.5-4B on · chat
Setup
Qwen3.5-4B (chat, thinking, on our GPU) · chat model
Model
qwen3.5-4b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$2.749 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 1.1 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $2.749. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-13 02:26:35–03:59:28 UTC
GPU time
run qwen3.5-4b_en · first answer 2026-11-13 02:24:20 UTC · last answer 2026-11-13 05:03:28 UTC · span 9548.0 s, pauses included (not charged) · charged 3975.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $2.7494 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.0 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 1 answer cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 4.66B dense · from the run's notes (dense; checkpoint 4,659,865,088 incl. vision 0.33B; card: 4B language model); also https://huggingface.co/Qwen/Qwen3.5-4B · read 2026-11-09 · counted from the checkpoint (4,659,865,088: language model 4.33B, vision encoder 0.33B); the card: 4B language model
Input
text, English
Endpoint
http://127.0.0.1:18604/v1/chat/completions
Started
13 Nov 2026 02:24 UTC; last answer 13 Nov 2026 05:03 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Qwen3.5-4B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from Qwen/Qwen3.5-4B@851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a, vLLM reasoning parser qwen3 (vision tower loaded, for Reading's Page images; text-only prompts here), one RTX PRO 6000 on our GPU host alone; thinking on (enable_thinking true; on/off only, no levels); sampling temperature 1, top_p 0.95, top_k 20, min_p 0, presence_penalty 1.5, repetition_penalty 1, sampling from the model card, generation_config has none (thinking mode, general tasks; the repo has no generation_config.json); seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 4.66B / 4.66B active / total (dense; checkpoint 4,659,865,088 incl. vision 0.33B; card: 4B language model). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "presence_penalty": 1.5,
  "repetition_penalty": 1.0,
  "seed": 0,
  "temperature": 1.0,
  "top_k": 20,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · gemma-4-e4b-it-think_en

Name
Gemma 4 E4B on · chat
Setup
Gemma 4 E4B (chat, thinking on, on our GPU) · chat model
Model
gemma-4-e4b-it-think · provider vllm
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.5037 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 2.37 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.504 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-13 03:46:54–07:00:16 UTC
GPU time
run gemma-4-e4b-it-think_en · first answer 2026-11-13 03:41:06 UTC · last answer 2026-11-13 09:16:28 UTC · span 20122.0 s, pauses included (not charged) · charged 8520.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.5037 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 30.8 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 6 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 8B dense · 4.5B effective · source https://huggingface.co/google/gemma-4-E4B-it · read 2026-11-10 · the card states 4.5B effective (8B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 8.00B (counted from the checkpoint (7,996,157,418: language model 7.52B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 8B with embeddings, 4.5B effective; not a mixture of experts)
Input
text, English
Endpoint
http://lan-host:18625/v1/chat/completions
Started
13 Nov 2026 03:40 UTC; last answer 13 Nov 2026 09:16 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Gemma 4 E4B, served with vLLM v0.30.0 (docker vllm/vllm-openai@sha256:8a69ffad015f138d7170c4ddc429e230a3bc1c1719f67e14324749df200a4b90, the v0.30.0 release's index digest, linux/arm64 build sha256:4864d46625cb…; vLLM commit ced6857afa0e), BF16 from google/gemma-4-E4B-it@4ecc2ba63bcac23ba4a767fc87025b1450f84c9c, vLLM reasoning parser gemma4, one DGX Spark (a shared Spark host, NVIDIA GB10, 121 GB unified memory) alone; thinking on (chat_template_kwargs enable_thinking true, the template's own switch: it opens a system turn with <|think|>; the template default is off; on/off only, no levels); sampling temperature 0, as the Gemma 4 31B chat run; no seed (as the Gemma 4 31B run); max_tokens 8,192; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 8.00B / 8.00B active / total (dense; checkpoint 7,996,157,418 incl. vision 0.17B and audio 0.31B towers; card: 8B with per-layer embeddings, 4.5B effective). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 8192,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · qwen3.5-4b_en_dontknow

Name
Qwen3.5-4B on · chat
Setup
Qwen3.5-4B (chat, thinking, on our GPU) · chat model · with the I don't know option F
Model
qwen3.5-4b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$2.238 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.899 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $2.238. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run qwen3.5-4b_en_dontknow · first answer 2026-11-13 06:34:52 UTC · last answer 2026-11-13 07:28:48 UTC · span 3236.0 s, pauses included (not charged) · charged 3236.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $2.2382 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.9 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 4.66B dense · from the run's notes (dense; checkpoint 4,659,865,088 incl. vision 0.33B; card: 4B language model); also https://huggingface.co/Qwen/Qwen3.5-4B · read 2026-11-09 · counted from the checkpoint (4,659,865,088: language model 4.33B, vision encoder 0.33B); the card: 4B language model
Input
text, English
Endpoint
http://127.0.0.1:18604/v1/chat/completions
Started
13 Nov 2026 06:34 UTC; last answer 13 Nov 2026 07:28 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Qwen3.5-4B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from Qwen/Qwen3.5-4B@851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a, vLLM reasoning parser qwen3 (vision tower loaded, for Reading's Page images; text-only prompts here), one RTX PRO 6000 on our GPU host alone; thinking on (enable_thinking true; on/off only, no levels); sampling temperature 1, top_p 0.95, top_k 20, min_p 0, presence_penalty 1.5, repetition_penalty 1, sampling from the model card, generation_config has none (thinking mode, general tasks; the repo has no generation_config.json); seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 4.66B / 4.66B active / total (dense; checkpoint 4,659,865,088 incl. vision 0.33B; card: 4B language model). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "presence_penalty": 1.5,
  "repetition_penalty": 1.0,
  "seed": 0,
  "temperature": 1.0,
  "top_k": 20,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · granite-4.2-3b_en

Name
Granite-4.2-3B high · chat
Setup
Granite-4.2-3B (chat, thinking, on our GPU) · chat model
Model
granite-4.2-3b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.07678 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0308 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.077. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run granite-4.2-3b_en · first answer 2026-11-13 09:43:20 UTC · last answer 2026-11-13 09:45:11 UTC · span 111.0 s, pauses included (not charged) · charged 111.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0768 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 3.0 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 3B dense · source https://huggingface.co/ibm-granite/granite-4.2-3b · read 2026-11-12 · the card states 3B (the maker's figure is shown); checkpoint count 3.66B (3,659,737,600 incl. untied lm_head, text only; dense)
Input
text, English
Endpoint
http://127.0.0.1:18616/v1/chat/completions
Started
13 Nov 2026 09:43 UTC; last answer 13 Nov 2026 09:45 UTC
Progress
25 answered of 2,163 asked; 0 request errors; smoke test at 32,768 tokens only; the full run uses a higher limit (run granite-4.2-3b-64k_en)
Notes
Granite-4.2-3B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from ibm-granite/granite-4.2-3b@4e9c275e232f10211ec88644d106a1f6a1b60e06, vLLM reasoning parser granite_thinking_parser, one RTX PRO 6000 on our GPU host alone; thinking on at full effort (enable_thinking true, reasoning_effort high; the template's levels: full or low); sampling temperature 1, top_p 0.95, from generation_config.json; seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 3.66B / 3.66B active / total (dense; checkpoint 3,659,737,600 incl. untied lm_head, text only; card: 3B). English translations (DeepSeek + review). SUPERSEDED by granite-4.2-3b-64k_en (cut-offs only; one retry at 65,536).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "reasoning_effort": "high"
  },
  "max_tokens": 32768,
  "seed": 0,
  "temperature": 1.0,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · granite-4.2-3b-64k_en

Name
Granite-4.2-3B high · chat
Setup
Granite-4.2-3B (chat, thinking, on our GPU) · chat model
Model
granite-4.2-3b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.689 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.678 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.689. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-13 10:00:39–14:44:55 UTC
GPU time
run granite-4.2-3b-64k_en · first answer 2026-11-13 09:58:48 UTC · last answer 2026-11-13 15:23:46 UTC · span 19498.0 s, pauses included (not charged) · charged 2442.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.6891 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 26.6 in flight (sum of request latencies / active time)
Output limit
65,536 tokens · 2 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 3B dense · source https://huggingface.co/ibm-granite/granite-4.2-3b · read 2026-11-12 · the card states 3B (the maker's figure is shown); checkpoint count 3.66B (3,659,737,600 incl. untied lm_head, text only; dense)
Input
text, English
Endpoint
http://127.0.0.1:18616/v1/chat/completions
Started
13 Nov 2026 09:58 UTC; last answer 13 Nov 2026 15:23 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Granite-4.2-3B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from ibm-granite/granite-4.2-3b@4e9c275e232f10211ec88644d106a1f6a1b60e06, vLLM reasoning parser granite_thinking_parser, one RTX PRO 6000 on our GPU host alone; thinking on at full effort (enable_thinking true, reasoning_effort high; the template's levels: full or low); sampling temperature 1, top_p 0.95, from generation_config.json; seed 0; max_tokens 65,536 (raised from 32,768 once: the smoke failed on cut-offs only); one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 3.66B / 3.66B active / total (dense; checkpoint 3,659,737,600 incl. untied lm_head, text only; card: 3B). English translations (DeepSeek + review). Full runs: 13 Nov 10:05-10:30Z at 48 requests in flight (vLLM --max-num-seqs 48; the KV cache filled, requests queued), from ~10:32Z at 32 in flight (--max-num-seqs 32), Main; the 25 smoke answers ran at 32 in flight with --max-num-seqs 16.
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "reasoning_effort": "high"
  },
  "max_tokens": 65536,
  "seed": 0,
  "temperature": 1.0,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · turion-jev-31b_en

Name
Turion Jev 31B
Setup
Turion Jev 31B · decision model
Model
turion-ai/turion-jev-31b · provider turion-jev
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.03778 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.177 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.038 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run turion-jev-31b_en · first answer 2026-11-13 11:42:48 UTC · last answer 2026-11-13 11:53:27 UTC · span 639.0 s, pauses included (not charged) · charged 639.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0378 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 31B dense · source https://huggingface.co/turion-ai/turion-jev-31b · read 2026-11-10 · the card states 31B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 31.27B (counted from the checkpoint (31,273,088,876: language model 30,697,345,340, vision tower 575,743,536, unused by its server); a fine-tune of Gemma 4 31B IT; the card: 31B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
13 Nov 2026 11:42 UTC; last answer 13 Nov 2026 11:53 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Turion Jev 31B (site run None); runner df9ae2734; turion-ai/turion-jev-31b@866c0fc6e040cba23fcc72113a80e5c99d5ebab1 (Apache-2.0; Gemma 4 31B IT LoRA merge; calibration.json temperatures) behind the card's server: ~/envs/turion-jev/bin/python -m turion_jev.serve --model ~/decision-models/turion-jev-31b@866c0fc6e040cba23fcc72113a80e5c99d5ebab1 --backend vllm --quantization fp8 --no-recheck --host 0.0.0.0 --port 19380; re-scoring off, as in the card's benchmark run; the server default is on; env ~/envs/turion-jev (vllm==0.32.0, torch 2.13.0+cu130, transformers 5.19.0, the repository's code/ with [serve]); text only; bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'turion-jev' (System One at /v1/systemone, model ignored, no confidence field, probabilities rounded to 6 places)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "turion-ai/turion-jev-31b",
    "--provider",
    "turion-jev",
    "--timeout",
    "120",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "6ad0eac51a4abb950bde74b24687209aec628aa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "turion-ai/turion-jev-31b",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "turion-jev",
  "questions": 2163,
  "request_seconds": 620.6344004901475,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "728d4f44794e0208975b6e7df88523a26cc1f1cdad32062de8f35ee61f205ce6",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 988236,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19380/v1/systemone
{
 "model": "turion-ai/turion-jev-31b",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · turion-jev-31b_en_dontknow

Name
Turion Jev 31B
Setup
Turion Jev 31B · decision model · with the I don't know option F
Model
turion-ai/turion-jev-31b · provider turion-jev
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.04197 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.197 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.042 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run turion-jev-31b_en_dontknow · first answer 2026-11-13 12:02:43 UTC · last answer 2026-11-13 12:14:33 UTC · span 710.0 s, pauses included (not charged) · charged 710.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0420 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 31B dense · source https://huggingface.co/turion-ai/turion-jev-31b · read 2026-11-10 · the card states 31B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 31.27B (counted from the checkpoint (31,273,088,876: language model 30,697,345,340, vision tower 575,743,536, unused by its server); a fine-tune of Gemma 4 31B IT; the card: 31B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
13 Nov 2026 12:02 UTC; last answer 13 Nov 2026 12:14 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Turion Jev 31B (site run None); runner df9ae2734; turion-ai/turion-jev-31b@866c0fc6e040cba23fcc72113a80e5c99d5ebab1 (Apache-2.0; Gemma 4 31B IT LoRA merge; calibration.json temperatures) behind the card's server: ~/envs/turion-jev/bin/python -m turion_jev.serve --model ~/decision-models/turion-jev-31b@866c0fc6e040cba23fcc72113a80e5c99d5ebab1 --backend vllm --quantization fp8 --no-recheck --host 0.0.0.0 --port 19380; re-scoring off, as in the card's benchmark run; the server default is on; env ~/envs/turion-jev (vllm==0.32.0, torch 2.13.0+cu130, transformers 5.19.0, the repository's code/ with [serve]); text only; bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'turion-jev' (System One at /v1/systemone, model ignored, no confidence field, probabilities rounded to 6 places)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "turion-ai/turion-jev-31b",
    "--provider",
    "turion-jev",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "6ad0eac51a4abb950bde74b24687209aec628aa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "turion-ai/turion-jev-31b",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "turion-jev",
  "questions": 2163,
  "request_seconds": 700.6721492392244,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "b314b89367057282bbef7b86886a985f698e46597d9a1ed76303e44c74d254d3",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 1195884,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19380/v1/systemone
{
 "model": "turion-ai/turion-jev-31b",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · gemma-4-e4b-it-think_en_dontknow

Name
Gemma 4 E4B on · chat
Setup
Gemma 4 E4B (chat, thinking on, on our GPU) · chat model · with the I don't know option F
Model
gemma-4-e4b-it-think · provider vllm
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.478 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 2.25 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.478 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gemma-4-e4b-it-think_en_dontknow · first answer 2026-11-13 12:25:36 UTC · last answer 2026-11-13 14:40:21 UTC · span 8085.0 s, pauses included (not charged) · charged 8085.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.4780 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 31.8 in flight (sum of request latencies / active time)
Output limit
8,192 tokens · 5 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 8B dense · 4.5B effective · source https://huggingface.co/google/gemma-4-E4B-it · read 2026-11-10 · the card states 4.5B effective (8B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 8.00B (counted from the checkpoint (7,996,157,418: language model 7.52B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 8B with embeddings, 4.5B effective; not a mixture of experts)
Input
text, English
Endpoint
http://lan-host:18625/v1/chat/completions
Started
13 Nov 2026 12:25 UTC; last answer 13 Nov 2026 14:40 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Gemma 4 E4B, served with vLLM v0.30.0 (docker vllm/vllm-openai@sha256:8a69ffad015f138d7170c4ddc429e230a3bc1c1719f67e14324749df200a4b90, the v0.30.0 release's index digest, linux/arm64 build sha256:4864d46625cb…; vLLM commit ced6857afa0e), BF16 from google/gemma-4-E4B-it@4ecc2ba63bcac23ba4a767fc87025b1450f84c9c, vLLM reasoning parser gemma4, one DGX Spark (a shared Spark host, NVIDIA GB10, 121 GB unified memory) alone; thinking on (chat_template_kwargs enable_thinking true, the template's own switch: it opens a system turn with <|think|>; the template default is off; on/off only, no levels); sampling temperature 0, as the Gemma 4 31B chat run; no seed (as the Gemma 4 31B run); max_tokens 8,192; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 8.00B / 8.00B active / total (dense; checkpoint 7,996,157,418 incl. vision 0.17B and audio 0.31B towers; card: 8B with per-layer embeddings, 4.5B effective). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 8192,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · startlux-4b_en

Name
StartLux-Decision-v1.1-4B-Instant
Setup
StartLux-Decision v1.1 4B · decision model
Model
startlux-models/StartLux-Decision-v1.1-4B-Instant · provider startlux
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.008867 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0417 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.009 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run startlux-4b_en · first answer 2026-11-13 16:01:48 UTC · last answer 2026-11-13 16:04:18 UTC · span 150.0 s, pauses included (not charged) · charged 150.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0089 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 4B dense · source https://huggingface.co/startlux-models/StartLux-Decision-v1.1-4B-Instant · read 2026-11-10 · the card states 4B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 4.66B (counted from the checkpoint (4,659,865,088: language model incl. MTP layer 4,326,350,848, vision tower 333,514,240); from Qwen3.5-4B; the card: 4B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
13 Nov 2026 16:01 UTC; last answer 13 Nov 2026 16:04 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup StartLux-Decision v1.1 4B (site run None); runner df9ae2734; startlux-models/StartLux-Decision-v1.1-4B-Instant@416088f77cfce09633906a3675c7cd2e5dabcf10 (weights CC BY-NC 4.0, non-commercial: run for this non-commercial benchmark; code Apache-2.0; Qwen3.5-4B, prefix-bidirectional vLLM plugin) behind the card's server (github.com/StartLuxLabs/StartLux-Decision at a99977d): ~/envs/startlux/bin/startlux-decide serve --model ~/decision-models/StartLux-Decision-v1.1-4B-Instant@416088f77cfce09633906a3675c7cd2e5dabcf10 --launch-vllm --gpus 0 --host 0.0.0.0 --port 19410 (env ~/envs/startlux: vllm==0.31.0, torch 2.13.0+cu130, the repository at a99977d with [vllm]; vLLM 0.31.0 because 0.19.1, the version of the card's published scores, has no aarch64 build; the plugin is validated on both); vision on (the server's default); bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'startlux' (System One at /v1/systemone, model ignored)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "startlux-models/StartLux-Decision-v1.1-4B-Instant",
    "--provider",
    "startlux",
    "--timeout",
    "60",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "6ad0eac51a4abb950bde74b24687209aec628aa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "startlux-models/StartLux-Decision-v1.1-4B-Instant",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "startlux",
  "questions": 2163,
  "request_seconds": 144.68232458422426,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "78a51ef9139644e805492f03e2893ec70b4df6f72584875a9b06e9e578e5908b",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 416668,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19410/v1/systemone
{
 "model": "startlux-models/StartLux-Decision-v1.1-4B-Instant",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · startlux-4b_en_dontknow

Name
StartLux-Decision-v1.1-4B-Instant
Setup
StartLux-Decision v1.1 4B · decision model · with the I don't know option F
Model
startlux-models/StartLux-Decision-v1.1-4B-Instant · provider startlux
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.009104 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0428 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.009 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run startlux-4b_en_dontknow · first answer 2026-11-13 16:05:59 UTC · last answer 2026-11-13 16:08:33 UTC · span 154.0 s, pauses included (not charged) · charged 154.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0091 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 4B dense · source https://huggingface.co/startlux-models/StartLux-Decision-v1.1-4B-Instant · read 2026-11-10 · the card states 4B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 4.66B (counted from the checkpoint (4,659,865,088: language model incl. MTP layer 4,326,350,848, vision tower 333,514,240); from Qwen3.5-4B; the card: 4B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
13 Nov 2026 16:05 UTC; last answer 13 Nov 2026 16:08 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup StartLux-Decision v1.1 4B (site run None); runner df9ae2734; startlux-models/StartLux-Decision-v1.1-4B-Instant@416088f77cfce09633906a3675c7cd2e5dabcf10 (weights CC BY-NC 4.0, non-commercial: run for this non-commercial benchmark; code Apache-2.0; Qwen3.5-4B, prefix-bidirectional vLLM plugin) behind the card's server (github.com/StartLuxLabs/StartLux-Decision at a99977d): ~/envs/startlux/bin/startlux-decide serve --model ~/decision-models/StartLux-Decision-v1.1-4B-Instant@416088f77cfce09633906a3675c7cd2e5dabcf10 --launch-vllm --gpus 0 --host 0.0.0.0 --port 19410 (env ~/envs/startlux: vllm==0.31.0, torch 2.13.0+cu130, the repository at a99977d with [vllm]; vLLM 0.31.0 because 0.19.1, the version of the card's published scores, has no aarch64 build; the plugin is validated on both); vision on (the server's default); bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'startlux' (System One at /v1/systemone, model ignored)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "startlux-models/StartLux-Decision-v1.1-4B-Instant",
    "--provider",
    "startlux",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "6ad0eac51a4abb950bde74b24687209aec628aa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "startlux-models/StartLux-Decision-v1.1-4B-Instant",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "startlux",
  "questions": 2163,
  "request_seconds": 148.73927892866777,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "eaca1194569ed7b4a587032f41e4d9bb80ae227e84798c0efa942c24ebe213af",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 477232,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19410/v1/systemone
{
 "model": "startlux-models/StartLux-Decision-v1.1-4B-Instant",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · claude-fable-5.1-sub_en

Name
Claude Fable 5.1 chat
Setup
Claude Fable 5.1 · chat model
Model
claude-fable-5-1 · provider claude-subscription
GPU
—
Cost
$0 for the run · Claude subscription
Answer time
run claude-fable-5.1-sub_en · first answer 2026-11-13 16:23:09 UTC · last answer 2026-11-14 03:18:42 UTC · span 39333.0 s, pauses included (not charged) · charged 10126.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker (closed weights) · source https://platform.claude.com/docs/en/about-claude/models · read 2026-11-13
Input
text, English
Endpoint
claude-code-subscription
Started
13 Nov 2026 16:23 UTC; last answer 14 Nov 2026 03:18 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Claude Fable 5.1 on the Claude subscription through the Claude Code CLI (`claude --print`, as turkce's runs 968/971), model claude-fable-5-1, --effort high (the highest of low/medium/high/xhigh/max; adaptive thinking always on); CLAUDE_CODE_MAX_OUTPUT_TOKENS 32768; no temperature sent (Anthropic's default 1.0); Claude Code's agent prompt replaced by a neutral one-line system prompt, tools, settings, MCP servers and session saving off, API-key variables removed; one question per request, one request at a time; cost $0 (Claude subscription); api_equivalent_usd is Claude Code's API-list-price estimate, not a cost. English translations.
Parameters
{
  "effort": "high",
  "max_output_tokens": 32768,
  "route": "claude-code-cli --print",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message."
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · qwen3.8-flash_en

Name
Qwen3.8 Flash off · chat
Setup
Qwen3.8 Flash · chat model
Model
qwen/qwen3.8-flash · provider openrouter
GPU
—
Cost
$0.8585 for the run · provider price × reported usage
Answer time
run qwen3.8-flash_en · first answer 2026-11-13 16:30:28 UTC · last answer 2026-11-13 17:05:51 UTC · span 2123.0 s, pauses included (not charged) · charged 2123.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 7.5 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: not disclosed by maker (closed weights) · source https://openrouter.ai/qwen/qwen3.8-flash · read 2026-11-13
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
13 Nov 2026 16:30 UTC; last answer 13 Nov 2026 17:05 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Qwen3.8 Flash through OpenRouter (qwen/qwen3.8-flash), provider pinned to alibaba, fallbacks off; reasoning {"enabled": false}; temperature 1.0; max_tokens 32768; one question per request; parameters not disclosed (closed weights). English translations.
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "alibaba"
    ]
  },
  "reasoning": {
    "enabled": false
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · cerno-9b_en

Name
Cerno 9B v1.2
Setup
Cerno 9B v1.2 · decision model
Model
apertixai/cerno-9b-v1.2 · provider cerno
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.02057 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0967 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.021 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run cerno-9b_en · first answer 2026-11-13 16:36:15 UTC · last answer 2026-11-13 16:42:03 UTC · span 348.0 s, pauses included (not charged) · charged 348.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0206 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/apertixai/cerno-9b-v1.2 · read 2026-11-10 · the card states 9B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 8.95B (counted from the checkpoint (8,953,803,264: backbone 7,936,684,544 + untied lm_head 1,017,118,720; text only); a full fine-tune of Qwen3.5-9B; the card: 9B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
13 Nov 2026 16:36 UTC; last answer 13 Nov 2026 16:42 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Cerno 9B v1.2 (site run None); runner df9ae2734; apertixai/cerno-9b-v1.2@f02a632fbe3a4698cd1ff5c0b929ea240e2b8eda (Apache-2.0; Qwen3.5-9B full fine-tune, block attention, read twice up to 3,000 tokens, temperature 1.256 from decider_config.json); served by our thin wrapper around the card's CernoEngine (decision/servers/cerno_server.py: request/response mapping only; vendored decider 1c15a48, decision-index kit 9eb2dbe): PYTHONPATH=~/decision-models/cerno-9b-v1.2@f02a632fbe3a4698cd1ff5c0b929ea240e2b8eda/inference HF_HUB_OFFLINE=1 ~/envs/cerno/bin/python ~/decision-src/cerno_server.py --model ~/decision-models/cerno-9b-v1.2@f02a632fbe3a4698cd1ff5c0b929ea240e2b8eda --name apertixai/cerno-9b-v1.2 --host 0.0.0.0 --port 19390 (env ~/envs/cerno exactly as the card's inference/requirements-v1.2.txt: torch 2.14.1+cu130, transformers 5.17.0, flash-linear-attention 0.5.2, decision-index kit 9eb2dbe; usage.input_tokens is the engine's own count); text only; bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'cerno'; probabilities rounded to 4 places
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "apertixai/cerno-9b-v1.2",
    "--provider",
    "cerno",
    "--timeout",
    "120",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "6ad0eac51a4abb950bde74b24687209aec628aa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "apertixai/cerno-9b-v1.2",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "cerno",
  "questions": 2163,
  "request_seconds": 342.1298995965626,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "e8c1e9a3da70d16662b7ab6a894aca48976dc4e2fa68035ef146bf1607a43e11",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 692659,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19390/v1/systemone
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · cerno-9b_en_dontknow

Name
Cerno 9B v1.2
Setup
Cerno 9B v1.2 · decision model · with the I don't know option F
Model
apertixai/cerno-9b-v1.2 · provider cerno
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.0227 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.107 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.023 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run cerno-9b_en_dontknow · first answer 2026-11-13 16:46:54 UTC · last answer 2026-11-13 16:53:18 UTC · span 384.0 s, pauses included (not charged) · charged 384.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0227 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/apertixai/cerno-9b-v1.2 · read 2026-11-10 · the card states 9B (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 8.95B (counted from the checkpoint (8,953,803,264: backbone 7,936,684,544 + untied lm_head 1,017,118,720; text only); a full fine-tune of Qwen3.5-9B; the card: 9B)
Input
text, English
Endpoint
internal address on a shared Spark host
Started
13 Nov 2026 16:46 UTC; last answer 13 Nov 2026 16:53 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Cerno 9B v1.2 (site run None); runner df9ae2734; apertixai/cerno-9b-v1.2@f02a632fbe3a4698cd1ff5c0b929ea240e2b8eda (Apache-2.0; Qwen3.5-9B full fine-tune, block attention, read twice up to 3,000 tokens, temperature 1.256 from decider_config.json); served by our thin wrapper around the card's CernoEngine (decision/servers/cerno_server.py: request/response mapping only; vendored decider 1c15a48, decision-index kit 9eb2dbe): PYTHONPATH=~/decision-models/cerno-9b-v1.2@f02a632fbe3a4698cd1ff5c0b929ea240e2b8eda/inference HF_HUB_OFFLINE=1 ~/envs/cerno/bin/python ~/decision-src/cerno_server.py --model ~/decision-models/cerno-9b-v1.2@f02a632fbe3a4698cd1ff5c0b929ea240e2b8eda --name apertixai/cerno-9b-v1.2 --host 0.0.0.0 --port 19390 (env ~/envs/cerno exactly as the card's inference/requirements-v1.2.txt: torch 2.14.1+cu130, transformers 5.17.0, flash-linear-attention 0.5.2, decision-index kit 9eb2dbe; usage.input_tokens is the engine's own count); text only; bound to 0.0.0.0, clients use the server's internal address; a shared Spark host (DGX Spark, GB10); runner provider 'cerno'; probabilities rounded to 4 places
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "apertixai/cerno-9b-v1.2",
    "--provider",
    "cerno",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "6ad0eac51a4abb950bde74b24687209aec628aa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "apertixai/cerno-9b-v1.2",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "cerno",
  "questions": 2163,
  "request_seconds": 365.6972717298195,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "7e59837306cd3bc28f261586e8ee3179995b45fc9536ffe6d8cb8aec87994609",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 818089,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19390/v1/systemone
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · qwen3.8-flash_en_dontknow

Name
Qwen3.8 Flash off · chat
Setup
Qwen3.8 Flash · chat model · with the I don't know option F
Model
qwen/qwen3.8-flash · provider openrouter
GPU
—
Cost
$0.8276 for the run · provider price × reported usage
Answer time
run qwen3.8-flash_en_dontknow · first answer 2026-11-13 17:05:57 UTC · last answer 2026-11-13 17:36:52 UTC · span 1855.0 s, pauses included (not charged) · charged 1855.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 8.0 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: not disclosed by maker (closed weights) · source https://openrouter.ai/qwen/qwen3.8-flash · read 2026-11-13
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
13 Nov 2026 17:05 UTC; last answer 13 Nov 2026 17:36 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Qwen3.8 Flash through OpenRouter (qwen/qwen3.8-flash), provider pinned to alibaba, fallbacks off; reasoning {"enabled": false}; temperature 1.0; max_tokens 32768; one question per request; parameters not disclosed (closed weights). English translations.
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "alibaba"
    ]
  },
  "reasoning": {
    "enabled": false
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · lfm3-1.7b-vllm_en

Name
LFM3-1.7B on · chat
Setup
LFM3-1.7B (chat, thinking, on our GPU) · chat model
Model
lfm3-1.7b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.2808 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.113 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.281. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run lfm3-1.7b-vllm_en · first answer 2026-11-13 17:17:18 UTC · last answer 2026-11-13 17:24:04 UTC · span 406.0 s, pauses included (not charged) · charged 406.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.2808 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 30.6 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 1.73B dense · from the run's notes (dense; BF16 safetensors 1,728,801,536, text only; card: 1.7B); also https://huggingface.co/LiquidAI/LFM3-1.7B-GGUF · read 2026-11-09 · the released checkpoint's tensors without registered buffers: 1,728,801,536 (the served BF16 GGUF also stores a 64-element RoPE buffer, 1,728,801,600); text only; card: 1.7B
Input
text, English
Endpoint
http://127.0.0.1:18634/v1/chat/completions
Started
13 Nov 2026 17:17 UTC; last answer 13 Nov 2026 17:24 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
LFM3-1.7B, served with vLLM 0.34.0rc1.dev302+gbe47189f3 (nightly, commit be47189f37f6, CUDA 13.5 image; docker vllm/vllm-openai@sha256:7f0618da95bac2f1cc97fcd1791380d7c1a06d24a4998da45aeb7fbe6c49179e), BF16 from LiquidAI/LFM3-1.7B@0dc4268b561499c648c7ba7c9466bf3d8d01517a, vLLM reasoning parser qwen3 (vLLM nightly because no vLLM release has Lfm3ForCausalLM; a clean rerun of the standard runs entirely on this engine with the dont_know runs' server and settings (Umit #6250), replacing lfm3-1.7b_en, whose answers mix llama.cpp b12020 and vLLM; the same weights as the llama.cpp setup's official BF16 GGUF (tensors byte-identical); qualification work/lfm3_vllm_qual.md), one RTX PRO 6000 on our GPU host alone; thinking on (enable_thinking true, the template default; on/off only); sampling temperature 0.1, top_k 50, repetition_penalty 1.1, top_p 1, min_p 0, from generation_config.json, passed 1:1 as vLLM sampling params; seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 1.73B / 1.73B active / total (dense; BF16 safetensors 1,728,801,536, text only; card: 1.7B). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "repetition_penalty": 1.1,
  "seed": 0,
  "temperature": 0.1,
  "top_k": 50,
  "top_p": 1.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · lfm3-1.7b-vllm-nothink_en

Name
LFM3-1.7B off · chat
Setup
LFM3-1.7B (chat, thinking off, on our GPU) · chat model
Model
lfm3-1.7b-vllm-nothink · provider vllm
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.1296 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.609 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.130 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run lfm3-1.7b-vllm-nothink_en · first answer 2026-11-13 17:23:40 UTC · last answer 2026-11-13 18:00:13 UTC · span 2193.0 s, pauses included (not charged) · charged 2193.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1296 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 29.2 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 1.73B dense · from the run's notes (dense; BF16 safetensors 1,728,801,536, text only; card: 1.7B); also https://huggingface.co/LiquidAI/LFM3-1.7B-GGUF · read 2026-11-09 · the released checkpoint's tensors without registered buffers: 1,728,801,536 (the served BF16 GGUF also stores a 64-element RoPE buffer, 1,728,801,600); text only; card: 1.7B
Input
text, English
Endpoint
http://lan-host:18626/v1/chat/completions
Started
13 Nov 2026 17:23 UTC; last answer 13 Nov 2026 18:00 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
LFM3-1.7B, served with vLLM 0.34.0rc1.dev302+gbe47189f3 (nightly, commit be47189f37f6, CUDA 13.5 image; docker vllm/vllm-openai@sha256:7f0618da95bac2f1cc97fcd1791380d7c1a06d24a4998da45aeb7fbe6c49179e, linux/arm64 build sha256:c41b69fa5f34…), BF16 from LiquidAI/LFM3-1.7B@0dc4268b561499c648c7ba7c9466bf3d8d01517a, vLLM reasoning parser qwen3 (thinking off via chat_template_kwargs enable_thinking false, the template's own switch (the assistant turn opens with an empty <think></think>), not a prompt instruction; otherwise the settings of the thinking-on vLLM runs lfm3-1.7b_en (Umit #6101); vLLM nightly because no vLLM release has Lfm3ForCausalLM; qualification work/lfm3_vllm_qual.md), one DGX Spark (a shared Spark host, NVIDIA GB10, 121 GB unified memory) alone; thinking off (chat_template_kwargs enable_thinking false; the template default true is overridden); sampling temperature 0.1, top_k 50, repetition_penalty 1.1, top_p 1, min_p 0, from generation_config.json, passed 1:1 as vLLM sampling params; seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 1.73B / 1.73B active / total (dense; BF16 safetensors 1,728,801,536, text only; card: 1.7B). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "repetition_penalty": 1.1,
  "seed": 0,
  "temperature": 0.1,
  "top_k": 50,
  "top_p": 1.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · minicpm5-2b-nothink_en

Name
MiniCPM5-2B off · chat
Setup
MiniCPM5-2B (chat, thinking off, on our GPU) · chat model
Model
minicpm5-2b-nothink · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.48 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.193 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.480. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-13 18:16:51–18:56:13 UTC
GPU time
run minicpm5-2b-nothink_en · first answer 2026-11-13 18:16:22 UTC · last answer 2026-11-13 19:07:18 UTC · span 3056.0 s, pauses included (not charged) · charged 694.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.4800 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 30.4 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 2.52B dense · from the run's notes (dense; checkpoint 2,516,756,480 incl. untied lm_head, text only; card: 2,516,756,480); also https://huggingface.co/openbmb/MiniCPM5-2B · read 2026-11-12 · counted from the checkpoint (2,516,756,480 incl. untied lm_head, text only; dense); the card: 2,516,756,480
Input
text, English
Endpoint
http://127.0.0.1:18615/v1/chat/completions
Started
13 Nov 2026 18:16 UTC; last answer 13 Nov 2026 19:07 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
MiniCPM5-2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from openbmb/MiniCPM5-2B@42433e90dad95123ab87729fd7011e7110629556, vLLM reasoning parser qwen3, one RTX PRO 6000 on our GPU host alone; thinking off (enable_thinking false, the template's own switch); sampling temperature 1, top_p 0.95, from generation_config.json; seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 2.52B / 2.52B active / total (dense; checkpoint 2,516,756,480 incl. untied lm_head, text only; card: 2,516,756,480). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "max_tokens": 32768,
  "seed": 0,
  "temperature": 1.0,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · minicpm5-2b-nothink_en_dontknow

Name
MiniCPM5-2B off · chat
Setup
MiniCPM5-2B (chat, thinking off, on our GPU) · chat model · with the I don't know option F
Model
minicpm5-2b-nothink · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.3631 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.146 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.363. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run minicpm5-2b-nothink_en_dontknow · first answer 2026-11-13 19:46:48 UTC · last answer 2026-11-13 19:55:33 UTC · span 525.0 s, pauses included (not charged) · charged 525.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.3631 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.2 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 2.52B dense · from the run's notes (dense; checkpoint 2,516,756,480 incl. untied lm_head, text only; card: 2,516,756,480); also https://huggingface.co/openbmb/MiniCPM5-2B · read 2026-11-12 · counted from the checkpoint (2,516,756,480 incl. untied lm_head, text only; dense); the card: 2,516,756,480
Input
text, English
Endpoint
http://127.0.0.1:18615/v1/chat/completions
Started
13 Nov 2026 19:46 UTC; last answer 13 Nov 2026 19:55 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
MiniCPM5-2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from openbmb/MiniCPM5-2B@42433e90dad95123ab87729fd7011e7110629556, vLLM reasoning parser qwen3, one RTX PRO 6000 on our GPU host alone; thinking off (enable_thinking false, the template's own switch); sampling temperature 1, top_p 0.95, from generation_config.json; seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 2.52B / 2.52B active / total (dense; checkpoint 2,516,756,480 incl. untied lm_head, text only; card: 2,516,756,480). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "max_tokens": 32768,
  "seed": 0,
  "temperature": 1.0,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · lfm3-1.7b-vllm-nothink_en_dontknow

Name
LFM3-1.7B off · chat
Setup
LFM3-1.7B (chat, thinking off, on our GPU) · chat model · with the I don't know option F
Model
lfm3-1.7b-vllm-nothink · provider vllm
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.126 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.592 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.126 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run lfm3-1.7b-vllm-nothink_en_dontknow · first answer 2026-11-13 20:27:04 UTC · last answer 2026-11-13 21:02:35 UTC · span 2131.0 s, pauses included (not charged) · charged 2131.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1260 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 30.2 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 1.73B dense · from the run's notes (dense; BF16 safetensors 1,728,801,536, text only; card: 1.7B); also https://huggingface.co/LiquidAI/LFM3-1.7B-GGUF · read 2026-11-09 · the released checkpoint's tensors without registered buffers: 1,728,801,536 (the served BF16 GGUF also stores a 64-element RoPE buffer, 1,728,801,600); text only; card: 1.7B
Input
text, English
Endpoint
http://lan-host:18626/v1/chat/completions
Started
13 Nov 2026 20:26 UTC; last answer 13 Nov 2026 21:02 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
LFM3-1.7B, served with vLLM 0.34.0rc1.dev302+gbe47189f3 (nightly, commit be47189f37f6, CUDA 13.5 image; docker vllm/vllm-openai@sha256:7f0618da95bac2f1cc97fcd1791380d7c1a06d24a4998da45aeb7fbe6c49179e, linux/arm64 build sha256:c41b69fa5f34…), BF16 from LiquidAI/LFM3-1.7B@0dc4268b561499c648c7ba7c9466bf3d8d01517a, vLLM reasoning parser qwen3 (thinking off via chat_template_kwargs enable_thinking false, the template's own switch (the assistant turn opens with an empty <think></think>), not a prompt instruction; otherwise the settings of the thinking-on vLLM runs lfm3-1.7b_en (Umit #6101); vLLM nightly because no vLLM release has Lfm3ForCausalLM; qualification work/lfm3_vllm_qual.md), one DGX Spark (a shared Spark host, NVIDIA GB10, 121 GB unified memory) alone; thinking off (chat_template_kwargs enable_thinking false; the template default true is overridden); sampling temperature 0.1, top_k 50, repetition_penalty 1.1, top_p 1, min_p 0, from generation_config.json, passed 1:1 as vLLM sampling params; seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 1.73B / 1.73B active / total (dense; BF16 safetensors 1,728,801,536, text only; card: 1.7B). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "repetition_penalty": 1.1,
  "seed": 0,
  "temperature": 0.1,
  "top_k": 50,
  "top_p": 1.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · gpt-6.1-terra-sub_en

Name
GPT-6.1 Terra chat
Setup
GPT-6.1 Terra · chat model
Model
gpt-6.1-terra · provider chatgpt-subscription
GPU
—
Cost
$0 for the run · ChatGPT subscription
Answer time
run gpt-6.1-terra-sub_en · first answer 2026-11-13 23:24:46 UTC · last answer 2026-11-14 02:09:04 UTC · span 9858.0 s, pauses included (not charged) · charged 9858.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not on record yet
Input
text, English
Endpoint
codex-chatgpt-subscription
Started
13 Nov 2026 23:24 UTC; last answer 14 Nov 2026 02:09 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
GPT-6.1 Terra on the ChatGPT subscription through the Codex CLI (`codex exec`, ChatGPT sign-in from OMP's openai-codex login in a private CODEX_HOME, no refresh token, since 13 Nov 19:35 UTC; as turkce's ask_openai), model gpt-6.1-terra, model_reasoning_effort high (Reading's setting); no output limit sent (Codex has no setting for it) and no temperature; Codex's agent instructions replaced by a neutral one-line system prompt, web search and tool features off, read-only sandbox, no user config, rules or saved session, API-key variables removed; one question per request, one request at a time; a usage-limit reply stops the run without retrying it (resumable); cost $0 (ChatGPT subscription); api_equivalent_usd is the tokens at OpenAI's API list price {"input": 1.0, "cached_input": 0.05, "cache_write": 1.25, "output": 5.0} per 1M tokens, not a cost. Codex reports no served model id. English translations.
Parameters
{
  "disabled_features": [
    "shell_tool",
    "unified_exec",
    "goals",
    "apps",
    "plugins",
    "remote_plugin",
    "tool_suggest",
    "skill_search",
    "view_image",
    "image_generation",
    "sleep_tool",
    "browser_use",
    "browser_use_external",
    "computer_use",
    "in_app_browser",
    "multi_agent",
    "unbounded_connection_retries"
  ],
  "effort": "high",
  "list_price_per_mtok": {
    "cache_write": 1.25,
    "cached_input": 0.05,
    "input": 1.0,
    "output": 5.0
  },
  "max_output_tokens": null,
  "route": "codex exec (ChatGPT sign-in)",
  "sandbox": "read-only",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message.",
  "web_search": "disabled"
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · gpt-6.1-terra-sub_en_dontknow

Name
GPT-6.1 Terra chat
Setup
GPT-6.1 Terra · chat model · with the I don't know option F
Model
gpt-6.1-terra · provider chatgpt-subscription
GPU
—
Cost
$0 for the run · ChatGPT subscription
Answer time
run gpt-6.1-terra-sub_en_dontknow · first answer 2026-11-14 02:09:09 UTC · last answer 2026-11-14 04:32:46 UTC · span 8617.0 s, pauses included (not charged) · charged 8617.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not on record yet
Input
text, English
Endpoint
codex-chatgpt-subscription
Started
14 Nov 2026 02:09 UTC; last answer 14 Nov 2026 04:32 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
GPT-6.1 Terra on the ChatGPT subscription through the Codex CLI (`codex exec`, ChatGPT sign-in from OMP's openai-codex login in a private CODEX_HOME, no refresh token, since 13 Nov 19:35 UTC; as turkce's ask_openai), model gpt-6.1-terra, model_reasoning_effort high (Reading's setting); no output limit sent (Codex has no setting for it) and no temperature; Codex's agent instructions replaced by a neutral one-line system prompt, web search and tool features off, read-only sandbox, no user config, rules or saved session, API-key variables removed; one question per request, one request at a time; a usage-limit reply stops the run without retrying it (resumable); cost $0 (ChatGPT subscription); api_equivalent_usd is the tokens at OpenAI's API list price {"input": 1.0, "cached_input": 0.05, "cache_write": 1.25, "output": 5.0} per 1M tokens, not a cost. Codex reports no served model id. English translations.
Parameters
{
  "disabled_features": [
    "shell_tool",
    "unified_exec",
    "goals",
    "apps",
    "plugins",
    "remote_plugin",
    "tool_suggest",
    "skill_search",
    "view_image",
    "image_generation",
    "sleep_tool",
    "browser_use",
    "browser_use_external",
    "computer_use",
    "in_app_browser",
    "multi_agent",
    "unbounded_connection_retries"
  ],
  "effort": "high",
  "list_price_per_mtok": {
    "cache_write": 1.25,
    "cached_input": 0.05,
    "input": 1.0,
    "output": 5.0
  },
  "max_output_tokens": null,
  "route": "codex exec (ChatGPT sign-in)",
  "sandbox": "read-only",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message.",
  "web_search": "disabled"
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · claude-fable-5.1-sub_en_dontknow

Name
Claude Fable 5.1 chat
Setup
Claude Fable 5.1 · chat model · with the I don't know option F
Model
claude-fable-5-1 · provider claude-subscription
GPU
—
Cost
$0 for the run · Claude subscription
Answer time
run claude-fable-5.1-sub_en_dontknow · first answer 2026-11-14 03:18:48 UTC · last answer 2026-11-14 06:40:23 UTC · span 12095.0 s, pauses included (not charged) · charged 12095.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker (closed weights) · source https://platform.claude.com/docs/en/about-claude/models · read 2026-11-13
Input
text, English
Endpoint
claude-code-subscription
Started
14 Nov 2026 03:18 UTC; last answer 14 Nov 2026 06:40 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Claude Fable 5.1 on the Claude subscription through the Claude Code CLI (`claude --print`, as turkce's runs 968/971), model claude-fable-5-1, --effort high (the highest of low/medium/high/xhigh/max; adaptive thinking always on); CLAUDE_CODE_MAX_OUTPUT_TOKENS 32768; no temperature sent (Anthropic's default 1.0); Claude Code's agent prompt replaced by a neutral one-line system prompt, tools, settings, MCP servers and session saving off, API-key variables removed; one question per request, one request at a time; cost $0 (Claude subscription); api_equivalent_usd is Claude Code's API-list-price estimate, not a cost. English translations.
Parameters
{
  "effort": "high",
  "max_output_tokens": 32768,
  "route": "claude-code-cli --print",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message."
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · lfm3-1.7b-vllm-seed1_en

Name
LFM3-1.7B on · chat
Setup
LFM3-1.7B (chat, thinking, on our GPU) · chat model
Model
lfm3-1.7b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.2774 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.111 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.277. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run lfm3-1.7b-vllm-seed1_en · first answer 2026-11-14 03:24:50 UTC · last answer 2026-11-14 03:31:31 UTC · span 401.0 s, pauses included (not charged) · charged 401.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.2774 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.0 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 1.73B dense · from the run's notes (dense; BF16 safetensors 1,728,801,536, text only; card: 1.7B); also https://huggingface.co/LiquidAI/LFM3-1.7B-GGUF · read 2026-11-09 · the released checkpoint's tensors without registered buffers: 1,728,801,536 (the served BF16 GGUF also stores a 64-element RoPE buffer, 1,728,801,600); text only; card: 1.7B
Input
text, English
Endpoint
http://127.0.0.1:18634/v1/chat/completions
Started
14 Nov 2026 03:24 UTC; last answer 14 Nov 2026 03:31 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
LFM3-1.7B, served with vLLM 0.34.0rc1.dev302+gbe47189f3 (nightly, commit be47189f37f6, CUDA 13.5 image; docker vllm/vllm-openai@sha256:7f0618da95bac2f1cc97fcd1791380d7c1a06d24a4998da45aeb7fbe6c49179e), BF16 from LiquidAI/LFM3-1.7B@0dc4268b561499c648c7ba7c9466bf3d8d01517a, vLLM reasoning parser qwen3 (seed 1 of three (Umit via zeki #6969: seeds 0, 1, 2; seed 0 is lfm3-1.7b-vllm_en); vLLM nightly because no vLLM release has Lfm3ForCausalLM; the same server and settings as seed 0), one RTX PRO 6000 on our GPU host alone; thinking on (enable_thinking true, the template default; on/off only); sampling temperature 0.1, top_k 50, repetition_penalty 1.1, top_p 1, min_p 0, from generation_config.json, passed 1:1 as vLLM sampling params; seed 1; max_tokens 32,768; one question per request (32 requests in flight), standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 1.73B / 1.73B active / total (dense; BF16 safetensors 1,728,801,536, text only; card: 1.7B). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "repetition_penalty": 1.1,
  "seed": 1,
  "temperature": 0.1,
  "top_k": 50,
  "top_p": 1.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · lfm3-1.7b-vllm-seed2_en

Name
LFM3-1.7B on · chat
Setup
LFM3-1.7B (chat, thinking, on our GPU) · chat model
Model
lfm3-1.7b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.3009 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.121 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.301. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run lfm3-1.7b-vllm-seed2_en · first answer 2026-11-14 04:04:43 UTC · last answer 2026-11-14 04:11:58 UTC · span 435.0 s, pauses included (not charged) · charged 435.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.3009 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 28.5 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 1.73B dense · from the run's notes (dense; BF16 safetensors 1,728,801,536, text only; card: 1.7B); also https://huggingface.co/LiquidAI/LFM3-1.7B-GGUF · read 2026-11-09 · the released checkpoint's tensors without registered buffers: 1,728,801,536 (the served BF16 GGUF also stores a 64-element RoPE buffer, 1,728,801,600); text only; card: 1.7B
Input
text, English
Endpoint
http://127.0.0.1:18634/v1/chat/completions
Started
14 Nov 2026 04:04 UTC; last answer 14 Nov 2026 04:11 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
LFM3-1.7B, served with vLLM 0.34.0rc1.dev302+gbe47189f3 (nightly, commit be47189f37f6, CUDA 13.5 image; docker vllm/vllm-openai@sha256:7f0618da95bac2f1cc97fcd1791380d7c1a06d24a4998da45aeb7fbe6c49179e), BF16 from LiquidAI/LFM3-1.7B@0dc4268b561499c648c7ba7c9466bf3d8d01517a, vLLM reasoning parser qwen3 (seed 2 of three (Umit via zeki #6969: seeds 0, 1, 2; seed 0 is lfm3-1.7b-vllm_en); vLLM nightly because no vLLM release has Lfm3ForCausalLM; the same server and settings as seed 0), one RTX PRO 6000 on our GPU host alone; thinking on (enable_thinking true, the template default; on/off only); sampling temperature 0.1, top_k 50, repetition_penalty 1.1, top_p 1, min_p 0, from generation_config.json, passed 1:1 as vLLM sampling params; seed 2; max_tokens 32,768; one question per request (32 requests in flight), standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 1.73B / 1.73B active / total (dense; BF16 safetensors 1,728,801,536, text only; card: 1.7B). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "repetition_penalty": 1.1,
  "seed": 2,
  "temperature": 0.1,
  "top_k": 50,
  "top_p": 1.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · minicpm5-2b-seed1_en

Name
MiniCPM5-2B on · chat
Setup
MiniCPM5-2B (chat, thinking, on our GPU) · chat model
Model
minicpm5-2b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.8943 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.359 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.894. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run minicpm5-2b-seed1_en · first answer 2026-11-14 06:41:40 UTC · last answer 2026-11-14 07:03:13 UTC · span 1293.0 s, pauses included (not charged) · charged 1293.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.8943 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.2 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 5 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 2.52B dense · from the run's notes (dense; checkpoint 2,516,756,480 incl. untied lm_head, text only; card: 2,516,756,480); also https://huggingface.co/openbmb/MiniCPM5-2B · read 2026-11-12 · counted from the checkpoint (2,516,756,480 incl. untied lm_head, text only; dense); the card: 2,516,756,480
Input
text, English
Endpoint
http://127.0.0.1:18614/v1/chat/completions
Started
14 Nov 2026 06:41 UTC; last answer 14 Nov 2026 07:03 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
MiniCPM5-2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from openbmb/MiniCPM5-2B@42433e90dad95123ab87729fd7011e7110629556, vLLM reasoning parser qwen3 (seed 1 of three (Umit via zeki #6979: seeds 0, 1, 2; seed 0 is minicpm5-2b_en); the same checkpoint, engine, prompt and sampling as seed 0; baseline server capacity not retained; repeats use max-num-seqs 16 (zeki #7001)), one RTX PRO 6000 on our GPU host alone; thinking on (enable_thinking true; on/off only, no levels); sampling temperature 1, top_p 0.95, from generation_config.json; seed 1; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 2.52B / 2.52B active / total (dense; checkpoint 2,516,756,480 incl. untied lm_head, text only; card: 2,516,756,480). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "seed": 1,
  "temperature": 1.0,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · minicpm5-2b-seed2_en

Name
MiniCPM5-2B on · chat
Setup
MiniCPM5-2B (chat, thinking, on our GPU) · chat model
Model
minicpm5-2b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.8591 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.345 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.859. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run minicpm5-2b-seed2_en · first answer 2026-11-14 08:36:34 UTC · last answer 2026-11-14 08:57:16 UTC · span 1242.0 s, pauses included (not charged) · charged 1242.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.8591 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.6 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 1 answer cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 2.52B dense · from the run's notes (dense; checkpoint 2,516,756,480 incl. untied lm_head, text only; card: 2,516,756,480); also https://huggingface.co/openbmb/MiniCPM5-2B · read 2026-11-12 · counted from the checkpoint (2,516,756,480 incl. untied lm_head, text only; dense); the card: 2,516,756,480
Input
text, English
Endpoint
http://127.0.0.1:18614/v1/chat/completions
Started
14 Nov 2026 08:36 UTC; last answer 14 Nov 2026 08:57 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
MiniCPM5-2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from openbmb/MiniCPM5-2B@42433e90dad95123ab87729fd7011e7110629556, vLLM reasoning parser qwen3 (seed 2 of three (Umit via zeki #6979: seeds 0, 1, 2; seed 0 is minicpm5-2b_en); the same checkpoint, engine, prompt and sampling as seed 0; baseline server capacity not retained; repeats use max-num-seqs 16 (zeki #7001)), one RTX PRO 6000 on our GPU host alone; thinking on (enable_thinking true; on/off only, no levels); sampling temperature 1, top_p 0.95, from generation_config.json; seed 2; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 2.52B / 2.52B active / total (dense; checkpoint 2,516,756,480 incl. untied lm_head, text only; card: 2,516,756,480). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "seed": 2,
  "temperature": 1.0,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · gpt-6-astra-sub_en

Name
GPT-6 Astra chat
Setup
GPT-6 Astra · chat model
Model
gpt-6-astra · provider chatgpt-subscription
GPU
—
Cost
$0 for the run · ChatGPT subscription
Answer time
run gpt-6-astra-sub_en · first answer 2026-11-14 09:58:55 UTC · last answer 2026-11-14 13:59:35 UTC · span 14440.0 s, pauses included (not charged) · charged 12365.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not on record yet
Input
text, English
Endpoint
codex-chatgpt-subscription
Started
14 Nov 2026 09:58 UTC; last answer 14 Nov 2026 13:59 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
GPT-6 Astra on the ChatGPT subscription through the Codex CLI (`codex exec`, ChatGPT sign-in from OMP's openai-codex login in a private CODEX_HOME, no refresh token, since 13 Nov 19:35 UTC; as turkce's ask_openai), model gpt-6-astra, model_reasoning_effort high (Reading's setting); no output limit sent (Codex has no setting for it) and no temperature; Codex's agent instructions replaced by a neutral one-line system prompt, web search and tool features off, read-only sandbox, no user config, rules or saved session, API-key variables removed; one question per request, one request at a time; a usage-limit reply stops the run without retrying it (resumable); cost $0 (ChatGPT subscription); api_equivalent_usd is the tokens at OpenAI's API list price {"input": 10.0, "cached_input": 1.0, "cache_write": 12.5, "output": 50.0} per 1M tokens, not a cost. Codex reports no served model id. English translations.
Parameters
{
  "disabled_features": [
    "shell_tool",
    "unified_exec",
    "goals",
    "apps",
    "plugins",
    "remote_plugin",
    "tool_suggest",
    "skill_search",
    "view_image",
    "image_generation",
    "sleep_tool",
    "browser_use",
    "browser_use_external",
    "computer_use",
    "in_app_browser",
    "multi_agent",
    "unbounded_connection_retries"
  ],
  "effort": "high",
  "list_price_per_mtok": {
    "cache_write": 12.5,
    "cached_input": 1.0,
    "input": 10.0,
    "output": 50.0
  },
  "max_output_tokens": null,
  "route": "codex exec (ChatGPT sign-in)",
  "sandbox": "read-only",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message.",
  "web_search": "disabled"
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · gemini-4-hydrogen-sub_en

Name
Gemini 4 Hydrogen chat
Setup
Gemini 4 Hydrogen (Antigravity subscription) · chat model
Model
gemini-4-hydrogen · provider Antigravity (Gemini subscription)
GPU
—
Cost
not measured
Answer time
run gemini-4-hydrogen-sub_en · first answer 2026-11-14 10:43:10 UTC · last answer 2026-11-14 14:16:38 UTC · span 12808.0 s, pauses included (not charged) · charged 10733.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: not on record yet
Input
text, English
Endpoint
http://lan-host:8899/v1/chat/completions
Started
14 Nov 2026 10:43 UTC; last answer 14 Nov 2026 14:16 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
[Turkish wording, not shown]
Parameters
{
  "max_tokens": 32768,
  "temperature": 1.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · gpt-6-astra-sub_en_dontknow

Name
GPT-6 Astra chat
Setup
GPT-6 Astra · chat model · with the I don't know option F
Model
gpt-6-astra · provider chatgpt-subscription
GPU
—
Cost
$0 for the run · ChatGPT subscription
Answer time
run gpt-6-astra-sub_en_dontknow · first answer 2026-11-14 13:59:41 UTC · last answer 2026-11-14 17:26:56 UTC · span 12435.0 s, pauses included (not charged) · charged 12435.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not on record yet
Input
text, English
Endpoint
codex-chatgpt-subscription
Started
14 Nov 2026 13:59 UTC; last answer 14 Nov 2026 17:26 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
GPT-6 Astra on the ChatGPT subscription through the Codex CLI (`codex exec`, ChatGPT sign-in from OMP's openai-codex login in a private CODEX_HOME, no refresh token, since 13 Nov 19:35 UTC; as turkce's ask_openai), model gpt-6-astra, model_reasoning_effort high (Reading's setting); no output limit sent (Codex has no setting for it) and no temperature; Codex's agent instructions replaced by a neutral one-line system prompt, web search and tool features off, read-only sandbox, no user config, rules or saved session, API-key variables removed; one question per request, one request at a time; a usage-limit reply stops the run without retrying it (resumable); cost $0 (ChatGPT subscription); api_equivalent_usd is the tokens at OpenAI's API list price {"input": 10.0, "cached_input": 1.0, "cache_write": 12.5, "output": 50.0} per 1M tokens, not a cost. Codex reports no served model id. English translations.
Parameters
{
  "disabled_features": [
    "shell_tool",
    "unified_exec",
    "goals",
    "apps",
    "plugins",
    "remote_plugin",
    "tool_suggest",
    "skill_search",
    "view_image",
    "image_generation",
    "sleep_tool",
    "browser_use",
    "browser_use_external",
    "computer_use",
    "in_app_browser",
    "multi_agent",
    "unbounded_connection_retries"
  ],
  "effort": "high",
  "list_price_per_mtok": {
    "cache_write": 12.5,
    "cached_input": 1.0,
    "input": 10.0,
    "output": 50.0
  },
  "max_output_tokens": null,
  "route": "codex exec (ChatGPT sign-in)",
  "sandbox": "read-only",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message.",
  "web_search": "disabled"
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · granite-4.2-3b-64k_en_dontknow

Name
Granite-4.2-3B high · chat
Setup
Granite-4.2-3B (chat, thinking, on our GPU) · chat model · with the I don't know option F
Model
granite-4.2-3b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.455 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.584 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.455. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run granite-4.2-3b-64k_en_dontknow · first answer 2026-11-14 14:13:15 UTC · last answer 2026-11-14 14:48:18 UTC · span 2103.0 s, pauses included (not charged) · charged 2103.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.4546 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 29.0 in flight (sum of request latencies / active time)
Output limit
65,536 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 3B dense · source https://huggingface.co/ibm-granite/granite-4.2-3b · read 2026-11-12 · the card states 3B (the maker's figure is shown); checkpoint count 3.66B (3,659,737,600 incl. untied lm_head, text only; dense)
Input
text, English
Endpoint
http://127.0.0.1:18616/v1/chat/completions
Started
14 Nov 2026 14:13 UTC; last answer 14 Nov 2026 14:48 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Granite-4.2-3B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from ibm-granite/granite-4.2-3b@4e9c275e232f10211ec88644d106a1f6a1b60e06, vLLM reasoning parser granite_thinking_parser, one RTX PRO 6000 on our GPU host alone; thinking on at full effort (enable_thinking true, reasoning_effort high; the template's levels: full or low); sampling temperature 1, top_p 0.95, from generation_config.json; seed 0; max_tokens 65,536 (raised from 32,768 once: the smoke failed on cut-offs only); one question per request (32 requests in flight), standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 3.66B / 3.66B active / total (dense; checkpoint 3,659,737,600 incl. untied lm_head, text only; card: 3B). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "reasoning_effort": "high"
  },
  "max_tokens": 65536,
  "seed": 0,
  "temperature": 1.0,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · gemini-4-hydrogen-sub_en_dontknow

Name
Gemini 4 Hydrogen chat
Setup
Gemini 4 Hydrogen (Antigravity subscription) · chat model · with the I don't know option F
Model
gemini-4-hydrogen · provider Antigravity (Gemini subscription)
GPU
—
Cost
not measured
Answer time
run gemini-4-hydrogen-sub_en_dontknow · first answer 2026-11-14 14:20:24 UTC · last answer 2026-11-14 19:23:02 UTC · span 18158.0 s, pauses included (not charged) · charged 13020.0 s in 3 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.3 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: not on record yet
Input
text, English
Endpoint
http://lan-host:8899/v1/chat/completions
Started
14 Nov 2026 14:20 UTC; last answer 14 Nov 2026 19:23 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
[Turkish wording, not shown]
Parameters
{
  "max_tokens": 32768,
  "temperature": 1.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · decider-flash-v1.3_en

Name
Perplexity Decider Flash v1.3
Setup
Perplexity Decider Flash v1.3 · decision model
Model
pplx-decider-flash-v1.3 · provider pplx-decider
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.04842 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0194 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.048. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run decider-flash-v1.3_en · first answer 2026-11-14 15:49:25 UTC · last answer 2026-11-14 15:50:35 UTC · span 70.0 s, pauses included (not charged) · charged 70.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0484 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 4B dense · source https://huggingface.co/perplexity-ai/pplx-decider-flash-v1.3 · read 2026-11-14 · 4B as v1.2 (built on Qwen3.5-4B); counted from the released files: 4.54B for decisions (backbone + readout 4,539,918,336, the same as v1.2); the Locate adapter and head (131,508,901) are not used for decisions
Input
text, English
Endpoint
http://127.0.0.1:19185
Started
14 Nov 2026 15:49 UTC; last answer 14 Nov 2026 15:50 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Perplexity Decider Flash v1.3 (site run None); runner df9ae2734; published on a DGX Spark (GB10); gpu-lanes runs it on an RTX PRO 6000
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "pplx-decider-flash-v1.3",
    "--provider",
    "pplx-decider",
    "--timeout",
    "30",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "6ad0eac51a4abb950bde74b24687209aec628aa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "pplx-decider-flash-v1.3",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "pplx-decider",
  "questions": 2163,
  "request_seconds": 61.80442487100845,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "635f983cc34cc26d64e74d4a4b7a0ba20c0f535df6d6dc10d135a888794c56fc",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 436135,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19185/v1/systemone
{
 "model": "pplx-decider-flash-v1.3",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · decider-flash-v1.3_en_dontknow

Name
Perplexity Decider Flash v1.3
Setup
Perplexity Decider Flash v1.3 · decision model · with the I don't know option F
Model
pplx-decider-flash-v1.3 · provider pplx-decider
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.05326 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0214 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.053. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run decider-flash-v1.3_en_dontknow · first answer 2026-11-14 15:51:25 UTC · last answer 2026-11-14 15:52:42 UTC · span 77.0 s, pauses included (not charged) · charged 77.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0533 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 4B dense · source https://huggingface.co/perplexity-ai/pplx-decider-flash-v1.3 · read 2026-11-14 · 4B as v1.2 (built on Qwen3.5-4B); counted from the released files: 4.54B for decisions (backbone + readout 4,539,918,336, the same as v1.2); the Locate adapter and head (131,508,901) are not used for decisions
Input
text, English
Endpoint
http://127.0.0.1:19185
Started
14 Nov 2026 15:51 UTC; last answer 14 Nov 2026 15:52 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Perplexity Decider Flash v1.3 (site run None); runner df9ae2734; published on a DGX Spark (GB10); gpu-lanes runs it on an RTX PRO 6000
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "pplx-decider-flash-v1.3",
    "--provider",
    "pplx-decider",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "6ad0eac51a4abb950bde74b24687209aec628aa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "pplx-decider-flash-v1.3",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "pplx-decider",
  "questions": 2163,
  "request_seconds": 69.97446158503226,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "8bfd0ecee623c9dad2cff74b278edf3e8cd6fe087aecbe944355328e78ce0cc2",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 501025,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19185/v1/systemone
{
 "model": "pplx-decider-flash-v1.3",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · decider-v1.3-local_en

Name
Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell
Setup
Perplexity Decider 27B v1.3 (local GPU) · decision model
Model
pplx-decider-v1.3 · provider pplx-decider
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.1425 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0572 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.142. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run decider-v1.3-local_en · first answer 2026-11-14 16:22:51 UTC · last answer 2026-11-14 16:26:17 UTC · span 206.0 s, pauses included (not charged) · charged 206.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1425 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/perplexity-ai/pplx-decider-v1.3 · read 2026-11-14 · the official model name: 27B; counted from the released files: 26.09B for decisions (backbone + readout 26,086,635,760, the same as v1.2); the Locate adapter and head (470,209,189) are not used for decisions
Input
text, English
Endpoint
http://127.0.0.1:19175
Started
14 Nov 2026 16:22 UTC; last answer 14 Nov 2026 16:26 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Perplexity Decider 27B v1.3 (local GPU) (site run None); runner df9ae2734; published on a DGX Spark (GB10); gpu-lanes runs it on an RTX PRO 6000
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "pplx-decider-v1.3",
    "--provider",
    "pplx-decider",
    "--timeout",
    "30",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "6ad0eac51a4abb950bde74b24687209aec628aa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "pplx-decider-v1.3",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "pplx-decider",
  "questions": 2163,
  "request_seconds": 198.37647967300109,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "9bf72b15e5d18cbb85ab8cef18c9e34fa7ca61d6c37786ad7aeb53296bdc0428",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 436135,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19175/v1/systemone
{
 "model": "pplx-decider-v1.3",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · decider-v1.3-local_en_dontknow

Name
Perplexity Decider 27B v1.3 NVIDIA RTX PRO 6000 Blackwell
Setup
Perplexity Decider 27B v1.3 (local GPU) · decision model · with the I don't know option F
Model
pplx-decider-v1.3 · provider pplx-decider
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.1446 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0581 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.145. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run decider-v1.3-local_en_dontknow · first answer 2026-11-14 16:28:45 UTC · last answer 2026-11-14 16:32:14 UTC · span 209.0 s, pauses included (not charged) · charged 209.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1446 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 27B dense · source https://huggingface.co/perplexity-ai/pplx-decider-v1.3 · read 2026-11-14 · the official model name: 27B; counted from the released files: 26.09B for decisions (backbone + readout 26,086,635,760, the same as v1.2); the Locate adapter and head (470,209,189) are not used for decisions
Input
text, English
Endpoint
http://127.0.0.1:19175
Started
14 Nov 2026 16:28 UTC; last answer 14 Nov 2026 16:32 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Perplexity Decider 27B v1.3 (local GPU) (site run None); runner df9ae2734; published on a DGX Spark (GB10); gpu-lanes runs it on an RTX PRO 6000
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "pplx-decider-v1.3",
    "--provider",
    "pplx-decider",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "6ad0eac51a4abb950bde74b24687209aec628aa8",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "pplx-decider-v1.3",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "pplx-decider",
  "questions": 2163,
  "request_seconds": 203.3957530650041,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "daaa40fae9bf7417261556ef7f0db04513dffa616c3d25f0d0de41e44498ace4",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 501025,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:19175/v1/systemone
{
 "model": "pplx-decider-v1.3",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · qwen3.5-4b-nothink_en

Name
Qwen3.5-4B off · chat
Setup
Qwen3.5-4B (chat, thinking off, on our GPU) · chat model
Model
qwen3.5-4b-nothink · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.6488 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.261 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.649. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-14 17:12:58–20:32:55 UTC
GPU time
run qwen3.5-4b-nothink_en · first answer 2026-11-14 17:12:36 UTC · last answer 2026-11-14 20:48:11 UTC · span 12935.0 s, pauses included (not charged) · charged 938.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.6488 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 30.8 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 4.66B dense · from the run's notes (dense; checkpoint 4,659,865,088 incl. vision 0.33B; card: 4B language model); also https://huggingface.co/Qwen/Qwen3.5-4B · read 2026-11-09 · counted from the checkpoint (4,659,865,088: language model 4.33B, vision encoder 0.33B); the card: 4B language model
Input
text, English
Endpoint
http://127.0.0.1:18642/v1/chat/completions
Started
14 Nov 2026 17:12 UTC; last answer 14 Nov 2026 20:48 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Qwen3.5-4B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from Qwen/Qwen3.5-4B@851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a, vLLM reasoning parser qwen3 (vision tower loaded, for Reading's Page images; text-only prompts here), one RTX PRO 6000 on our GPU host alone; thinking off (enable_thinking false; on/off only, no levels); sampling temperature 1, top_p 0.95, top_k 20, min_p 0, presence_penalty 1.5, repetition_penalty 1, sampling from the model card, generation_config has none (non-thinking mode for reasoning tasks; the repo has no generation_config.json); seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 4.66B / 4.66B active / total (dense; checkpoint 4,659,865,088 incl. vision 0.33B; card: 4B language model). English translations (DeepSeek + review). Paused 17:15 UTC 14 Nov so Gemma 4 E4B/E2B thinking on at 32k run first (#7721); resumed from stored answers afterwards (in-flight requests re-asked).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "presence_penalty": 1.5,
  "repetition_penalty": 1.0,
  "seed": 0,
  "temperature": 1.0,
  "top_k": 20,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · gemma-4-e4b-it-think-32k_en

Name
Gemma 4 E4B on · chat · 32k cap
Setup
Gemma 4 E4B (chat, thinking on, 32k, on our GPU) · chat model
Model
gemma-4-e4b-it-think-32k · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$1.121 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.45 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $1.121. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-14 17:26:48–18:07:26 UTC
GPU time
run gemma-4-e4b-it-think-32k_en · first answer 2026-11-14 17:25:49 UTC · last answer 2026-11-14 18:33:27 UTC · span 4058.0 s, pauses included (not charged) · charged 1620.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $1.1205 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 30.9 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 8B dense · 4.5B effective · source https://huggingface.co/google/gemma-4-E4B-it · read 2026-11-10 · the card states 4.5B effective (8B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 8.00B (counted from the checkpoint (7,996,157,418: language model 7.52B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 8B with embeddings, 4.5B effective; not a mixture of experts)
Input
text, English
Endpoint
http://127.0.0.1:18639/v1/chat/completions
Started
14 Nov 2026 17:25 UTC; last answer 14 Nov 2026 18:33 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Gemma 4 E4B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from google/gemma-4-E4B-it@4ecc2ba63bcac23ba4a767fc87025b1450f84c9c, vLLM reasoning parser gemma4, one RTX PRO 6000 on our GPU host alone; thinking on (chat_template_kwargs enable_thinking true, the template's own switch: it opens a system turn with <|think|>; the template default is off; on/off only, no levels); sampling temperature 0, as the Gemma 4 31B chat run; no seed (as the Gemma 4 31B run); max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 8.00B / 8.00B active / total (dense; checkpoint 7,996,157,418 incl. vision 0.17B and audio 0.31B towers; card: 8B with per-layer embeddings, 4.5B effective). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · gemma-4-e2b-it-think-32k_en

Name
Gemma 4 E2B on · chat · 32k cap
Setup
Gemma 4 E2B (chat, thinking on, 32k, on our GPU) · chat model
Model
gemma-4-e2b-it-think-32k · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.886 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.356 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.886. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-14 18:54:41–19:28:35 UTC
GPU time
run gemma-4-e2b-it-think-32k_en · first answer 2026-11-14 18:54:13 UTC · last answer 2026-11-14 19:49:28 UTC · span 3315.0 s, pauses included (not charged) · charged 1281.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.8860 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 29.3 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 6 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 5.1B dense · 2.3B effective · source https://huggingface.co/google/gemma-4-E2B-it · read 2026-11-10 · the card states 2.3B effective (5.1B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 5.12B (counted from the checkpoint (5,123,178,979: language model 4.65B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 5.1B with embeddings, 2.3B effective; not a mixture of experts)
Input
text, English
Endpoint
http://127.0.0.1:18638/v1/chat/completions
Started
14 Nov 2026 18:54 UTC; last answer 14 Nov 2026 19:49 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Gemma 4 E2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from google/gemma-4-E2B-it@4adf15748de9e1a0097b6274cee9c15836c41c37, vLLM reasoning parser gemma4, one RTX PRO 6000 on our GPU host alone; thinking on (chat_template_kwargs enable_thinking true, the template's own switch: it opens a system turn with <|think|>; the template default is off; on/off only, no levels); sampling temperature 0, as the Gemma 4 31B chat run; no seed (as the Gemma 4 31B run); max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 5.12B / 5.12B active / total (dense; checkpoint 5,123,178,979 incl. vision 0.17B and audio 0.31B towers; card: 5.1B with per-layer embeddings, 2.3B effective). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "temperature": 0.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · qwen3.5-4b-nothink_en_dontknow

Name
Qwen3.5-4B off · chat
Setup
Qwen3.5-4B (chat, thinking off, on our GPU) · chat model · with the I don't know option F
Model
qwen3.5-4b-nothink · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.6004 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.241 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.600. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run qwen3.5-4b-nothink_en_dontknow · first answer 2026-11-14 21:29:47 UTC · last answer 2026-11-14 21:44:15 UTC · span 868.0 s, pauses included (not charged) · charged 868.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.6004 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.9 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 4.66B dense · from the run's notes (dense; checkpoint 4,659,865,088 incl. vision 0.33B; card: 4B language model); also https://huggingface.co/Qwen/Qwen3.5-4B · read 2026-11-09 · counted from the checkpoint (4,659,865,088: language model 4.33B, vision encoder 0.33B); the card: 4B language model
Input
text, English
Endpoint
http://127.0.0.1:18642/v1/chat/completions
Started
14 Nov 2026 21:29 UTC; last answer 14 Nov 2026 21:44 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Qwen3.5-4B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from Qwen/Qwen3.5-4B@851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a, vLLM reasoning parser qwen3 (vision tower loaded, for Reading's Page images; text-only prompts here), one RTX PRO 6000 on our GPU host alone; thinking off (enable_thinking false; on/off only, no levels); sampling temperature 1, top_p 0.95, top_k 20, min_p 0, presence_penalty 1.5, repetition_penalty 1, sampling from the model card, generation_config has none (non-thinking mode for reasoning tasks; the repo has no generation_config.json); seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 4.66B / 4.66B active / total (dense; checkpoint 4,659,865,088 incl. vision 0.33B; card: 4B language model). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "presence_penalty": 1.5,
  "repetition_penalty": 1.0,
  "seed": 0,
  "temperature": 1.0,
  "top_k": 20,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · gpt-6.1-sol-sub_en

Name
GPT-6.1 Sol chat
Setup
GPT-6.1 Sol · chat model
Model
gpt-6.1-sol · provider chatgpt-subscription
GPU
—
Cost
$0 for the run · ChatGPT subscription
Answer time
run gpt-6.1-sol-sub_en · first answer 2026-11-14 23:17:01 UTC · last answer 2026-11-15 02:24:57 UTC · span 11276.0 s, pauses included (not charged) · charged 11276.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not on record yet
Input
text, English
Endpoint
codex-chatgpt-subscription
Started
14 Nov 2026 23:16 UTC; last answer 15 Nov 2026 02:24 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
GPT-6.1 Sol on the ChatGPT subscription through the Codex CLI (`codex exec`, ChatGPT sign-in from OMP's openai-codex login in a private CODEX_HOME, no refresh token, since 13 Nov 19:35 UTC; as turkce's ask_openai), model gpt-6.1-sol, model_reasoning_effort high (Reading's setting); no output limit sent (Codex has no setting for it) and no temperature; Codex's agent instructions replaced by a neutral one-line system prompt, web search and tool features off, read-only sandbox, no user config, rules or saved session, API-key variables removed; one question per request, one request at a time; a usage-limit reply stops the run without retrying it (resumable); cost $0 (ChatGPT subscription); api_equivalent_usd is the tokens at OpenAI's API list price {"input": 2.0, "cached_input": 0.1, "cache_write": 2.5, "output": 10.0} per 1M tokens, not a cost. Codex reports no served model id. English translations.
Parameters
{
  "disabled_features": [
    "shell_tool",
    "unified_exec",
    "goals",
    "apps",
    "plugins",
    "remote_plugin",
    "tool_suggest",
    "skill_search",
    "view_image",
    "image_generation",
    "sleep_tool",
    "browser_use",
    "browser_use_external",
    "computer_use",
    "in_app_browser",
    "multi_agent",
    "unbounded_connection_retries"
  ],
  "effort": "high",
  "list_price_per_mtok": {
    "cache_write": 2.5,
    "cached_input": 0.1,
    "input": 2.0,
    "output": 10.0
  },
  "max_output_tokens": null,
  "route": "codex exec (ChatGPT sign-in)",
  "sandbox": "read-only",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message.",
  "web_search": "disabled"
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · gemini-4-carbon-sub_en_dontknow

Name
Gemini 4 Carbon chat
Setup
Gemini 4 Carbon (Antigravity subscription) · chat model · with the I don't know option F
Model
gemini-4-carbon · provider Antigravity (Gemini subscription)
GPU
—
Cost
not measured
Answer time
run gemini-4-carbon-sub_en_dontknow · first answer 2026-11-15 01:42:11 UTC · last answer 2026-11-15 07:59:45 UTC · span 22654.0 s, pauses included (not charged) · charged 22654.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 2.0 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 2 empty replies asked once more
Model size
Parameters: not disclosed by maker (closed weights) · source https://ai.google.dev/gemini-api/docs/models · read 2026-11-11
Input
text, English
Endpoint
http://lan-host:8899/v1/chat/completions
Started
15 Nov 2026 01:41 UTC; last answer 15 Nov 2026 07:59 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Gemini 4 Carbon (gemini-4-carbon = gemini-4-carbon-high) through Can's Antigravity subscription (antigravity-benchmark #4762; Umit via zeki #5133): tools denied, ~10.7k tokens of Antigravity's own context per request, one request at a time; temperature 1.0 and max_tokens 32,768 sent but ignored by Antigravity (native sampling, output cap 262,144). A separate setup from the paid API run gemini-4-carbon_en.
Parameters
{
  "max_tokens": 32768,
  "temperature": 1.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · raon-1-nano-nothink_en

Name
Raon-1 nano off · chat
Setup
Raon-1 nano (chat, thinking off, KRAFTON vLLM fork, on our GPU) · chat model
Model
raon-1-nano-nothink · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$4.111 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 1.65 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $4.111. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-15 02:06:04–16:15:21 UTC†
GPU time
run raon-1-nano-nothink_en · first answer 2026-11-15 02:04:31 UTC · last answer 2026-11-16 17:52:52 UTC · span 143301.0 s, pauses included (not charged) · charged 5944.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $4.1113 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)†
Requests in flight
≈ 28.3 in flight (sum of request latencies / active time)†
Output limit
32,768 tokens · 60 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 3.1B dense · 1.5B effective · source https://huggingface.co/KRAFTON/Raon-1-nano · read 2026-11-13 · the card states 1.5B effective parameters (3.1B total, including a 1.5B per-layer embedding table) (the maker's figure is shown); checkpoint count 3.13B (3,130,797,328: per-layer embedding table 1.60B, token embeddings 0.40B tied, layers 1.12B; plus a materialised copy of the tied lm_head 0.40B and the MTP module 0.05B in separate files; dense)
Input
text, English
Endpoint
http://127.0.0.1:18637/v1/chat/completions
Started
15 Nov 2026 02:04 UTC; last answer 16 Nov 2026 17:52 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Raon-1 nano, served with KRAFTON's vLLM fork krafton-ai/vllm@13a5dd87 (branch raon-1-nano, precompiled; not a vLLM release) from KRAFTON/Raon-1-nano@fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee, vLLM reasoning parser raon_1 (the card lists Korean and English only; cost and speed are on another engine than the stock-vLLM rows), one RTX PRO 6000 on our GPU host alone; thinking off (enable_thinking false; the card: 'designed for thinking mode; disabling it is not recommended'); sampling temperature 0.1, presence_penalty 1.5, sampling from the model card, generation_config has none ('General' row); seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 1.5B / 3.1B active / total (card: 1.5B effective, 3.1B total incl. a 1.5B per-layer embedding table; MTP module 55M not loaded). English translations (DeepSeek + review). SMOKE FAILED (gate, 2026-11-15T02:06Z): 7 of 50 bad: 3 cut off, 0 empty, 4 no letter, 0 failed; full runs asked anyway at the same settings (bus #6515: both modes at 32,768, published whichever way they come out; zeki #8211); the smoke answers are kept as these runs' first answers. The full runs wait in the GPU 1 queue (zeki #8340: shown as not running, not as failed). SMOKE GATE FAILED (2026-11-16T12:10Z): 7 of 50 bad: 3 cut off, 0 empty, 4 no letter, 0 failed; full runs asked anyway at the same settings (bus #6515: both modes at 32,768, published whichever way they come out; zeki #8211); the smoke answers are kept as these runs' first answers.
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "truncate_history_thinking": false
  },
  "max_tokens": 32768,
  "presence_penalty": 1.5,
  "seed": 0,
  "temperature": 0.1
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · gpt-6.1-sol-sub_en_dontknow

Name
GPT-6.1 Sol chat
Setup
GPT-6.1 Sol · chat model · with the I don't know option F
Model
gpt-6.1-sol · provider chatgpt-subscription
GPU
—
Cost
$0 for the run · ChatGPT subscription
Answer time
run gpt-6.1-sol-sub_en_dontknow · first answer 2026-11-15 02:25:03 UTC · last answer 2026-11-15 05:28:59 UTC · span 11036.0 s, pauses included (not charged) · charged 11036.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not on record yet
Input
text, English
Endpoint
codex-chatgpt-subscription
Started
15 Nov 2026 02:24 UTC; last answer 15 Nov 2026 05:28 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
GPT-6.1 Sol on the ChatGPT subscription through the Codex CLI (`codex exec`, ChatGPT sign-in from OMP's openai-codex login in a private CODEX_HOME, no refresh token, since 13 Nov 19:35 UTC; as turkce's ask_openai), model gpt-6.1-sol, model_reasoning_effort high (Reading's setting); no output limit sent (Codex has no setting for it) and no temperature; Codex's agent instructions replaced by a neutral one-line system prompt, web search and tool features off, read-only sandbox, no user config, rules or saved session, API-key variables removed; one question per request, one request at a time; a usage-limit reply stops the run without retrying it (resumable); cost $0 (ChatGPT subscription); api_equivalent_usd is the tokens at OpenAI's API list price {"input": 2.0, "cached_input": 0.1, "cache_write": 2.5, "output": 10.0} per 1M tokens, not a cost. Codex reports no served model id. English translations.
Parameters
{
  "disabled_features": [
    "shell_tool",
    "unified_exec",
    "goals",
    "apps",
    "plugins",
    "remote_plugin",
    "tool_suggest",
    "skill_search",
    "view_image",
    "image_generation",
    "sleep_tool",
    "browser_use",
    "browser_use_external",
    "computer_use",
    "in_app_browser",
    "multi_agent",
    "unbounded_connection_retries"
  ],
  "effort": "high",
  "list_price_per_mtok": {
    "cache_write": 2.5,
    "cached_input": 0.1,
    "input": 2.0,
    "output": 10.0
  },
  "max_output_tokens": null,
  "route": "codex exec (ChatGPT sign-in)",
  "sandbox": "read-only",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message.",
  "web_search": "disabled"
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · raon-1-nano_en

Name
Raon-1 nano on · chat
Setup
Raon-1 nano (chat, thinking on, KRAFTON vLLM fork, on our GPU) · chat model
Model
raon-1-nano · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$3.613 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 1.45 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $3.613. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-15 02:47:31–14:28:55 UTC†
GPU time
run raon-1-nano_en · first answer 2026-11-15 02:46:41 UTC · last answer 2026-11-15 15:55:08 UTC · span 47307.0 s, pauses included (not charged) · charged 5223.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $3.6126 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)†
Requests in flight
≈ 29.0 in flight (sum of request latencies / active time)†
Output limit
32,768 tokens · 35 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 3.1B dense · 1.5B effective · source https://huggingface.co/KRAFTON/Raon-1-nano · read 2026-11-13 · the card states 1.5B effective parameters (3.1B total, including a 1.5B per-layer embedding table) (the maker's figure is shown); checkpoint count 3.13B (3,130,797,328: per-layer embedding table 1.60B, token embeddings 0.40B tied, layers 1.12B; plus a materialised copy of the tied lm_head 0.40B and the MTP module 0.05B in separate files; dense)
Input
text, English
Endpoint
http://127.0.0.1:18636/v1/chat/completions
Started
15 Nov 2026 02:46 UTC; last answer 15 Nov 2026 15:55 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; paused at 2,163 of 2,163 since 2026-11-15T04:56Z (compactor #8247, zeki #8248); it resumes from its stored answers
Notes
Raon-1 nano, served with KRAFTON's vLLM fork krafton-ai/vllm@13a5dd87 (branch raon-1-nano, precompiled; not a vLLM release) from KRAFTON/Raon-1-nano@fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee, vLLM reasoning parser raon_1 (the card lists Korean and English only; cost and speed are on another engine than the stock-vLLM rows), one RTX PRO 6000 on our GPU host alone; thinking on (enable_thinking true, the template default; on/off only, no levels); sampling temperature 0.1, presence_penalty 1.5, sampling from the model card, generation_config has none ('General' row); seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 1.5B / 3.1B active / total (card: 1.5B effective, 3.1B total incl. a 1.5B per-layer embedding table; MTP module 55M not loaded). English translations (DeepSeek + review). SMOKE GATE FAILED (2026-11-15T02:50Z): 9 of 50 bad: 9 cut off, 0 empty, 0 no letter, 0 failed; full runs asked anyway at the same settings (bus #6515: both modes at 32,768, published whichever way they come out; zeki #8211); the smoke answers are kept as these runs' first answers. PAUSED 2026-11-15T04:56:37Z (compactor #8247, zeki #8248): stopped on GPU 0 of our GPU host to free the card; 32 requests were in flight and not stored (the runner has no drain), so they are asked again as fresh requests on resume. Resumes from the stored answers at the same 32,768 settings and seed on its GPU 1 queue entry: a second segment on another card of the same model, not proof of an uninterrupted identical runtime; the pause is not counted in GPU time. PAUSE ACCOUNTING (zeki #8256): runner terminated 2026-11-15T04:56:45Z, server stopped 04:56:48Z; 32 requests in flight at 04:56:45 (read from the server's metrics just before the stop), none stored. The site's GPU time runs from first to last stored answer per stretch, so the active tail from the last stored answer of this benchmark (2026-11-15T02:47:31+00:00) to the termination is not in it: the GPU time of this segment is a lower bound by that tail. On resume only the missing questions are asked, as fresh requests after the interruption. RESUMED 2026-11-15T09:48Z on GPU 0 of our GPU host (booking r_93911c3b1645; compactor #8413/#8415, Umit #8343 item 3), not on the GPU 1 queue entry the pause note names; same 32,768 settings and seed, same KRAFTON vLLM fork; only the missing questions are asked.
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true,
    "truncate_history_thinking": false
  },
  "max_tokens": 32768,
  "presence_penalty": 1.5,
  "seed": 0,
  "temperature": 0.1
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · karpo-decision-31b_en

Name
Karpo Decision 31B
Setup
karpo-decision-31B · decision model
Model
karpo-decision · provider karpo
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.2435 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0978 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.243. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run karpo-decision-31b_en · first answer 2026-11-15 05:03:45 UTC · last answer 2026-11-15 05:09:37 UTC · span 352.0 s, pauses included (not charged) · charged 352.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.2435 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 31B dense · source https://huggingface.co/MachinePulseAI/karpo-decision-31B · read 2026-11-15 · the maker's figure from its name and card title (31B) is shown, as for every model (turkce #4263); not counted from the checkpoint
Input
text, English
Endpoint
http://127.0.0.1:18440
Started
15 Nov 2026 05:03 UTC; last answer 15 Nov 2026 05:09 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup karpo-decision-31B (site run None); runner 815031979; MachinePulseAI/karpo-decision-31B@d69cdd57eab23271cbe65236ca8a00d80d7bb8c7, served by compactor (karpo_decision.serve.server) on GPU 0 of our GPU host for Answer Check and Reading; Logic & Math on the same load (Umit #8157, compactor #8167) SHARED LOAD (turkce #8262, zeki #8260): compactor's karpo-decision-31B server on GPU 0 of our GPU host also answered turkce's Reading runs from 04:58 UTC, one request at a time, while these runs ran (05:00-05:20 UTC), so latency, throughput and GPU-time cost were measured on a shared server, not an isolated load; the answers are unaffected.
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "karpo-decision",
    "--provider",
    "karpo",
    "--timeout",
    "120",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "815031979442b7cd01229dcf844de47d47478ac9",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "karpo-decision",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "karpo",
  "questions": 2163,
  "request_seconds": 341.5160172368851,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "49ec4645fb53518742269f977de72f27db82a095a8c2a83b027fd5c4251a1a78",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 587127,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:18440/v1/systemone
{
 "model": "karpo-decision",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · karpo-decision-31b_en_dontknow

Name
Karpo Decision 31B
Setup
karpo-decision-31B · decision model · with the I don't know option F
Model
karpo-decision · provider karpo
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.2739 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.11 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.274. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run karpo-decision-31b_en_dontknow · first answer 2026-11-15 05:13:41 UTC · last answer 2026-11-15 05:20:17 UTC · span 396.0 s, pauses included (not charged) · charged 396.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.2739 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 31B dense · source https://huggingface.co/MachinePulseAI/karpo-decision-31B · read 2026-11-15 · the maker's figure from its name and card title (31B) is shown, as for every model (turkce #4263); not counted from the checkpoint
Input
text, English
Endpoint
http://127.0.0.1:18440
Started
15 Nov 2026 05:13 UTC; last answer 15 Nov 2026 05:20 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup karpo-decision-31B (site run None); runner 815031979; MachinePulseAI/karpo-decision-31B@d69cdd57eab23271cbe65236ca8a00d80d7bb8c7, served by compactor (karpo_decision.serve.server) on GPU 0 of our GPU host for Answer Check and Reading; Logic & Math on the same load (Umit #8157, compactor #8167) SHARED LOAD (turkce #8262, zeki #8260): compactor's karpo-decision-31B server on GPU 0 of our GPU host also answered turkce's Reading runs from 04:58 UTC, one request at a time, while these runs ran (05:00-05:20 UTC), so latency, throughput and GPU-time cost were measured on a shared server, not an isolated load; the answers are unaffected.
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "karpo-decision",
    "--provider",
    "karpo",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "815031979442b7cd01229dcf844de47d47478ac9",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "karpo-decision",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "karpo",
  "questions": 2163,
  "request_seconds": 388.83459482390026,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "106990d77c0231a14b81694811427887c8ca92ed154d5f4f1f799009bf7eae6a",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 695277,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:18440/v1/systemone
{
 "model": "karpo-decision",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · copernicus-31b_en

Name
Copernicus-31B
Setup
Copernicus-31B · decision model
Model
Copernicus-31B · provider copernicus
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.1826 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0733 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.183. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run copernicus-31b_en · first answer 2026-11-15 05:28:46 UTC · last answer 2026-11-15 05:33:10 UTC · span 264.0 s, pauses included (not charged) · charged 264.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1826 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 31B dense · source https://huggingface.co/Poindexter-Labs/Copernicus-31B · read 2026-11-15 · the maker's stated figure (31 billion parameters, the card) is shown, as for every model (turkce #4263); not counted from the checkpoint
Input
text, English
Endpoint
http://127.0.0.1:18441
Started
15 Nov 2026 05:28 UTC; last answer 15 Nov 2026 05:33 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Copernicus-31B (site run None); runner 815031979; Poindexter-Labs/Copernicus-31B@be715259e97a743341f2d7a3e6a6298dab23ca0e, served by compactor (copernicus_server.py --backend hf) on GPU 0 of our GPU host for Answer Check and Reading; Logic & Math on the same load (Umit #8157, compactor #8167)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "Copernicus-31B",
    "--provider",
    "copernicus",
    "--timeout",
    "120",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "815031979442b7cd01229dcf844de47d47478ac9",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "Copernicus-31B",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "copernicus",
  "questions": 2163,
  "request_seconds": 260.1994835900405,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "da7d5aa7902ec53286594a68855f0c5b860ca481b0b223cdeacd8f0f217dc265",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 925509,
    "output_tokens": 2163
  }
}
Prompt template
POST http://127.0.0.1:18441/v1/systemone
{
 "model": "Copernicus-31B",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · copernicus-31b_en_dontknow

Name
Copernicus-31B
Setup
Copernicus-31B · decision model · with the I don't know option F
Model
Copernicus-31B · provider copernicus
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.2047 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0822 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.205. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run copernicus-31b_en_dontknow · first answer 2026-11-15 05:37:25 UTC · last answer 2026-11-15 05:42:21 UTC · span 296.0 s, pauses included (not charged) · charged 296.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.2047 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 31B dense · source https://huggingface.co/Poindexter-Labs/Copernicus-31B · read 2026-11-15 · the maker's stated figure (31 billion parameters, the card) is shown, as for every model (turkce #4263); not counted from the checkpoint
Input
text, English
Endpoint
http://127.0.0.1:18441
Started
15 Nov 2026 05:37 UTC; last answer 15 Nov 2026 05:42 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Copernicus-31B (site run None); runner 815031979; Poindexter-Labs/Copernicus-31B@be715259e97a743341f2d7a3e6a6298dab23ca0e, served by compactor (copernicus_server.py --backend hf) on GPU 0 of our GPU host for Answer Check and Reading; Logic & Math on the same load (Umit #8157, compactor #8167)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "Copernicus-31B",
    "--provider",
    "copernicus",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "815031979442b7cd01229dcf844de47d47478ac9",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "Copernicus-31B",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "copernicus",
  "questions": 2163,
  "request_seconds": 291.937169674864,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "ca8df1c573aba4f9420fa54ca3454d9a86ffe4eda042f34bb2d7d7652973146d",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 1068267,
    "output_tokens": 2163
  }
}
Prompt template
POST http://127.0.0.1:18441/v1/systemone
{
 "model": "Copernicus-31B",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · qwen3.5-2b-nothink_en

Name
Qwen3.5-2B off · chat
Setup
Qwen3.5-2B (chat, thinking off, on our GPU) · chat model
Model
qwen3.5-2b-nothink · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.7857 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.316 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.786. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-15 06:20:10–07:06:48 UTC
GPU time
run qwen3.5-2b-nothink_en · first answer 2026-11-15 06:19:30 UTC · last answer 2026-11-15 07:25:04 UTC · span 3934.0 s, pauses included (not charged) · charged 1136.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.7857 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 30.5 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 2B dense · source https://huggingface.co/Qwen/Qwen3.5-2B · read 2026-11-13 · the card states Number of Parameters: 2B (the maker's figure is shown); checkpoint count 2.27B (2,274,069,824: language model 1.88B incl. tied embeddings, vision encoder 0.33B, MTP head 0.06B; dense)
Input
text, English
Endpoint
http://127.0.0.1:18643/v1/chat/completions
Started
15 Nov 2026 06:19 UTC; last answer 15 Nov 2026 07:25 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Qwen3.5-2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from Qwen/Qwen3.5-2B@15852e8c16360a2fea060d615a32b45270f8a8fc, vLLM reasoning parser qwen3 (vision tower loaded, for Reading's Page images; text-only prompts here), one RTX PRO 6000 on our GPU host alone; thinking off (enable_thinking false, the template default for this size; on/off only, no levels); sampling temperature 1, top_p 0.95, top_k 20, min_p 0, presence_penalty 1.5, repetition_penalty 1, sampling from the model card, generation_config has none (non-thinking mode for reasoning tasks; the repo has no generation_config.json); seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 2.27B / 2.27B active / total (dense; checkpoint 2,274,069,824: language model 1.88B incl. tied embeddings, vision encoder 0.33B, MTP head 0.06B (not loaded); card: 2B). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "presence_penalty": 1.5,
  "repetition_penalty": 1.0,
  "seed": 0,
  "temperature": 1.0,
  "top_k": 20,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · knowline-9b_en

Name
KnowLine-9B-Gen1
Setup
KnowLine-9B-Gen1 · decision model
Model
PelaAI/KnowLine-9B-Gen1 · provider knowline
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.05741 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0231 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.057. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run knowline-9b_en · first answer 2026-11-15 06:20:21 UTC · last answer 2026-11-15 06:21:44 UTC · span 83.0 s, pauses included (not charged) · charged 83.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0574 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/PelaAI/KnowLine-9B-Gen1 · read 2026-11-15 · the maker's figure from its name (9B) is shown, as for every model (turkce #4263); built on Qwen3.5-9B (9.65B counted in its own checkpoint); this checkpoint not counted
Input
text, English
Endpoint
http://127.0.0.1:18450
Started
15 Nov 2026 06:20 UTC; last answer 15 Nov 2026 06:21 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup KnowLine-9B-Gen1 (site run None); runner 815031979; PelaAI/KnowLine-9B-Gen1@325378d128e9f5eb5088196668f5874c5682b6c0, served by compactor on GPU 0 of our GPU host with the card's serve_knowline.sh (SGLang, then knowline_server.py, card defaults, read twice up to 3,000 tokens) in the lmsysorg/sglang image with SGLang 0.5.23; the card's release launcher names SGLang 0.5.21, not available here (compactor #8276). Logic & Math on the same load as Answer Check and Reading (Umit #8157, compactor #8167)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "PelaAI/KnowLine-9B-Gen1",
    "--provider",
    "knowline",
    "--timeout",
    "120",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "815031979442b7cd01229dcf844de47d47478ac9",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "PelaAI/KnowLine-9B-Gen1",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "knowline",
  "questions": 2163,
  "request_seconds": 80.25736258910183,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "ac41401f3c642c4b370e04f1f020f8a7d8d7c6e1ae25e642c89f5c7d9de7a1f5",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 924182,
    "output_tokens": 2163
  }
}
Prompt template
POST http://127.0.0.1:18450/v1/systemone
{
 "model": "PelaAI/KnowLine-9B-Gen1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · knowline-9b_en_dontknow

Name
KnowLine-9B-Gen1
Setup
KnowLine-9B-Gen1 · decision model · with the I don't know option F
Model
PelaAI/KnowLine-9B-Gen1 · provider knowline
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.06778 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0272 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.068. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run knowline-9b_en_dontknow · first answer 2026-11-15 06:22:49 UTC · last answer 2026-11-15 06:24:27 UTC · span 98.0 s, pauses included (not charged) · charged 98.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0678 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/PelaAI/KnowLine-9B-Gen1 · read 2026-11-15 · the maker's figure from its name (9B) is shown, as for every model (turkce #4263); built on Qwen3.5-9B (9.65B counted in its own checkpoint); this checkpoint not counted
Input
text, English
Endpoint
http://127.0.0.1:18450
Started
15 Nov 2026 06:22 UTC; last answer 15 Nov 2026 06:24 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup KnowLine-9B-Gen1 (site run None); runner 815031979; PelaAI/KnowLine-9B-Gen1@325378d128e9f5eb5088196668f5874c5682b6c0, served by compactor on GPU 0 of our GPU host with the card's serve_knowline.sh (SGLang, then knowline_server.py, card defaults, read twice up to 3,000 tokens) in the lmsysorg/sglang image with SGLang 0.5.23; the card's release launcher names SGLang 0.5.21, not available here (compactor #8276). Logic & Math on the same load as Answer Check and Reading (Umit #8157, compactor #8167)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "PelaAI/KnowLine-9B-Gen1",
    "--provider",
    "knowline",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "815031979442b7cd01229dcf844de47d47478ac9",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "PelaAI/KnowLine-9B-Gen1",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "knowline",
  "questions": 2163,
  "request_seconds": 93.69468663627777,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "e44bba014396687d6fe450b556445e3694ce4e8fb35d9e47e7f7a710866de273",
  "runner": "benchmark_ollama.py",
  "runner_commit": "df9ae2734d780a9a15ba80aea4ddfc6185c0b8f1",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 1053962,
    "output_tokens": 2163
  }
}
Prompt template
POST http://127.0.0.1:18450/v1/systemone
{
 "model": "PelaAI/KnowLine-9B-Gen1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · adjudex-9b_en

Name
Adjudex 9B v5
Setup
Adjudex 9B v5 · decision model
Model
jev-latest · provider pplx-decider
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.05741 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0231 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.057. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run adjudex-9b_en · first answer 2026-11-15 06:49:36 UTC · last answer 2026-11-15 06:50:59 UTC · span 83.0 s, pauses included (not charged) · charged 83.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0574 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/newsbang/adjudex-9b-v5 · read 2026-11-15 · the maker's figure from its name (9B) is shown, as for every model (turkce #4263); a fine-tune of Qwen3.5-9B; this checkpoint not counted
Input
text, English
Endpoint
http://127.0.0.1:18451
Started
15 Nov 2026 06:49 UTC; last answer 15 Nov 2026 06:50 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Adjudex 9B v5 (site run None); runner 815031979; newsbang/adjudex-9b-v5@ffdcb8254d1b22e40b2bbbf569f143e0b357aeca (Apache-2.0, a fine-tune of Qwen3.5-9B), served by compactor on GPU 0 of our GPU host with the card's own autojev server (the Perplexity Decider v1.1 source, unchanged: python -m autojev.server in the v1.1 environment, whose requirements are identical to the card's; default settings, max length 8,192); the Perplexity Decider request format (runner provider pplx-decider), model field 'jev-latest' (compactor #8298). Logic & Math on the same load as Answer Check and Reading (Umit #8157, compactor #8167)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "jev-latest",
    "--provider",
    "pplx-decider",
    "--timeout",
    "60",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "815031979442b7cd01229dcf844de47d47478ac9",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "jev-latest",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "pplx-decider",
  "questions": 2163,
  "request_seconds": 74.23565503106511,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "2e015e75eec690b184d1cdaa28a38694e22d966e47d02b065b7af98c8eb1afd1",
  "runner": "benchmark_ollama.py",
  "runner_commit": "815031979442b7cd01229dcf844de47d47478ac9",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 470743,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:18451/v1/systemone
{
 "model": "jev-latest",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · adjudex-9b_en_dontknow

Name
Adjudex 9B v5
Setup
Adjudex 9B v5 · decision model · with the I don't know option F
Model
jev-latest · provider pplx-decider
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.05118 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0206 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.051. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run adjudex-9b_en_dontknow · first answer 2026-11-15 06:51:59 UTC · last answer 2026-11-15 06:53:13 UTC · span 74.0 s, pauses included (not charged) · charged 74.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0512 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/newsbang/adjudex-9b-v5 · read 2026-11-15 · the maker's figure from its name (9B) is shown, as for every model (turkce #4263); a fine-tune of Qwen3.5-9B; this checkpoint not counted
Input
text, English
Endpoint
http://127.0.0.1:18451
Started
15 Nov 2026 06:51 UTC; last answer 15 Nov 2026 06:53 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Adjudex 9B v5 (site run None); runner 815031979; newsbang/adjudex-9b-v5@ffdcb8254d1b22e40b2bbbf569f143e0b357aeca (Apache-2.0, a fine-tune of Qwen3.5-9B), served by compactor on GPU 0 of our GPU host with the card's own autojev server (the Perplexity Decider v1.1 source, unchanged: python -m autojev.server in the v1.1 environment, whose requirements are identical to the card's; default settings, max length 8,192); the Perplexity Decider request format (runner provider pplx-decider), model field 'jev-latest' (compactor #8298). Logic & Math on the same load as Answer Check and Reading (Umit #8157, compactor #8167)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "jev-latest",
    "--provider",
    "pplx-decider",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "815031979442b7cd01229dcf844de47d47478ac9",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "jev-latest",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "pplx-decider",
  "questions": 2163,
  "request_seconds": 66.8572232171864,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "8508813875d22ab1a4a3b4e6a5c42f13b93c2187f42105375425bc2f0e51845f",
  "runner": "benchmark_ollama.py",
  "runner_commit": "815031979442b7cd01229dcf844de47d47478ac9",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 535633,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:18451/v1/systemone
{
 "model": "jev-latest",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · jeffan-9b_en

Name
Jeffan-9B-1211
Setup
Jeffan-9B-1211 · decision model
Model
Jeffan-9B-1211 · provider di-serve
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.06225 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.025 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.062. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run jeffan-9b_en · first answer 2026-11-15 07:10:55 UTC · last answer 2026-11-15 07:12:25 UTC · span 90.0 s, pauses included (not charged) · charged 90.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0623 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/tattrongvu/Jeffan-9B-1211 · read 2026-11-15 · the maker's figure from its name (9B) is shown, as for every model (turkce #4263); the card's table states 8.95B; a fine-tune of Qwen3.5-9B whose middle layers run twice per pass (no extra parameters)
Input
text, English
Endpoint
http://127.0.0.1:18453
Started
15 Nov 2026 07:10 UTC; last answer 15 Nov 2026 07:12 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Jeffan-9B-1211 (site run None); runner a75966fe5; tattrongvu/Jeffan-9B-1211@862d1a7a8b64a8d708ddb0b0cd775a740f01841a (Apache-2.0, a fine-tune of Qwen3.5-9B), served by compactor on GPU 0 of our GPU host with his di_serve.py, a thin System One wrapper around the card's own jeffan_engine:DecisionEngine loaded as the Decision Index kit does (decision-index@f8552a1; options max_len and budget 65,536, as the card; transformers 5.17.0, flash-linear-attention 0.5.2, without the optional causal-conv1d, so its PyTorch fallback); returns the engine's own answers, one request at a time (compactor #8319). Runner provider di-serve, which does not send a model field. Logic & Math on the same load as Answer Check and Reading (Umit #8157, compactor #8167)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "Jeffan-9B-1211",
    "--provider",
    "di-serve",
    "--timeout",
    "120",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "Jeffan-9B-1211",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "di-serve",
  "questions": 2163,
  "request_seconds": 84.131021940957,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "8f06aabe66fd884d52eec3d1c7498bd3d8e63632106192720d8dbca95cf3f25d",
  "runner": "benchmark_ollama.py",
  "runner_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 481558,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:18453/v1/systemone
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · jeffan-9b_en_dontknow

Name
Jeffan-9B-1211
Setup
Jeffan-9B-1211 · decision model · with the I don't know option F
Model
Jeffan-9B-1211 · provider di-serve
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.06917 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0278 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.069. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run jeffan-9b_en_dontknow · first answer 2026-11-15 07:13:38 UTC · last answer 2026-11-15 07:15:18 UTC · span 100.0 s, pauses included (not charged) · charged 100.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0692 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/tattrongvu/Jeffan-9B-1211 · read 2026-11-15 · the maker's figure from its name (9B) is shown, as for every model (turkce #4263); the card's table states 8.95B; a fine-tune of Qwen3.5-9B whose middle layers run twice per pass (no extra parameters)
Input
text, English
Endpoint
http://127.0.0.1:18453
Started
15 Nov 2026 07:13 UTC; last answer 15 Nov 2026 07:15 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Jeffan-9B-1211 (site run None); runner a75966fe5; tattrongvu/Jeffan-9B-1211@862d1a7a8b64a8d708ddb0b0cd775a740f01841a (Apache-2.0, a fine-tune of Qwen3.5-9B), served by compactor on GPU 0 of our GPU host with his di_serve.py, a thin System One wrapper around the card's own jeffan_engine:DecisionEngine loaded as the Decision Index kit does (decision-index@f8552a1; options max_len and budget 65,536, as the card; transformers 5.17.0, flash-linear-attention 0.5.2, without the optional causal-conv1d, so its PyTorch fallback); returns the engine's own answers, one request at a time (compactor #8319). Runner provider di-serve, which does not send a model field. Logic & Math on the same load as Answer Check and Reading (Umit #8157, compactor #8167)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "Jeffan-9B-1211",
    "--provider",
    "di-serve",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "Jeffan-9B-1211",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "di-serve",
  "questions": 2163,
  "request_seconds": 96.48324843894079,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "d1a729c555d3044fc0950e09283852594814c5e055c5806037ca9b1529ae2f3b",
  "runner": "benchmark_ollama.py",
  "runner_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 546448,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:18453/v1/systemone
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · chinese-pointer-4b_en

Name
Chinese Decision Pointer 4B
Setup
Chinese Decision Pointer 4B · decision model
Model
Chinese-Decision-Pointer-4B · provider di-serve
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.04911 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0197 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.049. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run chinese-pointer-4b_en · first answer 2026-11-15 07:49:29 UTC · last answer 2026-11-15 07:50:40 UTC · span 71.0 s, pauses included (not charged) · charged 71.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0491 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 4B dense · source https://huggingface.co/quincyqiang/chinese-decision-pointer-4b · read 2026-11-15 · the maker's figure from its name (4B) is shown, as for every model (turkce #4263); a fine-tune of Qwen3.5-4B; this checkpoint not counted
Input
text, English
Endpoint
http://127.0.0.1:18455
Started
15 Nov 2026 07:49 UTC; last answer 15 Nov 2026 07:50 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Chinese Decision Pointer 4B (site run None); runner a75966fe5; quincyqiang/chinese-decision-pointer-4b@889f23e2655e5cba1208bb17f90526092813f5f2 (Apache-2.0, a fine-tune of Qwen3.5-4B), served by compactor on GPU 0 of our GPU host with his di_serve.py around the card's ChineseDecisionEngine (yanqiangmiffy/chinese-decision@c16450b) with debias 'reverse' as in the card's run (averages the original and the reversed option order); Decision Index kit decision-index@9eb2dbe2 (the card names no kit commit), transformers 5.17.0, without causal-conv1d (compactor #8376). Runner provider di-serve, which sends no model field. Logic & Math on the same load as Answer Check and Reading (Umit #8157, compactor #8167)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "Chinese-Decision-Pointer-4B",
    "--provider",
    "di-serve",
    "--timeout",
    "120",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "Chinese-Decision-Pointer-4B",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "di-serve",
  "questions": 2163,
  "request_seconds": 62.35660099158122,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "9c253b0f1595eaf74b6633c2e17ae0c3d8ef6cff336acc5617d66c349d1243bc",
  "runner": "benchmark_ollama.py",
  "runner_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 837662,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:18455/v1/systemone
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · chinese-pointer-4b_en_dontknow

Name
Chinese Decision Pointer 4B
Setup
Chinese Decision Pointer 4B · decision model · with the I don't know option F
Model
Chinese-Decision-Pointer-4B · provider di-serve
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.05188 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0208 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.052. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run chinese-pointer-4b_en_dontknow · first answer 2026-11-15 07:51:33 UTC · last answer 2026-11-15 07:52:48 UTC · span 75.0 s, pauses included (not charged) · charged 75.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0519 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 4B dense · source https://huggingface.co/quincyqiang/chinese-decision-pointer-4b · read 2026-11-15 · the maker's figure from its name (4B) is shown, as for every model (turkce #4263); a fine-tune of Qwen3.5-4B; this checkpoint not counted
Input
text, English
Endpoint
http://127.0.0.1:18455
Started
15 Nov 2026 07:51 UTC; last answer 15 Nov 2026 07:52 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Chinese Decision Pointer 4B (site run None); runner a75966fe5; quincyqiang/chinese-decision-pointer-4b@889f23e2655e5cba1208bb17f90526092813f5f2 (Apache-2.0, a fine-tune of Qwen3.5-4B), served by compactor on GPU 0 of our GPU host with his di_serve.py around the card's ChineseDecisionEngine (yanqiangmiffy/chinese-decision@c16450b) with debias 'reverse' as in the card's run (averages the original and the reversed option order); Decision Index kit decision-index@9eb2dbe2 (the card names no kit commit), transformers 5.17.0, without causal-conv1d (compactor #8376). Runner provider di-serve, which sends no model field. Logic & Math on the same load as Answer Check and Reading (Umit #8157, compactor #8167)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "Chinese-Decision-Pointer-4B",
    "--provider",
    "di-serve",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "Chinese-Decision-Pointer-4B",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "di-serve",
  "questions": 2163,
  "request_seconds": 66.80018996635044,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "dba119a77d7818435e973742e4997542c21862b6c5e5e308d0710c349fda1c56",
  "runner": "benchmark_ollama.py",
  "runner_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 971768,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:18455/v1/systemone
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · gpt-6-luna-low-sub_en

Name
GPT-6 Luna chat
Setup
GPT-6 Luna (low) · chat model
Model
gpt-6-luna · provider chatgpt-subscription
GPU
—
Cost
$0 for the run · ChatGPT subscription
Answer time
run gpt-6-luna-low-sub_en · first answer 2026-11-15 08:00:33 UTC · last answer 2026-11-15 10:25:28 UTC · span 8695.0 s, pauses included (not charged) · charged 8695.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker (closed weights) · source https://openrouter.ai/openai/gpt-6-luna · read 2026-11-10
Input
text, English
Endpoint
codex-chatgpt-subscription
Started
15 Nov 2026 08:00 UTC; last answer 15 Nov 2026 10:25 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
GPT-6 Luna (low) on the ChatGPT subscription through the Codex CLI (`codex exec`, ChatGPT sign-in from OMP's openai-codex login in a private CODEX_HOME, no refresh token, since 13 Nov 19:35 UTC; as turkce's ask_openai), model gpt-6-luna, model_reasoning_effort low (Reading's setting); no output limit sent (Codex has no setting for it) and no temperature; Codex's agent instructions replaced by a neutral one-line system prompt, web search and tool features off, read-only sandbox, no user config, rules or saved session, API-key variables removed; one question per request, one request at a time; a usage-limit reply stops the run without retrying it (resumable); cost $0 (ChatGPT subscription); api_equivalent_usd is the tokens at OpenAI's API list price {"input": 0.1, "cached_input": 0.01, "cache_write": 0.125, "output": 0.5} per 1M tokens, not a cost. Codex reports no served model id. English translations.
Parameters
{
  "disabled_features": [
    "shell_tool",
    "unified_exec",
    "goals",
    "apps",
    "plugins",
    "remote_plugin",
    "tool_suggest",
    "skill_search",
    "view_image",
    "image_generation",
    "sleep_tool",
    "browser_use",
    "browser_use_external",
    "computer_use",
    "in_app_browser",
    "multi_agent",
    "unbounded_connection_retries"
  ],
  "effort": "low",
  "list_price_per_mtok": {
    "cache_write": 0.125,
    "cached_input": 0.01,
    "input": 0.1,
    "output": 0.5
  },
  "max_output_tokens": null,
  "route": "codex exec (ChatGPT sign-in)",
  "sandbox": "read-only",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message.",
  "web_search": "disabled"
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · qwen3.5-2b-nothink_en_dontknow

Name
Qwen3.5-2B off · chat
Setup
Qwen3.5-2B (chat, thinking off, on our GPU) · chat model · with the I don't know option F
Model
qwen3.5-2b-nothink · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.7186 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.289 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.719. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run qwen3.5-2b-nothink_en_dontknow · first answer 2026-11-15 08:19:53 UTC · last answer 2026-11-15 08:37:12 UTC · span 1039.0 s, pauses included (not charged) · charged 1039.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.7186 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.6 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 2B dense · source https://huggingface.co/Qwen/Qwen3.5-2B · read 2026-11-13 · the card states Number of Parameters: 2B (the maker's figure is shown); checkpoint count 2.27B (2,274,069,824: language model 1.88B incl. tied embeddings, vision encoder 0.33B, MTP head 0.06B; dense)
Input
text, English
Endpoint
http://127.0.0.1:18643/v1/chat/completions
Started
15 Nov 2026 08:19 UTC; last answer 15 Nov 2026 08:37 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Qwen3.5-2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from Qwen/Qwen3.5-2B@15852e8c16360a2fea060d615a32b45270f8a8fc, vLLM reasoning parser qwen3 (vision tower loaded, for Reading's Page images; text-only prompts here), one RTX PRO 6000 on our GPU host alone; thinking off (enable_thinking false, the template default for this size; on/off only, no levels); sampling temperature 1, top_p 0.95, top_k 20, min_p 0, presence_penalty 1.5, repetition_penalty 1, sampling from the model card, generation_config has none (non-thinking mode for reasoning tasks; the repo has no generation_config.json); seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 2.27B / 2.27B active / total (dense; checkpoint 2,274,069,824: language model 1.88B incl. tied embeddings, vision encoder 0.33B, MTP head 0.06B (not loaded); card: 2B). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "presence_penalty": 1.5,
  "repetition_penalty": 1.0,
  "seed": 0,
  "temperature": 1.0,
  "top_k": 20,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · logits-sigma-1-duo_en

Name
Logits Sigma-1 Duo
Setup
Logits Sigma-1 Duo · decision model
Model
Logits-Sigma-1-Duo · provider di-serve
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.3043 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.122 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.304. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run logits-sigma-1-duo_en · first answer 2026-11-15 08:42:58 UTC · last answer 2026-11-15 08:50:18 UTC · span 440.0 s, pauses included (not charged) · charged 440.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.3043 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 57.36B dense · source https://huggingface.co/logitshq/Logits-Sigma-1-Duo · read 2026-11-15 · the card states no total; counted from its two parts, as REAX 1.3 (both used on every question, option probabilities averaged): KnowLine 31B Gen1 with Logits' LoRA plus vLLM-SR d3-ultra (26.09B); stored in FP8
Input
text, English
Endpoint
http://127.0.0.1:18456
Started
15 Nov 2026 08:42 UTC; last answer 15 Nov 2026 08:50 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Logits Sigma-1 Duo (site run None); runner a75966fe5; logitshq/Logits-Sigma-1-Duo@a0c1b6e8242916aafdcf9a5a49c1a952cb56e091 (Apache-2.0; weights and code identical to d9641cd8, the revision its card's run names): two decision models averaged per option, KnowLine 31B Gen1 with Logits' LoRA and vLLM-SR d3-ultra, both stored in FP8. Served by compactor on GPU 0 of our GPU host with his di_serve.py around the card's engine (logits_sigma_1_duo.engine_pair:Duo, model=<snapshot>, as its serve.sh fixes them) in the card's own serving image (code/Dockerfile, kit decision-index@25c6ec7, causal-conv1d 1.7.0) plus only gcc and Python headers, which its Triton kernels need to compile at first use. FP8 matmuls on the engine's own Triton kernel (its cuBLAS block-scaled path is not taken on this GPU; the engine falls back by design); d3-ultra on fla 0.6.0 (compactor #8398). Runner provider di-serve, which sends no model field. Logic & Math on the same load as Answer Check and Reading (Umit #8157, compactor #8167)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "Logits-Sigma-1-Duo",
    "--provider",
    "di-serve",
    "--timeout",
    "120",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "Logits-Sigma-1-Duo",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "di-serve",
  "questions": 2163,
  "request_seconds": 429.98437585636566,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "0175ee156982a8a6488040f6dabcacc7ff3cba5b8655ad75433b80422ea3fa2e",
  "runner": "benchmark_ollama.py",
  "runner_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 0,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:18456/v1/systemone
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · logits-sigma-1-duo_en_dontknow

Name
Logits Sigma-1 Duo
Setup
Logits Sigma-1 Duo · decision model · with the I don't know option F
Model
Logits-Sigma-1-Duo · provider di-serve
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.3189 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.128 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.319. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run logits-sigma-1-duo_en_dontknow · first answer 2026-11-15 08:56:03 UTC · last answer 2026-11-15 09:03:44 UTC · span 461.0 s, pauses included (not charged) · charged 461.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.3189 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 57.36B dense · source https://huggingface.co/logitshq/Logits-Sigma-1-Duo · read 2026-11-15 · the card states no total; counted from its two parts, as REAX 1.3 (both used on every question, option probabilities averaged): KnowLine 31B Gen1 with Logits' LoRA plus vLLM-SR d3-ultra (26.09B); stored in FP8
Input
text, English
Endpoint
http://127.0.0.1:18456
Started
15 Nov 2026 08:56 UTC; last answer 15 Nov 2026 09:03 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Logits Sigma-1 Duo (site run None); runner a75966fe5; logitshq/Logits-Sigma-1-Duo@a0c1b6e8242916aafdcf9a5a49c1a952cb56e091 (Apache-2.0; weights and code identical to d9641cd8, the revision its card's run names): two decision models averaged per option, KnowLine 31B Gen1 with Logits' LoRA and vLLM-SR d3-ultra, both stored in FP8. Served by compactor on GPU 0 of our GPU host with his di_serve.py around the card's engine (logits_sigma_1_duo.engine_pair:Duo, model=<snapshot>, as its serve.sh fixes them) in the card's own serving image (code/Dockerfile, kit decision-index@25c6ec7, causal-conv1d 1.7.0) plus only gcc and Python headers, which its Triton kernels need to compile at first use. FP8 matmuls on the engine's own Triton kernel (its cuBLAS block-scaled path is not taken on this GPU; the engine falls back by design); d3-ultra on fla 0.6.0 (compactor #8398). Runner provider di-serve, which sends no model field. Logic & Math on the same load as Answer Check and Reading (Umit #8157, compactor #8167)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "Logits-Sigma-1-Duo",
    "--provider",
    "di-serve",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "Logits-Sigma-1-Duo",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "di-serve",
  "questions": 2163,
  "request_seconds": 448.5538808349811,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "9cc27356f678d08c739eddbe218a72456e9c098fbf5242564907ba72201e58f1",
  "runner": "benchmark_ollama.py",
  "runner_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 0,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:18456/v1/systemone
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · qwen3.5-2b_en

Name
Qwen3.5-2B on · chat
Setup
Qwen3.5-2B (chat, thinking, on our GPU) · chat model
Model
qwen3.5-2b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$5.342 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 2.15 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $5.342. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-15 10:06:01–12:02:04 UTC
GPU time
run qwen3.5-2b_en · first answer 2026-11-15 10:03:40 UTC · last answer 2026-11-15 14:08:27 UTC · span 14687.0 s, pauses included (not charged) · charged 7724.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $5.3424 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.5 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 98 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 2B dense · source https://huggingface.co/Qwen/Qwen3.5-2B · read 2026-11-13 · the card states Number of Parameters: 2B (the maker's figure is shown); checkpoint count 2.27B (2,274,069,824: language model 1.88B incl. tied embeddings, vision encoder 0.33B, MTP head 0.06B; dense)
Input
text, English
Endpoint
http://127.0.0.1:18635/v1/chat/completions
Started
15 Nov 2026 10:03 UTC; last answer 15 Nov 2026 14:08 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Qwen3.5-2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from Qwen/Qwen3.5-2B@15852e8c16360a2fea060d615a32b45270f8a8fc, vLLM reasoning parser qwen3 (vision tower loaded, for Reading's Page images; text-only prompts here; the card warns the 2B 'is more prone to entering thinking loops' in thinking mode), one RTX PRO 6000 on our GPU host alone; thinking on (enable_thinking true; the template default is off for this size; on/off only, no levels); sampling temperature 1, top_p 0.95, top_k 20, min_p 0, presence_penalty 1.5, repetition_penalty 1, sampling from the model card, generation_config has none (thinking mode, text tasks; the repo has no generation_config.json); seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 2.27B / 2.27B active / total (dense; checkpoint 2,274,069,824: language model 1.88B incl. tied embeddings, vision encoder 0.33B, MTP head 0.06B (not loaded); card: 2B). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "presence_penalty": 1.5,
  "repetition_penalty": 1.0,
  "seed": 0,
  "temperature": 1.0,
  "top_k": 20,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · gpt-6-luna-low-sub_en_dontknow

Name
GPT-6 Luna chat
Setup
GPT-6 Luna (low) · chat model · with the I don't know option F
Model
gpt-6-luna · provider chatgpt-subscription
GPU
—
Cost
not measured
Answer time
run gpt-6-luna-low-sub_en_dontknow · first answer 2026-11-15 10:25:33 UTC · last answer 2026-11-15 11:37:02 UTC · span 4289.0 s, pauses included (not charged) · charged 4289.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker (closed weights) · source https://openrouter.ai/openai/gpt-6-luna · read 2026-11-10
Input
text, English
Endpoint
codex-chatgpt-subscription
Started
15 Nov 2026 10:25 UTC; last answer 15 Nov 2026 11:37 UTC
Progress
1,075 answered of 2,163 asked; 0 request errors; paused for now at 1,075 of 2,163 (Ümit #8603, zeki #8637, 2026-11-15T11:37Z)
Notes
GPT-6 Luna (low) on the ChatGPT subscription through the Codex CLI (`codex exec`, ChatGPT sign-in from OMP's openai-codex login in a private CODEX_HOME, no refresh token, since 13 Nov 19:35 UTC; as turkce's ask_openai), model gpt-6-luna, model_reasoning_effort low (Reading's setting); no output limit sent (Codex has no setting for it) and no temperature; Codex's agent instructions replaced by a neutral one-line system prompt, web search and tool features off, read-only sandbox, no user config, rules or saved session, API-key variables removed; one question per request, one request at a time; a usage-limit reply stops the run without retrying it (resumable); cost $0 (ChatGPT subscription); api_equivalent_usd is the tokens at OpenAI's API list price {"input": 0.1, "cached_input": 0.01, "cache_write": 0.125, "output": 0.5} per 1M tokens, not a cost. Codex reports no served model id. English translations. PAUSED (Umit #8603, zeki #8637, 2026-11-15T11:37Z): skipped for now, stopped at 1075 answers; large models are left for now (in-flight requests at the stop were not stored); resumable from the stored answers at the same settings if asked.
Parameters
{
  "disabled_features": [
    "shell_tool",
    "unified_exec",
    "goals",
    "apps",
    "plugins",
    "remote_plugin",
    "tool_suggest",
    "skill_search",
    "view_image",
    "image_generation",
    "sleep_tool",
    "browser_use",
    "browser_use_external",
    "computer_use",
    "in_app_browser",
    "multi_agent",
    "unbounded_connection_retries"
  ],
  "effort": "low",
  "list_price_per_mtok": {
    "cache_write": 0.125,
    "cached_input": 0.01,
    "input": 0.1,
    "output": 0.5
  },
  "max_output_tokens": null,
  "route": "codex exec (ChatGPT sign-in)",
  "sandbox": "read-only",
  "system_prompt": "You are taking a multiple-choice benchmark. Follow the instructions in the user's message.",
  "web_search": "disabled"
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · verdin-4b_en

Name
Verdin-4B-v1.1
Setup
Verdin-4B-v1.1 · decision model
Model
Verdin-4B-v1.1 · provider di-serve
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.07263 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0292 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.073. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run verdin-4b_en · first answer 2026-11-15 11:23:31 UTC · last answer 2026-11-15 11:25:16 UTC · span 105.0 s, pauses included (not charged) · charged 105.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0726 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 4B dense · source https://huggingface.co/victorbennett/Verdin-4B-v1.1 · read 2026-11-15 · the maker's figure from its name (4B) is shown, as for every model (turkce #4263); the card states 4.21B served
Input
text, English
Endpoint
http://127.0.0.1:18457
Started
15 Nov 2026 11:23 UTC; last answer 15 Nov 2026 11:25 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Verdin-4B-v1.1 (site run None); runner a75966fe5; victorbennett/Verdin-4B-v1.1@45830d216bb7e45f43f0a482c131a7cafa6d189f (CC BY-NC 4.0, non-commercial; a fine-tune of Qwen3.5-4B), served by compactor on GPU 2 of our GPU host (jevgammon's booking r_265f5494128e, lent for the index models, #8539) with the card's own server (python -m verdin.server --model .) at the card's versions (torch 2.8.0 cu128, transformers 5.17.0, fla 0.5.2, triton 3.8.0), without causal-conv1d, so its torch conv path (compactor #8576). Runner provider di-serve (the System One request at the default path, no model field). Logic & Math on the same load as Answer Check and Reading (Umit #8157, #8448)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "Verdin-4B-v1.1",
    "--provider",
    "di-serve",
    "--timeout",
    "120",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "Verdin-4B-v1.1",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "di-serve",
  "questions": 2163,
  "request_seconds": 100.28281387522293,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "d3cddcf623a75283f10eb3e29f29ba772be6e7571d7b296cfd04c9c3dab19779",
  "runner": "benchmark_ollama.py",
  "runner_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 469533,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:18457/v1/systemone
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · verdin-4b_en_dontknow

Name
Verdin-4B-v1.1
Setup
Verdin-4B-v1.1 · decision model · with the I don't know option F
Model
Verdin-4B-v1.1 · provider di-serve
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.07263 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0292 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.073. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run verdin-4b_en_dontknow · first answer 2026-11-15 11:26:22 UTC · last answer 2026-11-15 11:28:07 UTC · span 105.0 s, pauses included (not charged) · charged 105.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0726 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 4B dense · source https://huggingface.co/victorbennett/Verdin-4B-v1.1 · read 2026-11-15 · the maker's figure from its name (4B) is shown, as for every model (turkce #4263); the card states 4.21B served
Input
text, English
Endpoint
http://127.0.0.1:18457
Started
15 Nov 2026 11:26 UTC; last answer 15 Nov 2026 11:28 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Verdin-4B-v1.1 (site run None); runner a75966fe5; victorbennett/Verdin-4B-v1.1@45830d216bb7e45f43f0a482c131a7cafa6d189f (CC BY-NC 4.0, non-commercial; a fine-tune of Qwen3.5-4B), served by compactor on GPU 2 of our GPU host (jevgammon's booking r_265f5494128e, lent for the index models, #8539) with the card's own server (python -m verdin.server --model .) at the card's versions (torch 2.8.0 cu128, transformers 5.17.0, fla 0.5.2, triton 3.8.0), without causal-conv1d, so its torch conv path (compactor #8576). Runner provider di-serve (the System One request at the default path, no model field). Logic & Math on the same load as Answer Check and Reading (Umit #8157, #8448)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "Verdin-4B-v1.1",
    "--provider",
    "di-serve",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "Verdin-4B-v1.1",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "di-serve",
  "questions": 2163,
  "request_seconds": 99.31648989433597,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "241de910d75019cc83a59c0da0b555b252a530cfb9552e64ace68b20d7e80c6e",
  "runner": "benchmark_ollama.py",
  "runner_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 577683,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:18457/v1/systemone
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · dexy-9b_en

Name
Dexy-Decision-9B-V2
Setup
Dexy-Decision-9B-V2 · decision model
Model
Dexy-Decision-9B-V2 · provider di-serve
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.06917 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0278 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.069. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run dexy-9b_en · first answer 2026-11-15 11:57:59 UTC · last answer 2026-11-15 11:59:39 UTC · span 100.0 s, pauses included (not charged) · charged 100.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0692 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/Patrick-18/Dexy-Decision-9B-V2 · read 2026-11-15 · the maker's figure from its name (9B) is shown, as for every model (turkce #4263); the card's table states 8.394B incl. a 0.456B vision encoder
Input
text, English
Endpoint
http://127.0.0.1:18459
Started
15 Nov 2026 11:57 UTC; last answer 15 Nov 2026 11:59 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Dexy-Decision-9B-V2 (site run None); runner a75966fe5; Patrick-18/Dexy-Decision-9B-V2@4f434e2b5fd9f0addfe3233489571c2dc8ce7a19 (Apache-2.0; built on vllm-sr/d3-flash), served by compactor on GPU 2 of our GPU host (jevgammon's booking r_265f5494128e, lent for the index models, #8539) with the card's Transformers d3_server.py at its requirements.txt pins (torch 2.14.0, transformers 5.19.0) (compactor #8697). Runner provider di-serve (the System One request at the default path, no model field; the server answers as 'Dexy-Decision-9B-V2'). Logic & Math on the same load as Answer Check and Reading (Umit #8157, #8448)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "Dexy-Decision-9B-V2",
    "--provider",
    "di-serve",
    "--timeout",
    "120",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "Dexy-Decision-9B-V2",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "di-serve",
  "questions": 2163,
  "request_seconds": 94.84468804263452,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "0fdd8b4f0566ae42103f7f0d94acb0eea35371f00a944e673331442dea269089",
  "runner": "benchmark_ollama.py",
  "runner_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 481558,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:18459/v1/systemone
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · dexy-9b_en_dontknow

Name
Dexy-Decision-9B-V2
Setup
Dexy-Decision-9B-V2 · decision model · with the I don't know option F
Model
Dexy-Decision-9B-V2 · provider di-serve
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.06709 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0269 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.067. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run dexy-9b_en_dontknow · first answer 2026-11-15 12:00:44 UTC · last answer 2026-11-15 12:02:21 UTC · span 97.0 s, pauses included (not charged) · charged 97.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0671 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/Patrick-18/Dexy-Decision-9B-V2 · read 2026-11-15 · the maker's figure from its name (9B) is shown, as for every model (turkce #4263); the card's table states 8.394B incl. a 0.456B vision encoder
Input
text, English
Endpoint
http://127.0.0.1:18459
Started
15 Nov 2026 12:00 UTC; last answer 15 Nov 2026 12:02 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Dexy-Decision-9B-V2 (site run None); runner a75966fe5; Patrick-18/Dexy-Decision-9B-V2@4f434e2b5fd9f0addfe3233489571c2dc8ce7a19 (Apache-2.0; built on vllm-sr/d3-flash), served by compactor on GPU 2 of our GPU host (jevgammon's booking r_265f5494128e, lent for the index models, #8539) with the card's Transformers d3_server.py at its requirements.txt pins (torch 2.14.0, transformers 5.19.0) (compactor #8697). Runner provider di-serve (the System One request at the default path, no model field; the server answers as 'Dexy-Decision-9B-V2'). Logic & Math on the same load as Answer Check and Reading (Umit #8157, #8448)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "Dexy-Decision-9B-V2",
    "--provider",
    "di-serve",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "Dexy-Decision-9B-V2",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "di-serve",
  "questions": 2163,
  "request_seconds": 92.6366439643898,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "6da6a72bd1b7793ca57b828d49202f0d1a797d2a146f9e296105a11d996ccf08",
  "runner": "benchmark_ollama.py",
  "runner_commit": "a75966fe551dc23f3c28e3a9c37bc500a8dac762",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 546448,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:18459/v1/systemone
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · vera-medium-omni_en

Name
vera-medium-omni
Setup
vera-medium-omni · decision model
Model
vera-medium-omni · provider di-serve
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.05049 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0203 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.050. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run vera-medium-omni_en · first answer 2026-11-15 12:18:12 UTC · last answer 2026-11-15 12:19:25 UTC · span 73.0 s, pauses included (not charged) · charged 73.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0505 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 4.2B dense · source https://huggingface.co/scar-ai/vera-medium-omni · read 2026-11-15 · the maker's stated figure for the text backbone (~4.2B) is shown, as for every model (turkce #4263); its name has none; the vision tower (~0.33B) is not used on text questions
Input
text, English
Endpoint
http://127.0.0.1:18458
Started
15 Nov 2026 12:18 UTC; last answer 15 Nov 2026 12:19 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup vera-medium-omni (site run None); runner c6e3cdddb; scar-ai/vera-medium-omni@1f23a6cf5304e8e53b7ded53980765d5f8f4047c (Apache-2.0; derived from Qwen3.5-4B), served by compactor on GPU 2 of our GPU host (jevgammon's booking r_265f5494128e, lent for the index models, #8539) with the card's own vera-serve from vera-s1 0.1.8, PyTorch engine (compactor #8676). Runner provider di-serve (the System One request at the default path, no model field). Logic & Math on the same load as Answer Check and Reading (Umit #8157, #8448)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "vera-medium-omni",
    "--provider",
    "di-serve",
    "--timeout",
    "120",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "c6e3cdddbbc06c50393f39bfa39d2d995bc5fbc1",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "vera-medium-omni",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "di-serve",
  "questions": 2163,
  "request_seconds": 68.50848529960786,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "95caaacf81679d10e59768397de65955784b80c97b7cb8aa549342b29ec8d1fd",
  "runner": "benchmark_ollama.py",
  "runner_commit": "c6e3cdddbbc06c50393f39bfa39d2d995bc5fbc1",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 481558,
    "output_tokens": 73515
  }
}
Prompt template
POST http://127.0.0.1:18458/v1/systemone
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · vera-medium-omni_en_dontknow

Name
vera-medium-omni
Setup
vera-medium-omni · decision model · with the I don't know option F
Model
vera-medium-omni · provider di-serve
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.04911 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0197 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.049. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run vera-medium-omni_en_dontknow · first answer 2026-11-15 12:20:08 UTC · last answer 2026-11-15 12:21:19 UTC · span 71.0 s, pauses included (not charged) · charged 71.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0491 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 4.2B dense · source https://huggingface.co/scar-ai/vera-medium-omni · read 2026-11-15 · the maker's stated figure for the text backbone (~4.2B) is shown, as for every model (turkce #4263); its name has none; the vision tower (~0.33B) is not used on text questions
Input
text, English
Endpoint
http://127.0.0.1:18458
Started
15 Nov 2026 12:20 UTC; last answer 15 Nov 2026 12:21 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup vera-medium-omni (site run None); runner c6e3cdddb; scar-ai/vera-medium-omni@1f23a6cf5304e8e53b7ded53980765d5f8f4047c (Apache-2.0; derived from Qwen3.5-4B), served by compactor on GPU 2 of our GPU host (jevgammon's booking r_265f5494128e, lent for the index models, #8539) with the card's own vera-serve from vera-s1 0.1.8, PyTorch engine (compactor #8676). Runner provider di-serve (the System One request at the default path, no model field). Logic & Math on the same load as Answer Check and Reading (Umit #8157, #8448)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "vera-medium-omni",
    "--provider",
    "di-serve",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "c6e3cdddbbc06c50393f39bfa39d2d995bc5fbc1",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "vera-medium-omni",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "di-serve",
  "questions": 2163,
  "request_seconds": 66.19773302822432,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "143240b553a5e3a2557e07c8298e4c68431afff2c0d7124eb29d8ee8e8e130ab",
  "runner": "benchmark_ollama.py",
  "runner_commit": "c6e3cdddbbc06c50393f39bfa39d2d995bc5fbc1",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 546448,
    "output_tokens": 81069
  }
}
Prompt template
POST http://127.0.0.1:18458/v1/systemone
{
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · valen-9b_en

Name
Valen-Decision-1.5-9B
Setup
Valen-Decision-1.5-9B · decision model
Model
valen · provider valen
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.08853 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0356 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.089. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run valen-9b_en · first answer 2026-11-15 12:45:26 UTC · last answer 2026-11-15 12:47:34 UTC · span 128.0 s, pauses included (not charged) · charged 128.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0885 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/Valen-Team/Valen-Decision-1.5-9B · read 2026-11-15 · the maker's figure from its name (9B) is shown, as for every model (turkce #4263); a fine-tune of Qwen3.5-9B; this checkpoint not counted
Input
text, English
Endpoint
http://127.0.0.1:18460
Started
15 Nov 2026 12:45 UTC; last answer 15 Nov 2026 12:47 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Valen-Decision-1.5-9B (site run None); runner 88e48d7a1; Valen-Team/Valen-Decision-1.5-9B@5a512117ebc0f2b69f311becfcf47cdef8abf568 (Apache-2.0; a fine-tune of Qwen3.5-9B), served by compactor on GPU 2 of our GPU host (jevgammon's booking r_265f5494128e, lent for the index models, #8539) with the card's vLLM deployment valen-vllm (plugin Valen-Team/Valen@773d3a64, installed from a GitHub archive with --no-deps) in the official vllm/vllm-openai:v0.33.0 image, container cmp-valen-decision-1.5-9b (compactor #8773). Runner provider valen (the System One request at the default path with model field 'valen', which this server requires; turkce #8783). Logic & Math on the same load as Answer Check and Reading (Umit #8157, #8448)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "valen",
    "--provider",
    "valen",
    "--timeout",
    "120",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "valen",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "valen",
  "questions": 2163,
  "request_seconds": 123.3042892729427,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "909b9bd419453595994002067fc305acbcf31510ac96b32a75a24b391accc1f0",
  "runner": "benchmark_ollama.py",
  "runner_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 375571,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:18460/v1/systemone
{
 "model": "valen",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · valen-9b_en_dontknow

Name
Valen-Decision-1.5-9B
Setup
Valen-Decision-1.5-9B · decision model · with the I don't know option F
Model
valen · provider valen
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.09061 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0364 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.091. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run valen-9b_en_dontknow · first answer 2026-11-15 12:48:54 UTC · last answer 2026-11-15 12:51:05 UTC · span 131.0 s, pauses included (not charged) · charged 131.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0906 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/Valen-Team/Valen-Decision-1.5-9B · read 2026-11-15 · the maker's figure from its name (9B) is shown, as for every model (turkce #4263); a fine-tune of Qwen3.5-9B; this checkpoint not counted
Input
text, English
Endpoint
http://127.0.0.1:18460
Started
15 Nov 2026 12:48 UTC; last answer 15 Nov 2026 12:51 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Valen-Decision-1.5-9B (site run None); runner 88e48d7a1; Valen-Team/Valen-Decision-1.5-9B@5a512117ebc0f2b69f311becfcf47cdef8abf568 (Apache-2.0; a fine-tune of Qwen3.5-9B), served by compactor on GPU 2 of our GPU host (jevgammon's booking r_265f5494128e, lent for the index models, #8539) with the card's vLLM deployment valen-vllm (plugin Valen-Team/Valen@773d3a64, installed from a GitHub archive with --no-deps) in the official vllm/vllm-openai:v0.33.0 image, container cmp-valen-decision-1.5-9b (compactor #8773). Runner provider valen (the System One request at the default path with model field 'valen', which this server requires; turkce #8783). Logic & Math on the same load as Answer Check and Reading (Umit #8157, #8448)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "valen",
    "--provider",
    "valen",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "valen",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "valen",
  "questions": 2163,
  "request_seconds": 126.27527871044003,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "5ce37a23da3359115b2a09f9fa48e43847e92a5239d8d9f8bafee4d87c8383d3",
  "runner": "benchmark_ollama.py",
  "runner_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 436135,
    "output_tokens": 0
  }
}
Prompt template
POST http://127.0.0.1:18460/v1/systemone
{
 "model": "valen",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · startlux-9b_en

Name
StartLux-Decision-v1.1-9B-Instant
Setup
StartLux-Decision v1.1 9B · decision model
Model
startlux-models/StartLux-Decision-v1.1-9B-Instant · provider startlux
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.01803 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0847 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.018 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run startlux-9b_en · first answer 2026-11-15 13:46:39 UTC · last answer 2026-11-15 13:51:44 UTC · span 305.0 s, pauses included (not charged) · charged 305.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0180 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/startlux-models/StartLux-Decision-v1.1-9B-Instant · read 2026-11-15 · the maker's figure from its name (9B) is shown, as for every model (turkce #4263); this checkpoint not counted
Input
text, English
Endpoint
internal address on a shared Spark host
Started
15 Nov 2026 13:46 UTC; last answer 15 Nov 2026 13:51 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup StartLux-Decision v1.1 9B (site run None); runner 88e48d7a1; startlux-models/StartLux-Decision-v1.1-9B-Instant@18a17f734966b213b2f00fa9c915e8b9af2a9b20 (weights CC BY-NC 4.0, non-commercial: run for this non-commercial benchmark; code Apache-2.0) behind the card's server, the same package and environment as StartLux 4B (github.com/StartLuxLabs/StartLux-Decision at a99977d, ~/envs/startlux: vllm==0.31.0): ~/envs/startlux/bin/startlux-decide serve --model ~/decision-models/StartLux-Decision-v1.1-9B-Instant@18a17f734966b213b2f00fa9c915e8b9af2a9b20 --launch-vllm --gpus 0 --host 0.0.0.0 --port 19420; vision on (the server's default); a shared Spark host (DGX Spark, GB10); runner provider 'startlux' (System One at /v1/systemone, model ignored). Logic & Math on the same load as Answer Check and Reading (Umit #8448, compactor #8468)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "startlux-models/StartLux-Decision-v1.1-9B-Instant",
    "--provider",
    "startlux",
    "--timeout",
    "60",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "startlux-models/StartLux-Decision-v1.1-9B-Instant",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "startlux",
  "questions": 2163,
  "request_seconds": 299.12650946922076,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "e6ec11eaf8cee12d6458f23b364926073b75d3e76560d1f8a6bd343c916ee251",
  "runner": "benchmark_ollama.py",
  "runner_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 416668,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19420/v1/systemone
{
 "model": "startlux-models/StartLux-Decision-v1.1-9B-Instant",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · startlux-9b_en_dontknow

Name
StartLux-Decision-v1.1-9B-Instant
Setup
StartLux-Decision v1.1 9B · decision model · with the I don't know option F
Model
startlux-models/StartLux-Decision-v1.1-9B-Instant · provider startlux
GPU
NVIDIA DGX Spark (GB10)
Cost
$0.0188 for the run, DGX Spark · estimate: NVIDIA DGX Spark (GB10) time while the run was running, first to last answer × $0.213/h · GPU time, at most: 0.0883 h × $0.213/h on an NVIDIA DGX Spark (GB10) = $0.019 (CHF 4,000 ÷ (3 years × 8,760 h) = $0.185/h + electricity (79.1 W chip draw × CHF 0.2838/kWh = $0.027/h; the GB10 chip's draw measured with nvidia-smi, not the whole machine; ewz.natur's high-rate tariff, the tariff basis, not our verified contract), = $0.213/h at 1.2185 USD/CHF). GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run startlux-9b_en_dontknow · first answer 2026-11-15 13:55:09 UTC · last answer 2026-11-15 14:00:27 UTC · span 318.0 s, pauses included (not charged) · charged 318.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0188 for the run (charged seconds × $0.213/GPU-hour × 1 NVIDIA DGX Spark (GB10), at most)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/startlux-models/StartLux-Decision-v1.1-9B-Instant · read 2026-11-15 · the maker's figure from its name (9B) is shown, as for every model (turkce #4263); this checkpoint not counted
Input
text, English
Endpoint
internal address on a shared Spark host
Started
15 Nov 2026 13:55 UTC; last answer 15 Nov 2026 14:00 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup StartLux-Decision v1.1 9B (site run None); runner 88e48d7a1; startlux-models/StartLux-Decision-v1.1-9B-Instant@18a17f734966b213b2f00fa9c915e8b9af2a9b20 (weights CC BY-NC 4.0, non-commercial: run for this non-commercial benchmark; code Apache-2.0) behind the card's server, the same package and environment as StartLux 4B (github.com/StartLuxLabs/StartLux-Decision at a99977d, ~/envs/startlux: vllm==0.31.0): ~/envs/startlux/bin/startlux-decide serve --model ~/decision-models/StartLux-Decision-v1.1-9B-Instant@18a17f734966b213b2f00fa9c915e8b9af2a9b20 --launch-vllm --gpus 0 --host 0.0.0.0 --port 19420; vision on (the server's default); a shared Spark host (DGX Spark, GB10); runner provider 'startlux' (System One at /v1/systemone, model ignored). Logic & Math on the same load as Answer Check and Reading (Umit #8448, compactor #8468)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "startlux-models/StartLux-Decision-v1.1-9B-Instant",
    "--provider",
    "startlux",
    "--timeout",
    "60",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "startlux-models/StartLux-Decision-v1.1-9B-Instant",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "startlux",
  "questions": 2163,
  "request_seconds": 309.75444100583263,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "6e92a314ecf6f53e119878a6e0ba00fdb5131a37224913dcf1158a51495fba52",
  "runner": "benchmark_ollama.py",
  "runner_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 60.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 477232,
    "output_tokens": 0
  }
}
Prompt template
POST http://spark2:19420/v1/systemone
{
 "model": "startlux-models/StartLux-Decision-v1.1-9B-Instant",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · qwen3.5-2b_en_dontknow

Name
Qwen3.5-2B on · chat
Setup
Qwen3.5-2B (chat, thinking, on our GPU) · chat model · with the I don't know option F
Model
qwen3.5-2b · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$3.9 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 1.57 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $3.900. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run qwen3.5-2b_en_dontknow · first answer 2026-11-15 15:57:32 UTC · last answer 2026-11-15 17:31:31 UTC · span 5639.0 s, pauses included (not charged) · charged 5639.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $3.9003 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.9 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 37 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 2B dense · source https://huggingface.co/Qwen/Qwen3.5-2B · read 2026-11-13 · the card states Number of Parameters: 2B (the maker's figure is shown); checkpoint count 2.27B (2,274,069,824: language model 1.88B incl. tied embeddings, vision encoder 0.33B, MTP head 0.06B; dense)
Input
text, English
Endpoint
http://127.0.0.1:18635/v1/chat/completions
Started
15 Nov 2026 15:57 UTC; last answer 15 Nov 2026 17:31 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Qwen3.5-2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from Qwen/Qwen3.5-2B@15852e8c16360a2fea060d615a32b45270f8a8fc, vLLM reasoning parser qwen3 (vision tower loaded, for Reading's Page images; text-only prompts here; the card warns the 2B 'is more prone to entering thinking loops' in thinking mode), one RTX PRO 6000 on our GPU host alone; thinking on (enable_thinking true; the template default is off for this size; on/off only, no levels); sampling temperature 1, top_p 0.95, top_k 20, min_p 0, presence_penalty 1.5, repetition_penalty 1, sampling from the model card, generation_config has none (thinking mode, text tasks; the repo has no generation_config.json); seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 2.27B / 2.27B active / total (dense; checkpoint 2,274,069,824: language model 1.88B incl. tied embeddings, vision encoder 0.33B, MTP head 0.06B (not loaded); card: 2B). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "presence_penalty": 1.5,
  "repetition_penalty": 1.0,
  "seed": 0,
  "temperature": 1.0,
  "top_k": 20,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · lfm3-1.7b-vllm-nothink-rtx-seed0_en

Name
LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell
Setup
LFM3-1.7B (chat, thinking off, RTX PRO 6000, on our GPU) · chat model
Model
lfm3-1.7b-vllm-nothink · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.3182 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.128 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.318. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-15 17:02:41–17:32:26 UTC
GPU time
run lfm3-1.7b-vllm-nothink-rtx-seed0_en · first answer 2026-11-15 17:02:21 UTC · last answer 2026-11-15 17:39:46 UTC · span 2245.0 s, pauses included (not charged) · charged 460.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.3182 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 28.6 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 1.73B dense · from the run's notes (dense; BF16 safetensors 1,728,801,536, text only; card: 1.7B); also https://huggingface.co/LiquidAI/LFM3-1.7B-GGUF · read 2026-11-09 · the released checkpoint's tensors without registered buffers: 1,728,801,536 (the served BF16 GGUF also stores a 64-element RoPE buffer, 1,728,801,600); text only; card: 1.7B
Input
text, English
Endpoint
http://127.0.0.1:18645/v1/chat/completions
Started
15 Nov 2026 17:02 UTC; last answer 15 Nov 2026 17:39 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
LFM3-1.7B, served with vLLM 0.34.0rc1.dev302+gbe47189f3 (nightly, commit be47189f37f6, CUDA 13.5 image; docker vllm/vllm-openai@sha256:7f0618da95bac2f1cc97fcd1791380d7c1a06d24a4998da45aeb7fbe6c49179e), BF16 from LiquidAI/LFM3-1.7B@0dc4268b561499c648c7ba7c9466bf3d8d01517a, vLLM reasoning parser qwen3 (seed 0 of three on the RTX PRO 6000 at 32 in flight (Umit #9150/#9151); the published thinking-off run (lfm3-1.7b-vllm-nothink_en) ran on a DGX Spark, so it is not one of the three), one RTX PRO 6000 on our GPU host alone; thinking off (enable_thinking false; on/off only); sampling temperature 0.1, top_k 50, repetition_penalty 1.1, top_p 1, min_p 0, from generation_config.json, passed 1:1 as vLLM sampling params; seed 0; max_tokens 32,768; one question per request (32 requests in flight), standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 1.73B / 1.73B active / total (dense; BF16 safetensors 1,728,801,536, text only; card: 1.7B). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "repetition_penalty": 1.1,
  "seed": 0,
  "temperature": 0.1,
  "top_k": 50,
  "top_p": 1.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · minicpm5-2b-nothink-seed1_en

Name
MiniCPM5-2B off · chat
Setup
MiniCPM5-2B (chat, thinking off, on our GPU) · chat model
Model
minicpm5-2b-nothink · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.4544 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.182 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.454. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-15 17:02:34–17:46:02 UTC
GPU time
run minicpm5-2b-nothink-seed1_en · first answer 2026-11-15 17:02:23 UTC · last answer 2026-11-15 17:56:48 UTC · span 3265.0 s, pauses included (not charged) · charged 657.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.4544 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 29.5 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 1 answer cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 2.52B dense · from the run's notes (dense; checkpoint 2,516,756,480 incl. untied lm_head, text only; card: 2,516,756,480); also https://huggingface.co/openbmb/MiniCPM5-2B · read 2026-11-12 · counted from the checkpoint (2,516,756,480 incl. untied lm_head, text only; dense); the card: 2,516,756,480
Input
text, English
Endpoint
http://127.0.0.1:18640/v1/chat/completions
Started
15 Nov 2026 17:02 UTC; last answer 15 Nov 2026 17:56 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
MiniCPM5-2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from openbmb/MiniCPM5-2B@42433e90dad95123ab87729fd7011e7110629556, vLLM reasoning parser qwen3 (seed 1 of three (Umit #9150/#9151: seeds 0, 1, 2 on the RTX PRO 6000 at ~30 in flight; seed 0 is minicpm5-2b-nothink_en, which ran there at 32 sent / server 16); otherwise its settings), one RTX PRO 6000 on our GPU host alone; thinking off (enable_thinking false; on/off only, no levels); sampling temperature 1, top_p 0.95, from generation_config.json; seed 1; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 2.52B / 2.52B active / total (dense; checkpoint 2,516,756,480 incl. untied lm_head, text only; card: 2,516,756,480). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "max_tokens": 32768,
  "seed": 1,
  "temperature": 1.0,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · lfm3-1.7b-vllm-nothink-rtx-seed1_en

Name
LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell
Setup
LFM3-1.7B (chat, thinking off, RTX PRO 6000, on our GPU) · chat model
Model
lfm3-1.7b-vllm-nothink · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.2919 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.117 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.292. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-15 17:55:58–18:24:41 UTC
GPU time
run lfm3-1.7b-vllm-nothink-rtx-seed1_en · first answer 2026-11-15 17:55:41 UTC · last answer 2026-11-15 18:31:26 UTC · span 2145.0 s, pauses included (not charged) · charged 422.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.2919 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 29.4 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 1.73B dense · from the run's notes (dense; BF16 safetensors 1,728,801,536, text only; card: 1.7B); also https://huggingface.co/LiquidAI/LFM3-1.7B-GGUF · read 2026-11-09 · the released checkpoint's tensors without registered buffers: 1,728,801,536 (the served BF16 GGUF also stores a 64-element RoPE buffer, 1,728,801,600); text only; card: 1.7B
Input
text, English
Endpoint
http://127.0.0.1:18645/v1/chat/completions
Started
15 Nov 2026 17:55 UTC; last answer 15 Nov 2026 18:31 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
LFM3-1.7B, served with vLLM 0.34.0rc1.dev302+gbe47189f3 (nightly, commit be47189f37f6, CUDA 13.5 image; docker vllm/vllm-openai@sha256:7f0618da95bac2f1cc97fcd1791380d7c1a06d24a4998da45aeb7fbe6c49179e), BF16 from LiquidAI/LFM3-1.7B@0dc4268b561499c648c7ba7c9466bf3d8d01517a, vLLM reasoning parser qwen3 (seed 1 of three on the RTX PRO 6000 at 32 in flight (Umit #9150/#9151); the published thinking-off run (lfm3-1.7b-vllm-nothink_en) ran on a DGX Spark, so it is not one of the three), one RTX PRO 6000 on our GPU host alone; thinking off (enable_thinking false; on/off only); sampling temperature 0.1, top_k 50, repetition_penalty 1.1, top_p 1, min_p 0, from generation_config.json, passed 1:1 as vLLM sampling params; seed 1; max_tokens 32,768; one question per request (32 requests in flight), standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 1.73B / 1.73B active / total (dense; BF16 safetensors 1,728,801,536, text only; card: 1.7B). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "repetition_penalty": 1.1,
  "seed": 1,
  "temperature": 0.1,
  "top_k": 50,
  "top_p": 1.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · minicpm5-2b-nothink-seed2_en

Name
MiniCPM5-2B off · chat
Setup
MiniCPM5-2B (chat, thinking off, on our GPU) · chat model
Model
minicpm5-2b-nothink · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.4856 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.195 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.486. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-15 18:07:24–18:43:40 UTC
GPU time
run minicpm5-2b-nothink-seed2_en · first answer 2026-11-15 18:06:51 UTC · last answer 2026-11-15 18:54:49 UTC · span 2878.0 s, pauses included (not charged) · charged 702.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.4856 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 29.4 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 1 answer cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 2.52B dense · from the run's notes (dense; checkpoint 2,516,756,480 incl. untied lm_head, text only; card: 2,516,756,480); also https://huggingface.co/openbmb/MiniCPM5-2B · read 2026-11-12 · counted from the checkpoint (2,516,756,480 incl. untied lm_head, text only; dense); the card: 2,516,756,480
Input
text, English
Endpoint
http://127.0.0.1:18640/v1/chat/completions
Started
15 Nov 2026 18:06 UTC; last answer 15 Nov 2026 18:54 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
MiniCPM5-2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from openbmb/MiniCPM5-2B@42433e90dad95123ab87729fd7011e7110629556, vLLM reasoning parser qwen3 (seed 2 of three (Umit #9150/#9151: seeds 0, 1, 2 on the RTX PRO 6000 at ~30 in flight; seed 0 is minicpm5-2b-nothink_en, which ran there at 32 sent / server 16); otherwise its settings), one RTX PRO 6000 on our GPU host alone; thinking off (enable_thinking false; on/off only, no levels); sampling temperature 1, top_p 0.95, from generation_config.json; seed 2; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 2.52B / 2.52B active / total (dense; checkpoint 2,516,756,480 incl. untied lm_head, text only; card: 2,516,756,480). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "max_tokens": 32768,
  "seed": 2,
  "temperature": 1.0,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · lfm3-1.7b-vllm-nothink-rtx-seed2_en

Name
LFM3-1.7B off · chat · NVIDIA RTX PRO 6000 Blackwell
Setup
LFM3-1.7B (chat, thinking off, RTX PRO 6000, on our GPU) · chat model
Model
lfm3-1.7b-vllm-nothink · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.3099 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.124 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.310. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-15 18:39:28–19:09:46 UTC
GPU time
run lfm3-1.7b-vllm-nothink-rtx-seed2_en · first answer 2026-11-15 18:39:09 UTC · last answer 2026-11-15 19:16:55 UTC · span 2266.0 s, pauses included (not charged) · charged 448.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.3099 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 29.7 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 1.73B dense · from the run's notes (dense; BF16 safetensors 1,728,801,536, text only; card: 1.7B); also https://huggingface.co/LiquidAI/LFM3-1.7B-GGUF · read 2026-11-09 · the released checkpoint's tensors without registered buffers: 1,728,801,536 (the served BF16 GGUF also stores a 64-element RoPE buffer, 1,728,801,600); text only; card: 1.7B
Input
text, English
Endpoint
http://127.0.0.1:18645/v1/chat/completions
Started
15 Nov 2026 18:39 UTC; last answer 15 Nov 2026 19:16 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
LFM3-1.7B, served with vLLM 0.34.0rc1.dev302+gbe47189f3 (nightly, commit be47189f37f6, CUDA 13.5 image; docker vllm/vllm-openai@sha256:7f0618da95bac2f1cc97fcd1791380d7c1a06d24a4998da45aeb7fbe6c49179e), BF16 from LiquidAI/LFM3-1.7B@0dc4268b561499c648c7ba7c9466bf3d8d01517a, vLLM reasoning parser qwen3 (seed 2 of three on the RTX PRO 6000 at 32 in flight (Umit #9150/#9151); the published thinking-off run (lfm3-1.7b-vllm-nothink_en) ran on a DGX Spark, so it is not one of the three), one RTX PRO 6000 on our GPU host alone; thinking off (enable_thinking false; on/off only); sampling temperature 0.1, top_k 50, repetition_penalty 1.1, top_p 1, min_p 0, from generation_config.json, passed 1:1 as vLLM sampling params; seed 2; max_tokens 32,768; one question per request (32 requests in flight), standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 1.73B / 1.73B active / total (dense; BF16 safetensors 1,728,801,536, text only; card: 1.7B). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "max_tokens": 32768,
  "min_p": 0.0,
  "repetition_penalty": 1.1,
  "seed": 2,
  "temperature": 0.1,
  "top_k": 50,
  "top_p": 1.0
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · knowline-9b-gen2_en

Name
KnowLine-9B-Gen2
Setup
KnowLine-9B-Gen2 · decision model
Model
PelaAI/KnowLine-9B-Gen2 · provider knowline
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.06709 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0269 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.067. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run knowline-9b-gen2_en · first answer 2026-11-16 10:20:46 UTC · last answer 2026-11-16 10:22:23 UTC · span 97.0 s, pauses included (not charged) · charged 97.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0671 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/PelaAI/KnowLine-9B-Gen2 · read 2026-11-16 · the maker's figure (9B) is shown, as for every model (turkce #4263); this checkpoint not counted
Input
text, English
Endpoint
http://127.0.0.1:18461
Started
16 Nov 2026 10:20 UTC; last answer 16 Nov 2026 10:22 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup KnowLine-9B-Gen2 (site run None); runner 88e48d7a1; PelaAI/KnowLine-9B-Gen2@8ec8eb609bc10aa6050ae3754aee419d458fae42 (Apache-2.0; built on Qwen3.5-9B), served by compactor on GPU 3 of our GPU host (his booking r_d665bb34a8c2) with the card's serve_knowline.sh (SGLang, then knowline_server.py, card defaults, read twice) in the lmsysorg/sglang image with SGLang 0.5.23, as KnowLine-9B-Gen1; the card names SGLang 0.5.21 (compactor #9859). Logic & Math on the same load as Answer Check and Reading (Umit #9814)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "PelaAI/KnowLine-9B-Gen2",
    "--provider",
    "knowline",
    "--timeout",
    "120",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "PelaAI/KnowLine-9B-Gen2",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "knowline",
  "questions": 2163,
  "request_seconds": 88.36948855317314,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "1f5130673514f55287cec838d483e5c4a0780dd703fa01cb205f6bb228e16c18",
  "runner": "benchmark_ollama.py",
  "runner_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 924182,
    "output_tokens": 2163
  }
}
Prompt template
POST http://127.0.0.1:18461/v1/systemone
{
 "model": "PelaAI/KnowLine-9B-Gen2",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · knowline-9b-gen2_en_dontknow

Name
KnowLine-9B-Gen2
Setup
KnowLine-9B-Gen2 · decision model · with the I don't know option F
Model
PelaAI/KnowLine-9B-Gen2 · provider knowline
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.0664 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0267 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.066. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run knowline-9b-gen2_en_dontknow · first answer 2026-11-16 10:23:37 UTC · last answer 2026-11-16 10:25:13 UTC · span 96.0 s, pauses included (not charged) · charged 96.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0664 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 9B dense · source https://huggingface.co/PelaAI/KnowLine-9B-Gen2 · read 2026-11-16 · the maker's figure (9B) is shown, as for every model (turkce #4263); this checkpoint not counted
Input
text, English
Endpoint
http://127.0.0.1:18461
Started
16 Nov 2026 10:23 UTC; last answer 16 Nov 2026 10:25 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup KnowLine-9B-Gen2 (site run None); runner 88e48d7a1; PelaAI/KnowLine-9B-Gen2@8ec8eb609bc10aa6050ae3754aee419d458fae42 (Apache-2.0; built on Qwen3.5-9B), served by compactor on GPU 3 of our GPU host (his booking r_d665bb34a8c2) with the card's serve_knowline.sh (SGLang, then knowline_server.py, card defaults, read twice) in the lmsysorg/sglang image with SGLang 0.5.23, as KnowLine-9B-Gen1; the card names SGLang 0.5.21 (compactor #9859). Logic & Math on the same load as Answer Check and Reading (Umit #9814)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "PelaAI/KnowLine-9B-Gen2",
    "--provider",
    "knowline",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "PelaAI/KnowLine-9B-Gen2",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "knowline",
  "questions": 2163,
  "request_seconds": 87.01432495203335,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "580fd87135c9d72af6ca102c45d6af99b412c9d2f35fb2d3c29882a752c8b24a",
  "runner": "benchmark_ollama.py",
  "runner_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 1053962,
    "output_tokens": 2163
  }
}
Prompt template
POST http://127.0.0.1:18461/v1/systemone
{
 "model": "PelaAI/KnowLine-9B-Gen2",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · knowline-12b-gen1_en

Name
KnowLine-12B-Gen1
Setup
KnowLine-12B-Gen1 · decision model
Model
PelaAI/KnowLine-12B-Gen1 · provider knowline
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.07816 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0314 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.078. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run knowline-12b-gen1_en · first answer 2026-11-16 10:24:02 UTC · last answer 2026-11-16 10:25:55 UTC · span 113.0 s, pauses included (not charged) · charged 113.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0782 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 12B dense · source https://huggingface.co/PelaAI/KnowLine-12B-Gen1 · read 2026-11-16 · the maker's figure (12B) is shown, as for every model (turkce #4263); this checkpoint not counted
Input
text, English
Endpoint
http://127.0.0.1:18462
Started
16 Nov 2026 10:24 UTC; last answer 16 Nov 2026 10:25 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup KnowLine-12B-Gen1 (site run None); runner 88e48d7a1; PelaAI/KnowLine-12B-Gen1@f26fbc2d64092bb2f8caaedbec243172bb3e908f (Apache-2.0; built on Gemma 4 12B), served by compactor on GPU 2 of our GPU host (his booking r_298d21956009) with the card's serve_knowline.sh (SGLang, then knowline_server.py, card defaults, read twice) in the lmsysorg/sglang image with SGLang 0.5.23; the card names SGLang 0.5.21 (compactor #9859). Logic & Math on the same load as Answer Check and Reading (Umit #9814)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "PelaAI/KnowLine-12B-Gen1",
    "--provider",
    "knowline",
    "--timeout",
    "120",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "PelaAI/KnowLine-12B-Gen1",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "knowline",
  "questions": 2163,
  "request_seconds": 103.76845989571302,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "aa628ffea0657e4803a04505ace0adff60d52347dd9d9aa1ef61f3fc114f4b64",
  "runner": "benchmark_ollama.py",
  "runner_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 925509,
    "output_tokens": 2163
  }
}
Prompt template
POST http://127.0.0.1:18462/v1/systemone
{
 "model": "PelaAI/KnowLine-12B-Gen1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · knowline-12b-gen1_en_dontknow

Name
KnowLine-12B-Gen1
Setup
KnowLine-12B-Gen1 · decision model · with the I don't know option F
Model
PelaAI/KnowLine-12B-Gen1 · provider knowline
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.08784 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0353 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.088. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run knowline-12b-gen1_en_dontknow · first answer 2026-11-16 10:27:30 UTC · last answer 2026-11-16 10:29:37 UTC · span 127.0 s, pauses included (not charged) · charged 127.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0878 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 12B dense · source https://huggingface.co/PelaAI/KnowLine-12B-Gen1 · read 2026-11-16 · the maker's figure (12B) is shown, as for every model (turkce #4263); this checkpoint not counted
Input
text, English
Endpoint
http://127.0.0.1:18462
Started
16 Nov 2026 10:27 UTC; last answer 16 Nov 2026 10:29 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup KnowLine-12B-Gen1 (site run None); runner 88e48d7a1; PelaAI/KnowLine-12B-Gen1@f26fbc2d64092bb2f8caaedbec243172bb3e908f (Apache-2.0; built on Gemma 4 12B), served by compactor on GPU 2 of our GPU host (his booking r_298d21956009) with the card's serve_knowline.sh (SGLang, then knowline_server.py, card defaults, read twice) in the lmsysorg/sglang image with SGLang 0.5.23; the card names SGLang 0.5.21 (compactor #9859). Logic & Math on the same load as Answer Check and Reading (Umit #9814)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "PelaAI/KnowLine-12B-Gen1",
    "--provider",
    "knowline",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "PelaAI/KnowLine-12B-Gen1",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "knowline",
  "questions": 2163,
  "request_seconds": 118.46864381173509,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "eda6a479b08c8240d6580d448fa7a0c45c0a515416fb56b59dad878fa0eaa4ce",
  "runner": "benchmark_ollama.py",
  "runner_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 1068267,
    "output_tokens": 2163
  }
}
Prompt template
POST http://127.0.0.1:18462/v1/systemone
{
 "model": "PelaAI/KnowLine-12B-Gen1",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · knowline-31b-gen2_en

Name
KnowLine-31B-Gen2
Setup
KnowLine-31B-Gen2 · decision model
Model
PelaAI/KnowLine-31B-Gen2 · provider knowline
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.1798 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0722 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.180. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run knowline-31b-gen2_en · first answer 2026-11-16 10:43:24 UTC · last answer 2026-11-16 10:47:44 UTC · span 260.0 s, pauses included (not charged) · charged 260.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1798 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 31B dense · source https://huggingface.co/PelaAI/KnowLine-31B-Gen2 · read 2026-11-16 · the maker's figure (31B) is shown, as for every model (turkce #4263); this checkpoint not counted
Input
text, English
Endpoint
http://127.0.0.1:18463
Started
16 Nov 2026 10:43 UTC; last answer 16 Nov 2026 10:47 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup KnowLine-31B-Gen2 (site run None); runner 88e48d7a1; PelaAI/KnowLine-31B-Gen2@fd99a9ed2255d8a67dd47a44cc429e638f611593 (Apache-2.0; built on Gemma 4 31B), served by compactor on GPU 1 of our GPU host with the card's serve_knowline.sh (SGLang, then knowline_server.py, card defaults, read twice) in the lmsysorg/sglang image with SGLang 0.5.23; the card names SGLang 0.5.21 (compactor #9859). Logic & Math on the same load as Answer Check and Reading (Umit #9814)
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "PelaAI/KnowLine-31B-Gen2",
    "--provider",
    "knowline",
    "--timeout",
    "120",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "local",
  "model": "PelaAI/KnowLine-31B-Gen2",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "47f5776e71db",
  "provider": "knowline",
  "questions": 2163,
  "request_seconds": 240.78494618460536,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "32ae4de6fb931f8f8d91f81e9da6fc866cfe8b00e517fb20ae46711dd921b7ad",
  "runner": "benchmark_ollama.py",
  "runner_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 925509,
    "output_tokens": 2163
  }
}
Prompt template
POST http://127.0.0.1:18463/v1/systemone
{
 "model": "PelaAI/KnowLine-31B-Gen2",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · knowline-31b-gen2_en_dontknow

Name
KnowLine-31B-Gen2
Setup
KnowLine-31B-Gen2 · decision model · with the I don't know option F
Model
PelaAI/KnowLine-31B-Gen2 · provider knowline
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.1951 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0783 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.195. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-16 10:54:05–11:06:17 UTC
GPU time
run knowline-31b-gen2_en_dontknow · first answer 2026-11-16 10:51:21 UTC · last answer 2026-11-16 11:08:15 UTC · span 1014.0 s, pauses included (not charged) · charged 282.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.1951 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 0.9 in flight (sum of request latencies / active time)
Model size
Parameters: 31B dense · source https://huggingface.co/PelaAI/KnowLine-31B-Gen2 · read 2026-11-16 · the maker's figure (31B) is shown, as for every model (turkce #4263); this checkpoint not counted
Input
text, English
Endpoint
http://127.0.0.1:18463
Started
16 Nov 2026 10:51 UTC; last answer 16 Nov 2026 11:08 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup KnowLine-31B-Gen2 (site run None); runner 88e48d7a1; PelaAI/KnowLine-31B-Gen2@fd99a9ed2255d8a67dd47a44cc429e638f611593 (Apache-2.0; built on Gemma 4 31B), served by compactor on GPU 1 of our GPU host with the card's serve_knowline.sh (SGLang, then knowline_server.py, card defaults, read twice) in the lmsysorg/sglang image with SGLang 0.5.23; the card names SGLang 0.5.21 (compactor #9859). Logic & Math on the same load as Answer Check and Reading (Umit #9814) RESUMED 2026-11-16T11:06Z: the run was cut at 1,305 of 2,163 stored answers by our GPU host's node fault and reboot (GPU 1 server lost ~10:52 UTC, compactor #9957); the same pin, image and serve script were reloaded on the same GPU as a new process after the 11:00 UTC reboot, and only the missing questions were asked.
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "PelaAI/KnowLine-31B-Gen2",
    "--provider",
    "knowline",
    "--timeout",
    "120",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "local",
  "model": "PelaAI/KnowLine-31B-Gen2",
  "perplexity_mode": null,
  "price": null,
  "prompt_version": "413d6775fc58",
  "provider": "knowline",
  "questions": 2163,
  "request_seconds": 264.8658379879032,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "1fb96b0e2873da9b8dcb37dbc1e5ea7369c321875f30c5862c46beb4cddf303d",
  "runner": "benchmark_ollama.py",
  "runner_commit": "88e48d7a105591ae2fdc820271c91095643d401d",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 120.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 1068267,
    "output_tokens": 2163
  }
}
Prompt template
POST http://127.0.0.1:18463/v1/systemone
{
 "model": "PelaAI/KnowLine-31B-Gen2",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · inkling-2-small_en

Name
Inkling 2 Small high · chat
Setup
Inkling 2 Small · chat model
Model
thinkingmachines/inkling-2-small · provider openrouter
GPU
—
Cost
$1.381 for the run · provider price × reported usage
Answer time
run inkling-2-small_en · first answer 2026-11-16 15:26:49 UTC · last answer 2026-11-16 18:45:06 UTC · span 11897.0 s, pauses included (not charged) · charged 1249.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 7.4 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 2 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 594B MoE · 17B active · from the run's notes (Thinking Machines (Hugging Face card)); also https://huggingface.co/thinkingmachines/Inkling-2-Small · read 2026-11-16 · the card at 2ad1157016e1aac77b6ff1b770e7c05229403134: '594B total, 17B active'
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
16 Nov 2026 15:26 UTC; last answer 16 Nov 2026 18:45 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Inkling 2 Small through OpenRouter (thinkingmachines/inkling-2-small), provider pinned to nebius/fp4, fallbacks off; reasoning {"effort": "high"}; temperature 1.0; max_tokens 32768; one question per request; parameters 17B / 594B active / total (Thinking Machines (Hugging Face card)). English translations.
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "nebius/fp4"
    ]
  },
  "reasoning": {
    "effort": "high"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · qwen4-max_en

Name
Qwen4-Max high · chat
Setup
Qwen4-Max · chat model
Model
qwen/qwen4-max · provider openrouter
GPU
—
Cost
$3.197 for the run · provider price × reported usage
Answer time
run qwen4-max_en · first answer 2026-11-16 15:26:51 UTC · last answer 2026-11-16 18:02:23 UTC · span 9332.0 s, pauses included (not charged) · charged 1587.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 8.0 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: not disclosed by maker (closed weights) · source https://openrouter.ai/qwen/qwen4-max · read 2026-11-16
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
16 Nov 2026 15:26 UTC; last answer 16 Nov 2026 18:02 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Qwen4-Max through OpenRouter (qwen/qwen4-max), provider pinned to alibaba, fallbacks off; reasoning {"effort": "high"}; temperature 1.0; max_tokens 32768; one question per request; parameters not disclosed (closed weights). English translations.
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "alibaba"
    ]
  },
  "reasoning": {
    "effort": "high"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · gemma-4-e2b-it-think-32k-sampled-seed0_en

Name
Gemma 4 E2B on · chat · 32k cap · sampled
Setup
Gemma 4 E2B (chat, thinking on, 32k, sampled, on our GPU) · chat model
Model
gemma-4-e2b-it-think-32k-sampled-seed0 · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.4219 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.169 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.422. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-16 15:50:46–16:09:06 UTC
GPU time
run gemma-4-e2b-it-think-32k-sampled-seed0_en · first answer 2026-11-16 15:50:30 UTC · last answer 2026-11-16 16:19:00 UTC · span 1710.0 s, pauses included (not charged) · charged 610.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.4219 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 30.7 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 5.1B dense · 2.3B effective · source https://huggingface.co/google/gemma-4-E2B-it · read 2026-11-10 · the card states 2.3B effective (5.1B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 5.12B (counted from the checkpoint (5,123,178,979: language model 4.65B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 5.1B with embeddings, 2.3B effective; not a mixture of experts)
Input
text, English
Endpoint
http://127.0.0.1:18670/v1/chat/completions
Started
16 Nov 2026 15:50 UTC; last answer 16 Nov 2026 16:19 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Gemma 4 E2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from google/gemma-4-E2B-it@4adf15748de9e1a0097b6274cee9c15836c41c37, vLLM reasoning parser gemma4 (seed 0 of three at the maker's sampling (Umit #10533, logic #10541); the published temperature-0 run is not one of them), one RTX PRO 6000 on our GPU host alone; thinking on (chat_template_kwargs enable_thinking true, the template's own switch: it opens a system turn with <|think|>; the template default is off; on/off only, no levels); sampling temperature 1, top_p 0.95, top_k 64, from generation_config.json (temperature 1.0, top_p 0.95, top_k 64); seed 0; max_tokens 32,768; one question per request (32 requests in flight), standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 5.12B / 5.12B active / total (dense; checkpoint 5,123,178,979 incl. vision 0.17B and audio 0.31B towers; card: 5.1B with per-layer embeddings, 2.3B effective). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "seed": 0,
  "temperature": 1.0,
  "top_k": 64,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · gemma-4-e4b-it-think-32k-sampled-seed0_en

Name
Gemma 4 E4B on · chat · 32k cap · sampled
Setup
Gemma 4 E4B (chat, thinking on, 32k, sampled, on our GPU) · chat model
Model
gemma-4-e4b-it-think-32k-sampled-seed0 · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.7034 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.282 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.703. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-16 16:07:02–16:36:52 UTC
GPU time
run gemma-4-e4b-it-think-32k-sampled-seed0_en · first answer 2026-11-16 16:06:12 UTC · last answer 2026-11-16 16:52:59 UTC · span 2807.0 s, pauses included (not charged) · charged 1017.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.7034 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 29.1 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 8B dense · 4.5B effective · source https://huggingface.co/google/gemma-4-E4B-it · read 2026-11-10 · the card states 4.5B effective (8B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 8.00B (counted from the checkpoint (7,996,157,418: language model 7.52B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 8B with embeddings, 4.5B effective; not a mixture of experts)
Input
text, English
Endpoint
http://127.0.0.1:18671/v1/chat/completions
Started
16 Nov 2026 16:06 UTC; last answer 16 Nov 2026 16:52 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Gemma 4 E4B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from google/gemma-4-E4B-it@4ecc2ba63bcac23ba4a767fc87025b1450f84c9c, vLLM reasoning parser gemma4 (seed 0 of three at the maker's sampling (Umit #10533, logic #10541); the published temperature-0 run is not one of them), one RTX PRO 6000 on our GPU host alone; thinking on (chat_template_kwargs enable_thinking true, the template's own switch: it opens a system turn with <|think|>; the template default is off; on/off only, no levels); sampling temperature 1, top_p 0.95, top_k 64, from generation_config.json (temperature 1.0, top_p 0.95, top_k 64); seed 0; max_tokens 32,768; one question per request (32 requests in flight), standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 8.00B / 8.00B active / total (dense; checkpoint 7,996,157,418 incl. vision 0.17B and audio 0.31B towers; card: 8B with per-layer embeddings, 4.5B effective). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "seed": 0,
  "temperature": 1.0,
  "top_k": 64,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · gemma-4-e2b-it-think-32k-sampled-seed0_en_dontknow

Name
Gemma 4 E2B on · chat · 32k cap · sampled
Setup
Gemma 4 E2B (chat, thinking on, 32k, sampled, on our GPU) · chat model · with the I don't know option F
Model
gemma-4-e2b-it-think-32k-sampled-seed0 · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.4309 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.173 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.431. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gemma-4-e2b-it-think-32k-sampled-seed0_en_dontknow · first answer 2026-11-16 16:40:20 UTC · last answer 2026-11-16 16:50:43 UTC · span 623.0 s, pauses included (not charged) · charged 623.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.4309 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 30.7 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 5.1B dense · 2.3B effective · source https://huggingface.co/google/gemma-4-E2B-it · read 2026-11-10 · the card states 2.3B effective (5.1B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 5.12B (counted from the checkpoint (5,123,178,979: language model 4.65B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 5.1B with embeddings, 2.3B effective; not a mixture of experts)
Input
text, English
Endpoint
http://127.0.0.1:18670/v1/chat/completions
Started
16 Nov 2026 16:40 UTC; last answer 16 Nov 2026 16:50 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Gemma 4 E2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from google/gemma-4-E2B-it@4adf15748de9e1a0097b6274cee9c15836c41c37, vLLM reasoning parser gemma4 (seed 0 of three at the maker's sampling (Umit #10533, logic #10541); the published temperature-0 run is not one of them), one RTX PRO 6000 on our GPU host alone; thinking on (chat_template_kwargs enable_thinking true, the template's own switch: it opens a system turn with <|think|>; the template default is off; on/off only, no levels); sampling temperature 1, top_p 0.95, top_k 64, from generation_config.json (temperature 1.0, top_p 0.95, top_k 64); seed 0; max_tokens 32,768; one question per request (32 requests in flight), standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 5.12B / 5.12B active / total (dense; checkpoint 5,123,178,979 incl. vision 0.17B and audio 0.31B towers; card: 5.1B with per-layer embeddings, 2.3B effective). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "seed": 0,
  "temperature": 1.0,
  "top_k": 64,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · gemma-4-e2b-it-think-32k-sampled-seed1_en

Name
Gemma 4 E2B on · chat · 32k cap · sampled
Setup
Gemma 4 E2B (chat, thinking on, 32k, sampled, on our GPU) · chat model
Model
gemma-4-e2b-it-think-32k-sampled-seed1 · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.4233 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.17 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.423. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-16 16:59:52–17:18:51 UTC
GPU time
run gemma-4-e2b-it-think-32k-sampled-seed1_en · first answer 2026-11-16 16:59:31 UTC · last answer 2026-11-16 17:28:42 UTC · span 1751.0 s, pauses included (not charged) · charged 612.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.4233 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.1 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 5.1B dense · 2.3B effective · source https://huggingface.co/google/gemma-4-E2B-it · read 2026-11-10 · the card states 2.3B effective (5.1B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 5.12B (counted from the checkpoint (5,123,178,979: language model 4.65B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 5.1B with embeddings, 2.3B effective; not a mixture of experts)
Input
text, English
Endpoint
http://127.0.0.1:18670/v1/chat/completions
Started
16 Nov 2026 16:59 UTC; last answer 16 Nov 2026 17:28 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Gemma 4 E2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from google/gemma-4-E2B-it@4adf15748de9e1a0097b6274cee9c15836c41c37, vLLM reasoning parser gemma4 (seed 1 of three at the maker's sampling (Umit #10533, logic #10541); the published temperature-0 run is not one of them), one RTX PRO 6000 on our GPU host alone; thinking on (chat_template_kwargs enable_thinking true, the template's own switch: it opens a system turn with <|think|>; the template default is off; on/off only, no levels); sampling temperature 1, top_p 0.95, top_k 64, from generation_config.json (temperature 1.0, top_p 0.95, top_k 64); seed 1; max_tokens 32,768; one question per request (32 requests in flight), standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 5.12B / 5.12B active / total (dense; checkpoint 5,123,178,979 incl. vision 0.17B and audio 0.31B towers; card: 5.1B with per-layer embeddings, 2.3B effective). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "seed": 1,
  "temperature": 1.0,
  "top_k": 64,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · gemma-4-e4b-it-think-32k-sampled-seed0_en_dontknow

Name
Gemma 4 E4B on · chat · 32k cap · sampled
Setup
Gemma 4 E4B (chat, thinking on, 32k, sampled, on our GPU) · chat model · with the I don't know option F
Model
gemma-4-e4b-it-think-32k-sampled-seed0 · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.635 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.255 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.635. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run gemma-4-e4b-it-think-32k-sampled-seed0_en_dontknow · first answer 2026-11-16 17:29:14 UTC · last answer 2026-11-16 17:44:32 UTC · span 918.0 s, pauses included (not charged) · charged 918.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.6350 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 31.1 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 8B dense · 4.5B effective · source https://huggingface.co/google/gemma-4-E4B-it · read 2026-11-10 · the card states 4.5B effective (8B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 8.00B (counted from the checkpoint (7,996,157,418: language model 7.52B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 8B with embeddings, 4.5B effective; not a mixture of experts)
Input
text, English
Endpoint
http://127.0.0.1:18671/v1/chat/completions
Started
16 Nov 2026 17:29 UTC; last answer 16 Nov 2026 17:44 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Gemma 4 E4B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from google/gemma-4-E4B-it@4ecc2ba63bcac23ba4a767fc87025b1450f84c9c, vLLM reasoning parser gemma4 (seed 0 of three at the maker's sampling (Umit #10533, logic #10541); the published temperature-0 run is not one of them), one RTX PRO 6000 on our GPU host alone; thinking on (chat_template_kwargs enable_thinking true, the template's own switch: it opens a system turn with <|think|>; the template default is off; on/off only, no levels); sampling temperature 1, top_p 0.95, top_k 64, from generation_config.json (temperature 1.0, top_p 0.95, top_k 64); seed 0; max_tokens 32,768; one question per request (32 requests in flight), standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 8.00B / 8.00B active / total (dense; checkpoint 7,996,157,418 incl. vision 0.17B and audio 0.31B towers; card: 8B with per-layer embeddings, 4.5B effective). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "seed": 0,
  "temperature": 1.0,
  "top_k": 64,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · gemma-4-e2b-it-think-32k-sampled-seed2_en

Name
Gemma 4 E2B on · chat · 32k cap · sampled
Setup
Gemma 4 E2B (chat, thinking on, 32k, sampled, on our GPU) · chat model
Model
gemma-4-e2b-it-think-32k-sampled-seed2 · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.4254 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.171 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.425. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-16 17:37:05–17:55:27 UTC
GPU time
run gemma-4-e2b-it-think-32k-sampled-seed2_en · first answer 2026-11-16 17:36:45 UTC · last answer 2026-11-16 18:05:22 UTC · span 1717.0 s, pauses included (not charged) · charged 615.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.4254 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 30.1 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 5.1B dense · 2.3B effective · source https://huggingface.co/google/gemma-4-E2B-it · read 2026-11-10 · the card states 2.3B effective (5.1B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 5.12B (counted from the checkpoint (5,123,178,979: language model 4.65B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 5.1B with embeddings, 2.3B effective; not a mixture of experts)
Input
text, English
Endpoint
http://127.0.0.1:18670/v1/chat/completions
Started
16 Nov 2026 17:36 UTC; last answer 16 Nov 2026 18:05 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Gemma 4 E2B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from google/gemma-4-E2B-it@4adf15748de9e1a0097b6274cee9c15836c41c37, vLLM reasoning parser gemma4 (seed 2 of three at the maker's sampling (Umit #10533, logic #10541); the published temperature-0 run is not one of them), one RTX PRO 6000 on our GPU host alone; thinking on (chat_template_kwargs enable_thinking true, the template's own switch: it opens a system turn with <|think|>; the template default is off; on/off only, no levels); sampling temperature 1, top_p 0.95, top_k 64, from generation_config.json (temperature 1.0, top_p 0.95, top_k 64); seed 2; max_tokens 32,768; one question per request (32 requests in flight), standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 5.12B / 5.12B active / total (dense; checkpoint 5,123,178,979 incl. vision 0.17B and audio 0.31B towers; card: 5.1B with per-layer embeddings, 2.3B effective). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "seed": 2,
  "temperature": 1.0,
  "top_k": 64,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · gemma-4-e4b-it-think-32k-sampled-seed1_en

Name
Gemma 4 E4B on · chat · 32k cap · sampled
Setup
Gemma 4 E4B (chat, thinking on, 32k, sampled, on our GPU) · chat model
Model
gemma-4-e4b-it-think-32k-sampled-seed1 · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.6737 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.271 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.674. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-16 17:52:46–18:23:05 UTC
GPU time
run gemma-4-e4b-it-think-32k-sampled-seed1_en · first answer 2026-11-16 17:52:01 UTC · last answer 2026-11-16 18:38:34 UTC · span 2793.0 s, pauses included (not charged) · charged 974.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.6737 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 30.6 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 8B dense · 4.5B effective · source https://huggingface.co/google/gemma-4-E4B-it · read 2026-11-10 · the card states 4.5B effective (8B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 8.00B (counted from the checkpoint (7,996,157,418: language model 7.52B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 8B with embeddings, 4.5B effective; not a mixture of experts)
Input
text, English
Endpoint
http://127.0.0.1:18671/v1/chat/completions
Started
16 Nov 2026 17:51 UTC; last answer 16 Nov 2026 18:38 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Gemma 4 E4B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from google/gemma-4-E4B-it@4ecc2ba63bcac23ba4a767fc87025b1450f84c9c, vLLM reasoning parser gemma4 (seed 1 of three at the maker's sampling (Umit #10533, logic #10541); the published temperature-0 run is not one of them), one RTX PRO 6000 on our GPU host alone; thinking on (chat_template_kwargs enable_thinking true, the template's own switch: it opens a system turn with <|think|>; the template default is off; on/off only, no levels); sampling temperature 1, top_p 0.95, top_k 64, from generation_config.json (temperature 1.0, top_p 0.95, top_k 64); seed 1; max_tokens 32,768; one question per request (32 requests in flight), standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 8.00B / 8.00B active / total (dense; checkpoint 7,996,157,418 incl. vision 0.17B and audio 0.31B towers; card: 8B with per-layer embeddings, 4.5B effective). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "seed": 1,
  "temperature": 1.0,
  "top_k": 64,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · qwen4-max_en_dontknow

Name
Qwen4-Max high · chat
Setup
Qwen4-Max · chat model · with the I don't know option F
Model
qwen/qwen4-max · provider openrouter
GPU
—
Cost
$3.104 for the run · provider price × reported usage
Answer time
run qwen4-max_en_dontknow · first answer 2026-11-16 18:02:28 UTC · last answer 2026-11-16 18:28:30 UTC · span 1562.0 s, pauses included (not charged) · charged 1562.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 8.0 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: not disclosed by maker (closed weights) · source https://openrouter.ai/qwen/qwen4-max · read 2026-11-16
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
16 Nov 2026 18:02 UTC; last answer 16 Nov 2026 18:28 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Qwen4-Max through OpenRouter (qwen/qwen4-max), provider pinned to alibaba, fallbacks off; reasoning {"effort": "high"}; temperature 1.0; max_tokens 32768; one question per request; parameters not disclosed (closed weights). English translations.
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "alibaba"
    ]
  },
  "reasoning": {
    "effort": "high"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · inkling-2-small_en_dontknow

Name
Inkling 2 Small high · chat
Setup
Inkling 2 Small · chat model · with the I don't know option F
Model
thinkingmachines/inkling-2-small · provider openrouter
GPU
—
Cost
$1.39 for the run · provider price × reported usage
Answer time
run inkling-2-small_en_dontknow · first answer 2026-11-16 18:45:08 UTC · last answer 2026-11-16 19:06:47 UTC · span 1299.0 s, pauses included (not charged) · charged 1299.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 7.8 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 1 answer cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 594B MoE · 17B active · from the run's notes (Thinking Machines (Hugging Face card)); also https://huggingface.co/thinkingmachines/Inkling-2-Small · read 2026-11-16 · the card at 2ad1157016e1aac77b6ff1b770e7c05229403134: '594B total, 17B active'
Input
text, English
Endpoint
https://openrouter.ai/api/v1/chat/completions
Started
16 Nov 2026 18:45 UTC; last answer 16 Nov 2026 19:06 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Inkling 2 Small through OpenRouter (thinkingmachines/inkling-2-small), provider pinned to nebius/fp4, fallbacks off; reasoning {"effort": "high"}; temperature 1.0; max_tokens 32768; one question per request; parameters 17B / 594B active / total (Thinking Machines (Hugging Face card)). English translations.
Parameters
{
  "max_tokens": 32768,
  "provider": {
    "allow_fallbacks": false,
    "order": [
      "nebius/fp4"
    ]
  },
  "reasoning": {
    "effort": "high"
  },
  "temperature": 1.0,
  "usage": {
    "include": true
  }
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}

Math · gemma-4-e4b-it-think-32k-sampled-seed2_en

Name
Gemma 4 E4B on · chat · 32k cap · sampled
Setup
Gemma 4 E4B (chat, thinking on, 32k, sampled, on our GPU) · chat model
Model
gemma-4-e4b-it-think-32k-sampled-seed2 · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.6841 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.275 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.684. GPU time = first to last stored answer, less gaps of more than 10 minutes; pauses left out: 2026-11-16 18:47:29–19:16:49 UTC
GPU time
run gemma-4-e4b-it-think-32k-sampled-seed2_en · first answer 2026-11-16 18:46:19 UTC · last answer 2026-11-16 19:32:08 UTC · span 2749.0 s, pauses included (not charged) · charged 989.0 s in 2 stretchs (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.6841 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 29.9 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 8B dense · 4.5B effective · source https://huggingface.co/google/gemma-4-E4B-it · read 2026-11-10 · the card states 4.5B effective (8B with embeddings) (Ümit/jevgammon #4260: the maker's figure is shown); checkpoint count 8.00B (counted from the checkpoint (7,996,157,418: language model 7.52B with per-layer embeddings, vision tower 0.17B, audio tower 0.31B); the card: 8B with embeddings, 4.5B effective; not a mixture of experts)
Input
text, English
Endpoint
http://127.0.0.1:18671/v1/chat/completions
Started
16 Nov 2026 18:46 UTC; last answer 16 Nov 2026 19:32 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Gemma 4 E4B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from google/gemma-4-E4B-it@4ecc2ba63bcac23ba4a767fc87025b1450f84c9c, vLLM reasoning parser gemma4 (seed 2 of three at the maker's sampling (Umit #10533, logic #10541); the published temperature-0 run is not one of them), one RTX PRO 6000 on our GPU host alone; thinking on (chat_template_kwargs enable_thinking true, the template's own switch: it opens a system turn with <|think|>; the template default is off; on/off only, no levels); sampling temperature 1, top_p 0.95, top_k 64, from generation_config.json (temperature 1.0, top_p 0.95, top_k 64); seed 2; max_tokens 32,768; one question per request (32 requests in flight), standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 8.00B / 8.00B active / total (dense; checkpoint 7,996,157,418 incl. vision 0.17B and audio 0.31B towers; card: 8B with per-layer embeddings, 4.5B effective). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "max_tokens": 32768,
  "seed": 2,
  "temperature": 1.0,
  "top_k": 64,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · d1.2-ultra_en

Name
d1.2-ultra
Setup
Liquid d1.2-ultra · decision model
Model
d1.2-ultra · provider liquid
GPU
—
Cost
$0.01328 for the run · provider price × reported usage
Answer time
run d1.2-ultra_en · first answer 2026-11-16 19:03:15 UTC · last answer 2026-11-16 19:12:40 UTC · span 565.0 s, pauses included (not charged) · charged 565.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://www.liquid.ai/blog/d1-2 · read 2026-11-16
Input
text, English
Endpoint
https://api.liquid.ai/decisions
Started
16 Nov 2026 19:03 UTC; last answer 16 Nov 2026 19:12 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Liquid d1.2-ultra (site run None); runner 1cc2ce5d9; Liquid's hosted decision API, released 16 November 2026 (liquid.ai/blog/d1-2); run at once on Umit's word, whatever the cost (#10926, zeki #10928), one request at a time as d1-ultra ran
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "d1.2-ultra",
    "--provider",
    "liquid",
    "--timeout",
    "30",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "0f7123a347725d64b71f99d55f89d66f32fa9eb0",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "d1.2-ultra",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Liquid AI's d1.2-ultra price",
    "date": "2026-11-16",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.042,
      "cache_write_input_tokens": 0.042,
      "cached_input_tokens": 0.042,
      "input_tokens": 0.042,
      "output_tokens": 0.0
    },
    "source": "https://www.liquid.ai/blog/d1-2",
    "text": "$0.042 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "liquid",
  "questions": 2163,
  "request_seconds": 559.7850853279342,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "482ab74707f80ad934c37e834dfdae5c7a8c62ef07fb96c31c1fd1e0d74735db",
  "runner": "benchmark_ollama.py",
  "runner_commit": "1cc2ce5d9e9148092d004abf953cbbbffa12110a",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 316269,
    "output_tokens": 0
  }
}
Prompt template
POST https://api.liquid.ai/decisions/v1/systemone
{
 "model": "d1.2-ultra",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · d1.2-ultra_en_dontknow

Name
d1.2-ultra
Setup
Liquid d1.2-ultra · decision model · with the I don't know option F
Model
d1.2-ultra · provider liquid
GPU
—
Cost
$0.01569 for the run · provider price × reported usage
Answer time
run d1.2-ultra_en_dontknow · first answer 2026-11-16 19:18:01 UTC · last answer 2026-11-16 19:27:38 UTC · span 577.0 s, pauses included (not charged) · charged 577.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://www.liquid.ai/blog/d1-2 · read 2026-11-16
Input
text, English
Endpoint
https://api.liquid.ai/decisions
Started
16 Nov 2026 19:18 UTC; last answer 16 Nov 2026 19:27 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Liquid d1.2-ultra (site run None); runner 1cc2ce5d9; Liquid's hosted decision API, released 16 November 2026 (liquid.ai/blog/d1-2); run at once on Umit's word, whatever the cost (#10926, zeki #10928), one request at a time as d1-ultra ran
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "d1.2-ultra",
    "--provider",
    "liquid",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "0f7123a347725d64b71f99d55f89d66f32fa9eb0",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "d1.2-ultra",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Liquid AI's d1.2-ultra price",
    "date": "2026-11-16",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.042,
      "cache_write_input_tokens": 0.042,
      "cached_input_tokens": 0.042,
      "input_tokens": 0.042,
      "output_tokens": 0.0
    },
    "source": "https://www.liquid.ai/blog/d1-2",
    "text": "$0.042 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "liquid",
  "questions": 2163,
  "request_seconds": 564.4860392139199,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "8f9e7b0bfc204c7725060dd1ae825cfb15596b42366c62c30b454af469755a7e",
  "runner": "benchmark_ollama.py",
  "runner_commit": "1cc2ce5d9e9148092d004abf953cbbbffa12110a",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 373535,
    "output_tokens": 0
  }
}
Prompt template
POST https://api.liquid.ai/decisions/v1/systemone
{
 "model": "d1.2-ultra",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · d1.2_en

Name
d1.2
Setup
Liquid d1.2 · decision model
Model
d1.2 · provider liquid
GPU
—
Cost
$0.006325 for the run · provider price × reported usage
Answer time
run d1.2_en · first answer 2026-11-16 19:37:30 UTC · last answer 2026-11-16 19:46:38 UTC · span 548.0 s, pauses included (not charged) · charged 548.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://www.liquid.ai/blog/d1-2 · read 2026-11-16
Input
text, English
Endpoint
https://api.liquid.ai/decisions
Started
16 Nov 2026 19:37 UTC; last answer 16 Nov 2026 19:46 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Liquid d1.2 (site run None); runner 1cc2ce5d9; Liquid's hosted decision API, released 16 November 2026 (liquid.ai/blog/d1-2); run at once on Umit's word, both models whatever the cost (#10926), one request at a time as d1.1 ran
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "d1.2",
    "--provider",
    "liquid",
    "--timeout",
    "30",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": false,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "0f7123a347725d64b71f99d55f89d66f32fa9eb0",
  "instructions": "Which option is the correct answer to this English multiple-choice question?",
  "kind": "api",
  "model": "d1.2",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Liquid AI's d1.2 price",
    "date": "2026-11-16",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.02,
      "cache_write_input_tokens": 0.02,
      "cached_input_tokens": 0.02,
      "input_tokens": 0.02,
      "output_tokens": 0.0
    },
    "source": "https://www.liquid.ai/blog/d1-2",
    "text": "$0.02 input, output free per 1M tokens"
  },
  "prompt_version": "47f5776e71db",
  "provider": "liquid",
  "questions": 2163,
  "request_seconds": 531.9973838349615,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "61b75aa3b18b27de3187f540468e5fca337db42056b8b2310b5124bb51ae8e1c",
  "runner": "benchmark_ollama.py",
  "runner_commit": "1cc2ce5d9e9148092d004abf953cbbbffa12110a",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 316269,
    "output_tokens": 0
  }
}
Prompt template
POST https://api.liquid.ai/decisions/v1/systemone
{
 "model": "d1.2",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question?",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}"
   }
  }
 }
}
([...] only when the question has a context)

Math · granite-4.2-3b-nothink_en

Name
Granite-4.2-3B off · chat
Setup
Granite-4.2-3B (chat, thinking off, on our GPU) · chat model
Model
granite-4.2-3b-nothink · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.05741 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0231 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.057. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run granite-4.2-3b-nothink_en · first answer 2026-11-16 19:47:21 UTC · last answer 2026-11-16 19:48:44 UTC · span 83.0 s, pauses included (not charged) · charged 83.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0574 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 2.3 in flight (sum of request latencies / active time)
Output limit
32,768 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 3B dense · source https://huggingface.co/ibm-granite/granite-4.2-3b · read 2026-11-12 · the card states 3B (the maker's figure is shown); checkpoint count 3.66B (3,659,737,600 incl. untied lm_head, text only; dense)
Input
text, English
Endpoint
http://127.0.0.1:18641/v1/chat/completions
Started
16 Nov 2026 19:47 UTC; last answer 16 Nov 2026 19:48 UTC
Progress
25 answered of 2,163 asked; 0 request errors; smoke test at 32,768 tokens only; the full run uses a higher limit (run granite-4.2-3b-nothink-64k_en)
Notes
Granite-4.2-3B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from ibm-granite/granite-4.2-3b@4e9c275e232f10211ec88644d106a1f6a1b60e06, vLLM reasoning parser granite_thinking_parser, one RTX PRO 6000 on our GPU host alone; thinking off (enable_thinking false, the template's own switch); sampling temperature 1, top_p 0.95, from generation_config.json; seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 3.66B / 3.66B active / total (dense; checkpoint 3,659,737,600 incl. untied lm_head, text only; card: 3B). English translations (DeepSeek + review). SUPERSEDED by granite-4.2-3b-nothink-64k_en (cut-offs only; one retry at 65,536).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "max_tokens": 32768,
  "seed": 0,
  "temperature": 1.0,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · d1.2_en_dontknow

Name
d1.2
Setup
Liquid d1.2 · decision model · with the I don't know option F
Model
d1.2 · provider liquid
GPU
—
Cost
$0.007471 for the run · provider price × reported usage
Answer time
run d1.2_en_dontknow · first answer 2026-11-16 19:51:23 UTC · last answer 2026-11-16 19:59:58 UTC · span 515.0 s, pauses included (not charged) · charged 515.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out)
Requests in flight
≈ 1.0 in flight (sum of request latencies / active time)
Model size
Parameters: not disclosed by maker · source https://www.liquid.ai/blog/d1-2 · read 2026-11-16
Input
text, English
Endpoint
https://api.liquid.ai/decisions
Started
16 Nov 2026 19:51 UTC; last answer 16 Nov 2026 19:59 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Decision model, published setup Liquid d1.2 (site run None); runner 1cc2ce5d9; Liquid's hosted decision API, released 16 November 2026 (liquid.ai/blog/d1-2); run at once on Umit's word, both models whatever the cost (#10926), one request at a time as d1.1 ran
Parameters
{
  "bank": "math_en.sqlite",
  "bank_sha256": "5ae5d81916a347f75aecd53f7b9a8f77518164845b344d54bdea605a092a7a33",
  "batch_size": 1,
  "command": [
    "d1.2",
    "--provider",
    "liquid",
    "--timeout",
    "30",
    "--dont-know",
    "--english",
    "banks/math_en.jsonl",
    "--questions",
    "banks/math_all.txt"
  ],
  "concurrency": 1,
  "dont_know": true,
  "english_sha256": "c0b080e95f4eb8d61ed1d77447bf461f918eb4870678fe928b993984b20f4d61",
  "harness_commit": "0f7123a347725d64b71f99d55f89d66f32fa9eb0",
  "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
  "kind": "api",
  "model": "d1.2",
  "perplexity_mode": null,
  "price": {
    "applies_to": "billed",
    "basis": "Liquid AI's d1.2 price",
    "date": "2026-11-16",
    "extra_input_per_request": 0,
    "per_request": 0.0,
    "rates_per_1m": {
      "cache_write_1h_input_tokens": 0.02,
      "cache_write_input_tokens": 0.02,
      "cached_input_tokens": 0.02,
      "input_tokens": 0.02,
      "output_tokens": 0.0
    },
    "source": "https://www.liquid.ai/blog/d1-2",
    "text": "$0.02 input, output free per 1M tokens"
  },
  "prompt_version": "413d6775fc58",
  "provider": "liquid",
  "questions": 2163,
  "request_seconds": 500.0808357080459,
  "requests": 2163,
  "results_db": "(local file, not published)",
  "run_key": "ff45233555b0068a676640c0666d7dfaac88e5ed848dc03079e8804874788ee5",
  "runner": "benchmark_ollama.py",
  "runner_commit": "1cc2ce5d9e9148092d004abf953cbbbffa12110a",
  "runner_run_id": 2,
  "selection_sha256": "31614a3d3234dcf437f693c611a53ed05cee2bf1e8ad7596cdacb74a2b9e2c6d",
  "site_run": null,
  "status": "completed",
  "timeout": 30.0,
  "tokens": {
    "cache_write_1h_input_tokens": 0,
    "cache_write_input_tokens": 0,
    "cached_input_tokens": 0,
    "input_tokens": 373535,
    "output_tokens": 0
  }
}
Prompt template
POST https://api.liquid.ai/decisions/v1/systemone
{
 "model": "d1.2",
 "state": "[Shared text / context:\n{context}\n\n]{question}\n\nOptions:\nA. {option_A}\nB. {option_B}\nC. {option_C}\nD. {option_D}\nE. {option_E}\nF. I don't know",
 "questions": {
  "answer": {
   "type": "choice",
   "instructions": "Which option is the correct answer to this English multiple-choice question? If you do not know the answer, choose I don't know.",
   "criteria": {
    "A": "{option_A}",
    "B": "{option_B}",
    "C": "{option_C}",
    "D": "{option_D}",
    "E": "{option_E}",
    "F": "I don't know"
   }
  }
 }
}
([...] only when the question has a context)

Math · granite-4.2-3b-nothink-64k_en

Name
Granite-4.2-3B off · chat
Setup
Granite-4.2-3B (chat, thinking off, on our GPU) · chat model
Model
granite-4.2-3b-nothink · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$0.07193 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 0.0289 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $0.072. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes
GPU time
run granite-4.2-3b-nothink-64k_en · first answer 2026-11-16 20:00:25 UTC · last answer 2026-11-16 20:02:09 UTC · span 104.0 s, pauses included (not charged) · charged 104.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $0.0719 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)
Requests in flight
≈ 2.2 in flight (sum of request latencies / active time)
Output limit
65,536 tokens · 0 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 3B dense · source https://huggingface.co/ibm-granite/granite-4.2-3b · read 2026-11-12 · the card states 3B (the maker's figure is shown); checkpoint count 3.66B (3,659,737,600 incl. untied lm_head, text only; dense)
Input
text, English
Endpoint
http://127.0.0.1:18641/v1/chat/completions
Started
16 Nov 2026 20:00 UTC; last answer 16 Nov 2026 20:02 UTC
Progress
25 answered of 2,163 asked; 0 request errors; in progress
Notes
Granite-4.2-3B, served with vLLM v0.30.0 (docker vllm/vllm-openai:v0.30.0), BF16 from ibm-granite/granite-4.2-3b@4e9c275e232f10211ec88644d106a1f6a1b60e06, vLLM reasoning parser granite_thinking_parser, one RTX PRO 6000 on our GPU host alone; thinking off (enable_thinking false, the template's own switch); sampling temperature 1, top_p 0.95, from generation_config.json; seed 0; max_tokens 65,536 (raised from 32,768 once: the smoke failed on cut-offs only); one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 3.66B / 3.66B active / total (dense; checkpoint 3,659,737,600 incl. untied lm_head, text only; card: 3B). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false
  },
  "max_tokens": 65536,
  "seed": 0,
  "temperature": 1.0,
  "top_p": 0.95
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D or E).

{question}

Math · raon-1-nano-nothink_en_dontknow

Name
Raon-1 nano off · chat
Setup
Raon-1 nano (chat, thinking off, KRAFTON vLLM fork, on our GPU) · chat model · with the I don't know option F
Model
raon-1-nano-nothink · provider vllm
GPU
NVIDIA RTX PRO 6000 Blackwell (96 GB)
Cost
$2.502 for the run, RTX PRO 6000 Blackwell · estimate: NVIDIA RTX PRO 6000 Blackwell (96 GB) time while the run was running, first to last answer × $2.49/h · GPU time, at most: 1 h × $2.49/h on an NVIDIA RTX PRO 6000 Blackwell (96 GB) = $2.502. GPU time = first to last stored answer, less gaps of more than 10 minutes; no pauses over 10 minutes†
GPU time
run raon-1-nano-nothink_en_dontknow · first answer 2026-11-16 21:05:06 UTC · last answer 2026-11-16 22:05:23 UTC · span 3617.0 s, pauses included (not charged) · charged 3617.0 s in 1 stretch (estimated from the answer times: a gap of more than 10 minutes between answers is taken as the run being stopped and resumed, and left out) · GPU-time cost $2.5018 for the run (charged seconds × $2.49/GPU-hour × 1 NVIDIA RTX PRO 6000 Blackwell (96 GB), at most)†
Requests in flight
≈ 27.4 in flight (sum of request latencies / active time)†
Output limit
32,768 tokens · 32 answers cut off at it with no letter (counted wrong) · 0 empty replies asked once more
Model size
Parameters: 3.1B dense · 1.5B effective · source https://huggingface.co/KRAFTON/Raon-1-nano · read 2026-11-13 · the card states 1.5B effective parameters (3.1B total, including a 1.5B per-layer embedding table) (the maker's figure is shown); checkpoint count 3.13B (3,130,797,328: per-layer embedding table 1.60B, token embeddings 0.40B tied, layers 1.12B; plus a materialised copy of the tied lm_head 0.40B and the MTP module 0.05B in separate files; dense)
Input
text, English
Endpoint
http://127.0.0.1:18637/v1/chat/completions
Started
16 Nov 2026 21:05 UTC; last answer 16 Nov 2026 22:05 UTC
Progress
2,163 answered of 2,163 asked; 0 request errors; complete
Notes
Raon-1 nano, served with KRAFTON's vLLM fork krafton-ai/vllm@13a5dd87 (branch raon-1-nano, precompiled; not a vLLM release) from KRAFTON/Raon-1-nano@fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee, vLLM reasoning parser raon_1 (the card lists Korean and English only; cost and speed are on another engine than the stock-vLLM rows), one RTX PRO 6000 on our GPU host alone; thinking off (enable_thinking false; the card: 'designed for thinking mode; disabling it is not recommended'); sampling temperature 0.1, presence_penalty 1.5, sampling from the model card, generation_config has none ('General' row); seed 0; max_tokens 32,768; one question per request, standard chat prompt with ANSWER: X on the last line, no JSON schema; parameters 1.5B / 3.1B active / total (card: 1.5B effective, 3.1B total incl. a 1.5B per-layer embedding table; MTP module 55M not loaded). English translations (DeepSeek + review).
Parameters
{
  "chat_template_kwargs": {
    "enable_thinking": false,
    "truncate_history_thinking": false
  },
  "max_tokens": 32768,
  "presence_penalty": 1.5,
  "seed": 0,
  "temperature": 0.1
}
Prompt template
Solve the following multiple-choice question. On the last line of your reply write only 'ANSWER: X' (X = A, B, C, D, E or F). If you do not know the answer, choose I don't know.

{question}