Tasks

Each task is a short request from a user. The model sees the request and a list of tools, decides whether and how to call them, gets the tool results back and answers. A task passes only when the calls, their arguments and order, and the resulting account state are right; on restraint tasks, only when no tool is called. 42 tasks per track, 7 of each type, 3 repetitions each.

The 8 finance tools

Decoy tools

22 tools that have nothing to do with finance (get_weather, list_calendar, search_email, translate_text, set_alarm, find_recipe, …). Some selection and restraint tasks offer all 30 tools, the 8 finance tools and the decoys together. They test whether the model still picks the right tool from a long list, and leaves an unrelated tool alone just because it is there.

The six task types

Renamed tools

Every task also exists in a renamed version: the same request and the same tools, but every tool and argument has a neutral name (get_price becomes fetch_quote, symbol becomes ticker). A model that only recognises familiar names fails there; a model that reads the tool descriptions does not. Both versions are scored, and the Renamed tools tab shows them side by side.

Synthetic only

All data are fixed synthetic fixtures (prices, balances, news), "today" is 2026-11-12, and orders are dry runs on a paper account. Nothing touches a market or a real account, and no real trade is ever made.

Not published

The tasks themselves, their expected answers and the models' replies are not published, so models cannot be trained on them. The tool definitions and each task's pass or fail (by task id) are. The "Public" and "Private" columns split the scores by an older public/private task split. Tool definitions · Tool definitions, renamed · Per-task results (pass/fail by task id)

Hard set

A second, harder set of 24 tasks per track (also in a renamed version) tests longer chains and stricter instruction following. A task that says "in this order" is graded in that order; an order call sent together with other calls in one message counts as a format failure, because the model must see each order result first. Same tools, fixtures and grading; 4 tasks of each type:

Per-task results (pass/fail by task id)

Toolbench

Results · 42 tasks per track · 3 repetitions · original/renamed are pairedHard set results · 24 tasks per track · 3 repetitions · original/renamed are paired

Standard set · 42 tasks · All

Task set
Class
Measure

Accuracy Task pass rate — higher is better.

Tiers within each size class: for each pair of setups, every task (42 per track, public and private) is one paired item, won by the setup that passed more of its 3 repetitions; ties drop out. Exact two-sided sign test, Holm-corrected over all pairs of the class, p < 0.05. Rows in accuracy order: the top row starts Tier 1, and a row starts the next tier when the test separates it from its tier's top row. Accuracy: both tracks (84 tasks); Renamed tools: each track (42 tasks). – : not tiered (class not verified, incomplete or not qualified).ModelAccuracyPublicPrivate
Tiny
1Gemma 4 E2B onOpen · 5.1BDetails94.0%94.1%93.9%
1Granite 4.2 3B onOpen · 3.0BDetails92.9%90.3%100.0%
1LFM3-1.7B onOpen · 1.7BDetails91.7%91.4%92.4%
1Qwen3.5 4B offOpen · 4.7BDetails91.3%88.7%98.5%
1MiniCPM5-2B onOpen · 2.5BDetails90.5%87.1%100.0%
1Qwen3.5 4B onOpen · 4.7BDetails89.7%86.0%100.0%
1Gemma 4 E2B (tool-result cue template) offOpen · 5.1BDetails89.3%85.5%100.0%
1Raon-1 nano onOpen · 3.1BDetails88.9%85.5%98.5%
1Qwen3.5-2B onOpen · 2.0BDetails87.3%83.9%97.0%
1LFM3-1.7B offOpen · 1.7BDetails84.5%79.0%100.0%
2MiniCPM5-2B offOpen · 2.5BDetails81.3%76.3%95.5%
2Raon-1 nano offOpen · 3.1BDetails79.8%72.6%100.0%
2Granite 4.2 3B offOpen · 3.0BDetails79.4%75.8%89.4%
2Qwen3.5-2B offOpen · 2.0BDetails73.4%69.9%83.3%
–Gemma 4 E2B offOpen · 5.1BDetailsNot qualifiedNot scoredNot scored
Small
1Gemma 4 12B offOpen · 12BDetails94.4%93.5%97.0%
1Granite 4.2 8B onOpen · 8.0BDetails93.3%91.4%98.5%
1Gemma 4 12B onOpen · 12BDetails92.9%90.3%100.0%
2Granite 4.2 8B offOpen · 8.0BDetails83.7%83.3%84.8%
Medium
1Gemma 4 31B offOpen · 31BDetails100.0%100.0%100.0%
1Gemma 4 31B onOpen · 31BDetails98.8%99.5%97.0%
1Qwen3.8-27B offOpen · 27BDetails96.8%95.7%100.0%
1LFM2.5-24B-A2B onOpen · 24B MoE · 2.5B activeDetails92.9%90.9%98.5%
2Qwen3.8-27B onOpen · 27BDetails92.1%89.8%98.5%

No row of this kind yet.

Task pass rate averaged over the preregistered repetitions. A pass requires the expected tool calls, arguments, order and mock-state checks. Verified serving or template failures remain unscored, not model zeroes.

Tiers within each size class: for each pair of setups, every task (42 per track, public and private) is one paired item, won by the setup that passed more of its 3 repetitions; ties drop out. Exact two-sided sign test, Holm-corrected over all pairs of the class, p < 0.05. Rows in accuracy order: the top row starts Tier 1, and a row starts the next tier when the test separates it from its tier's top row. Accuracy: both tracks (84 tasks); Renamed tools: each track (42 tasks). – : not tiered (class not verified, incomplete or not qualified). Serving and template failures are unscored, not model zeroes.

Accuracy Task pass rate — higher is better.

Tiers within each size class: for each pair of setups, every task (24 per track, public and private) is one paired item, won by the setup that passed more of its 3 repetitions; ties drop out. Exact two-sided sign test, Holm-corrected over all pairs of the class, p < 0.05. Rows in accuracy order: the top row starts Tier 1, and a row starts the next tier when the test separates it from its tier's top row. Accuracy: both tracks (48 tasks); Renamed tools: each track (24 tasks). – : not tiered (class not verified, incomplete or not qualified).ModelAccuracyPublicPrivate
Tiny
1Qwen3.5 4B offOpen · 4.7BDetails89.6%88.0%94.4%
1Granite 4.2 3B onOpen · 3.0BDetails88.2%85.2%97.2%
1MiniCPM5-2B onOpen · 2.5BDetails83.3%84.3%80.6%
1Qwen3.5 4B onOpen · 4.7BDetails80.6%81.5%77.8%
1Gemma 4 E2B (tool-result cue template) offOpen · 5.1BDetails77.1%78.7%72.2%
1Gemma 4 E2B onOpen · 5.1BDetails73.6%69.4%86.1%
2Raon-1 nano onOpen · 3.1BDetails70.8%67.6%80.6%
2Qwen3.5-2B onOpen · 2.0BDetails68.1%66.7%72.2%
2LFM3-1.7B onOpen · 1.7BDetails64.6%59.3%80.6%
2Qwen3.5-2B offOpen · 2.0BDetails57.6%54.6%66.7%
3Granite 4.2 3B offOpen · 3.0BDetails40.3%43.5%30.6%
3LFM3-1.7B offOpen · 1.7BDetails29.2%31.5%22.2%
3MiniCPM5-2B offOpen · 2.5BDetails29.2%33.3%16.7%
3Raon-1 nano offOpen · 3.1BDetails25.0%26.9%19.4%
Small
1Gemma 4 12B onOpen · 12BDetails95.1%96.3%91.7%
–Granite 4.2 8B onOpen · 8.0BDetails91.6%91.6%91.7%
2Gemma 4 12B offOpen · 12BDetails71.5%73.1%66.7%
3Granite 4.2 8B offOpen · 8.0BDetails42.4%42.6%41.7%
Medium
1Gemma 4 31B onOpen · 31BDetails91.0%88.0%100.0%
1LFM2.5-24B-A2B onOpen · 24B MoE · 2.5B activeDetails90.3%88.9%94.4%
1Qwen3.8-27B onOpen · 27BDetails88.2%88.0%88.9%
1Qwen3.8-27B offOpen · 27BDetails74.3%70.4%86.1%
2Gemma 4 31B offOpen · 31BDetails64.6%58.3%83.3%

No row of this kind yet.

Task pass rate averaged over the preregistered repetitions. A pass requires the expected tool calls, arguments, order and mock-state checks. Verified serving or template failures remain unscored, not model zeroes.

Tiers within each size class: for each pair of setups, every task (24 per track, public and private) is one paired item, won by the setup that passed more of its 3 repetitions; ties drop out. Exact two-sided sign test, Holm-corrected over all pairs of the class, p < 0.05. Rows in accuracy order: the top row starts Tier 1, and a row starts the next tier when the test separates it from its tier's top row. Accuracy: both tracks (48 tasks); Renamed tools: each track (24 tasks). – : not tiered (class not verified, incomplete or not qualified). Serving and template failures are unscored, not model zeroes.

Accuracy by category Per-category pass rate — higher is better.

ModelAccuracy by categoryScored attemptsUnscored attempts
–LFM3-1.7B off · singleOpen · 1.7BDetails100.0%420
–LFM3-1.7B off · selectionOpen · 1.7BDetails64.3%420
–LFM3-1.7B off · argsOpen · 1.7BDetails100.0%420
–LFM3-1.7B off · multi-stepOpen · 1.7BDetails59.5%420
–LFM3-1.7B off · restraintOpen · 1.7BDetails100.0%420
–LFM3-1.7B off · recoveryOpen · 1.7BDetails83.3%420
–LFM3-1.7B on · singleOpen · 1.7BDetails100.0%420
–LFM3-1.7B on · selectionOpen · 1.7BDetails97.6%420
–LFM3-1.7B on · argsOpen · 1.7BDetails100.0%420
–LFM3-1.7B on · multi-stepOpen · 1.7BDetails76.2%420
–LFM3-1.7B on · restraintOpen · 1.7BDetails90.5%420
–LFM3-1.7B on · recoveryOpen · 1.7BDetails85.7%420
–Gemma 4 E2B on · singleOpen · 5.1BDetails100.0%420
–Gemma 4 E2B on · selectionOpen · 5.1BDetails85.7%420
–Gemma 4 E2B on · argsOpen · 5.1BDetails100.0%420
–Gemma 4 E2B on · multi-stepOpen · 5.1BDetails92.9%420
–Gemma 4 E2B on · restraintOpen · 5.1BDetails100.0%420
–Gemma 4 E2B on · recoveryOpen · 5.1BDetails85.7%420
–MiniCPM5-2B off · singleOpen · 2.5BDetails100.0%420
–MiniCPM5-2B off · selectionOpen · 2.5BDetails71.4%420
–MiniCPM5-2B off · argsOpen · 2.5BDetails100.0%420
–MiniCPM5-2B off · multi-stepOpen · 2.5BDetails38.1%420
–MiniCPM5-2B off · restraintOpen · 2.5BDetails97.6%420
–MiniCPM5-2B off · recoveryOpen · 2.5BDetails81.0%420
–MiniCPM5-2B on · singleOpen · 2.5BDetails100.0%420
–MiniCPM5-2B on · selectionOpen · 2.5BDetails71.4%420
–MiniCPM5-2B on · argsOpen · 2.5BDetails100.0%420
–MiniCPM5-2B on · multi-stepOpen · 2.5BDetails76.2%420
–MiniCPM5-2B on · restraintOpen · 2.5BDetails100.0%420
–MiniCPM5-2B on · recoveryOpen · 2.5BDetails95.2%420
–Qwen3.5-2B off · singleOpen · 2.0BDetails95.2%420
–Qwen3.5-2B off · selectionOpen · 2.0BDetails61.9%420
–Qwen3.5-2B off · argsOpen · 2.0BDetails88.1%420
–Qwen3.5-2B off · multi-stepOpen · 2.0BDetails45.2%420
–Qwen3.5-2B off · restraintOpen · 2.0BDetails83.3%420
–Qwen3.5-2B off · recoveryOpen · 2.0BDetails66.7%420
–Qwen3.5-2B on · singleOpen · 2.0BDetails100.0%420
–Qwen3.5-2B on · selectionOpen · 2.0BDetails81.0%420
–Qwen3.5-2B on · argsOpen · 2.0BDetails100.0%420
–Qwen3.5-2B on · multi-stepOpen · 2.0BDetails85.7%420
–Qwen3.5-2B on · restraintOpen · 2.0BDetails88.1%420
–Qwen3.5-2B on · recoveryOpen · 2.0BDetails69.0%420
–Qwen3.5 4B off · singleOpen · 4.7BDetails100.0%420
–Qwen3.5 4B off · selectionOpen · 4.7BDetails88.1%420
–Qwen3.5 4B off · argsOpen · 4.7BDetails100.0%420
–Qwen3.5 4B off · multi-stepOpen · 4.7BDetails100.0%420
–Qwen3.5 4B off · restraintOpen · 4.7BDetails95.2%420
–Qwen3.5 4B off · recoveryOpen · 4.7BDetails64.3%420
–Qwen3.5 4B on · singleOpen · 4.7BDetails100.0%420
–Qwen3.5 4B on · selectionOpen · 4.7BDetails95.2%420
–Qwen3.5 4B on · argsOpen · 4.7BDetails97.6%420
–Qwen3.5 4B on · multi-stepOpen · 4.7BDetails71.4%420
–Qwen3.5 4B on · restraintOpen · 4.7BDetails95.2%420
–Qwen3.5 4B on · recoveryOpen · 4.7BDetails78.6%420
–Raon-1 nano off · singleOpen · 3.1BDetails100.0%420
–Raon-1 nano off · selectionOpen · 3.1BDetails71.4%420
–Raon-1 nano off · argsOpen · 3.1BDetails95.2%420
–Raon-1 nano off · multi-stepOpen · 3.1BDetails57.1%420
–Raon-1 nano off · restraintOpen · 3.1BDetails76.2%420
–Raon-1 nano off · recoveryOpen · 3.1BDetails78.6%420
–Raon-1 nano on · singleOpen · 3.1BDetails97.6%420
–Raon-1 nano on · selectionOpen · 3.1BDetails85.7%420
–Raon-1 nano on · argsOpen · 3.1BDetails100.0%420
–Raon-1 nano on · multi-stepOpen · 3.1BDetails61.9%420
–Raon-1 nano on · restraintOpen · 3.1BDetails100.0%420
–Raon-1 nano on · recoveryOpen · 3.1BDetails88.1%420
–Gemma 4 E2B (tool-result cue template) off · singleOpen · 5.1BDetails100.0%420
–Gemma 4 E2B (tool-result cue template) off · selectionOpen · 5.1BDetails71.4%420
–Gemma 4 E2B (tool-result cue template) off · argsOpen · 5.1BDetails100.0%420
–Gemma 4 E2B (tool-result cue template) off · multi-stepOpen · 5.1BDetails73.8%420
–Gemma 4 E2B (tool-result cue template) off · restraintOpen · 5.1BDetails92.9%420
–Gemma 4 E2B (tool-result cue template) off · recoveryOpen · 5.1BDetails97.6%420
–Granite 4.2 3B off · singleOpen · 3.0BDetails100.0%420
–Granite 4.2 3B off · selectionOpen · 3.0BDetails59.5%420
–Granite 4.2 3B off · argsOpen · 3.0BDetails100.0%420
–Granite 4.2 3B off · multi-stepOpen · 3.0BDetails61.9%420
–Granite 4.2 3B off · restraintOpen · 3.0BDetails100.0%420
–Granite 4.2 3B off · recoveryOpen · 3.0BDetails54.8%420
–Granite 4.2 3B on · singleOpen · 3.0BDetails97.6%420
–Granite 4.2 3B on · selectionOpen · 3.0BDetails85.7%420
–Granite 4.2 3B on · argsOpen · 3.0BDetails100.0%420
–Granite 4.2 3B on · multi-stepOpen · 3.0BDetails100.0%420
–Granite 4.2 3B on · restraintOpen · 3.0BDetails100.0%420
–Granite 4.2 3B on · recoveryOpen · 3.0BDetails73.8%420
–Granite 4.2 8B off · singleOpen · 8.0BDetails100.0%420
–Granite 4.2 8B off · selectionOpen · 8.0BDetails76.2%420
–Granite 4.2 8B off · argsOpen · 8.0BDetails97.6%420
–Granite 4.2 8B off · multi-stepOpen · 8.0BDetails64.3%420
–Granite 4.2 8B off · restraintOpen · 8.0BDetails100.0%420
–Granite 4.2 8B off · recoveryOpen · 8.0BDetails64.3%420
–Granite 4.2 8B on · singleOpen · 8.0BDetails100.0%420
–Granite 4.2 8B on · selectionOpen · 8.0BDetails83.3%420
–Granite 4.2 8B on · argsOpen · 8.0BDetails100.0%420
–Granite 4.2 8B on · multi-stepOpen · 8.0BDetails95.2%420
–Granite 4.2 8B on · restraintOpen · 8.0BDetails100.0%420
–Granite 4.2 8B on · recoveryOpen · 8.0BDetails81.0%420
–Gemma 4 12B off · singleOpen · 12BDetails100.0%420
–Gemma 4 12B off · selectionOpen · 12BDetails100.0%420
–Gemma 4 12B off · argsOpen · 12BDetails100.0%420
–Gemma 4 12B off · multi-stepOpen · 12BDetails100.0%420
–Gemma 4 12B off · restraintOpen · 12BDetails100.0%420
–Gemma 4 12B off · recoveryOpen · 12BDetails66.7%420
–Gemma 4 12B on · singleOpen · 12BDetails100.0%420
–Gemma 4 12B on · selectionOpen · 12BDetails90.5%420
–Gemma 4 12B on · argsOpen · 12BDetails100.0%420
–Gemma 4 12B on · multi-stepOpen · 12BDetails97.6%420
–Gemma 4 12B on · restraintOpen · 12BDetails100.0%420
–Gemma 4 12B on · recoveryOpen · 12BDetails69.0%420
–LFM2.5-24B-A2B on · singleOpen · 24B MoE · 2.5B activeDetails100.0%420
–LFM2.5-24B-A2B on · selectionOpen · 24B MoE · 2.5B activeDetails88.1%420
–LFM2.5-24B-A2B on · argsOpen · 24B MoE · 2.5B activeDetails100.0%420
–LFM2.5-24B-A2B on · multi-stepOpen · 24B MoE · 2.5B activeDetails100.0%420
–LFM2.5-24B-A2B on · restraintOpen · 24B MoE · 2.5B activeDetails88.1%420
–LFM2.5-24B-A2B on · recoveryOpen · 24B MoE · 2.5B activeDetails81.0%420
–Qwen3.8-27B off · singleOpen · 27BDetails100.0%420
–Qwen3.8-27B off · selectionOpen · 27BDetails100.0%420
–Qwen3.8-27B off · argsOpen · 27BDetails100.0%420
–Qwen3.8-27B off · multi-stepOpen · 27BDetails88.1%420
–Qwen3.8-27B off · restraintOpen · 27BDetails100.0%420
–Qwen3.8-27B off · recoveryOpen · 27BDetails92.9%420
–Qwen3.8-27B on · singleOpen · 27BDetails100.0%420
–Qwen3.8-27B on · selectionOpen · 27BDetails97.6%420
–Qwen3.8-27B on · argsOpen · 27BDetails88.1%420
–Qwen3.8-27B on · multi-stepOpen · 27BDetails85.7%420
–Qwen3.8-27B on · restraintOpen · 27BDetails100.0%420
–Qwen3.8-27B on · recoveryOpen · 27BDetails81.0%420
–Gemma 4 31B off · singleOpen · 31BDetails100.0%420
–Gemma 4 31B off · selectionOpen · 31BDetails100.0%420
–Gemma 4 31B off · argsOpen · 31BDetails100.0%420
–Gemma 4 31B off · multi-stepOpen · 31BDetails100.0%420
–Gemma 4 31B off · restraintOpen · 31BDetails100.0%420
–Gemma 4 31B off · recoveryOpen · 31BDetails100.0%420
–Gemma 4 31B on · singleOpen · 31BDetails100.0%420
–Gemma 4 31B on · selectionOpen · 31BDetails100.0%420
–Gemma 4 31B on · argsOpen · 31BDetails100.0%420
–Gemma 4 31B on · multi-stepOpen · 31BDetails95.2%420
–Gemma 4 31B on · restraintOpen · 31BDetails100.0%420
–Gemma 4 31B on · recoveryOpen · 31BDetails97.6%420
–Gemma 4 E2B offOpen · 5.1BDetailsNot qualifiedNot scoredNot scored

No row of this kind yet.

Task pass rates shown separately for single, selection, arguments, multi-step, restraint and recovery tasks. These are categories of the same task bank, not independent benchmark banks.

Not tiered. Serving and template failures are unscored, not model zeroes.

Accuracy by category Per-category pass rate — higher is better.

ModelAccuracy by categoryScored attemptsUnscored attempts
–LFM3-1.7B off · long-chainOpen · 1.7BDetails0.0%240
–LFM3-1.7B off · computed-argsOpen · 1.7BDetails8.3%240
–LFM3-1.7B off · multi-constraintOpen · 1.7BDetails41.7%240
–LFM3-1.7B off · branchingOpen · 1.7BDetails75.0%240
–LFM3-1.7B off · cascading-recoveryOpen · 1.7BDetails37.5%240
–LFM3-1.7B off · long-outputOpen · 1.7BDetails12.5%240
–LFM3-1.7B on · long-chainOpen · 1.7BDetails50.0%240
–LFM3-1.7B on · computed-argsOpen · 1.7BDetails75.0%240
–LFM3-1.7B on · multi-constraintOpen · 1.7BDetails79.2%240
–LFM3-1.7B on · branchingOpen · 1.7BDetails100.0%240
–LFM3-1.7B on · cascading-recoveryOpen · 1.7BDetails58.3%240
–LFM3-1.7B on · long-outputOpen · 1.7BDetails25.0%240
–Gemma 4 E2B on · long-chainOpen · 5.1BDetails79.2%240
–Gemma 4 E2B on · computed-argsOpen · 5.1BDetails62.5%240
–Gemma 4 E2B on · multi-constraintOpen · 5.1BDetails83.3%240
–Gemma 4 E2B on · branchingOpen · 5.1BDetails100.0%240
–Gemma 4 E2B on · cascading-recoveryOpen · 5.1BDetails87.5%240
–Gemma 4 E2B on · long-outputOpen · 5.1BDetails29.2%240
–MiniCPM5-2B off · long-chainOpen · 2.5BDetails8.3%240
–MiniCPM5-2B off · computed-argsOpen · 2.5BDetails16.7%240
–MiniCPM5-2B off · multi-constraintOpen · 2.5BDetails8.3%240
–MiniCPM5-2B off · branchingOpen · 2.5BDetails66.7%240
–MiniCPM5-2B off · cascading-recoveryOpen · 2.5BDetails70.8%240
–MiniCPM5-2B off · long-outputOpen · 2.5BDetails4.2%240
–MiniCPM5-2B on · long-chainOpen · 2.5BDetails87.5%240
–MiniCPM5-2B on · computed-argsOpen · 2.5BDetails70.8%240
–MiniCPM5-2B on · multi-constraintOpen · 2.5BDetails91.7%240
–MiniCPM5-2B on · branchingOpen · 2.5BDetails100.0%240
–MiniCPM5-2B on · cascading-recoveryOpen · 2.5BDetails95.8%240
–MiniCPM5-2B on · long-outputOpen · 2.5BDetails54.2%240
–Qwen3.5-2B off · long-chainOpen · 2.0BDetails75.0%240
–Qwen3.5-2B off · computed-argsOpen · 2.0BDetails29.2%240
–Qwen3.5-2B off · multi-constraintOpen · 2.0BDetails62.5%240
–Qwen3.5-2B off · branchingOpen · 2.0BDetails83.3%240
–Qwen3.5-2B off · cascading-recoveryOpen · 2.0BDetails87.5%240
–Qwen3.5-2B off · long-outputOpen · 2.0BDetails8.3%240
–Qwen3.5-2B on · long-chainOpen · 2.0BDetails75.0%240
–Qwen3.5-2B on · computed-argsOpen · 2.0BDetails45.8%240
–Qwen3.5-2B on · multi-constraintOpen · 2.0BDetails83.3%240
–Qwen3.5-2B on · branchingOpen · 2.0BDetails100.0%240
–Qwen3.5-2B on · cascading-recoveryOpen · 2.0BDetails83.3%240
–Qwen3.5-2B on · long-outputOpen · 2.0BDetails20.8%240
–Qwen3.5 4B off · long-chainOpen · 4.7BDetails100.0%240
–Qwen3.5 4B off · computed-argsOpen · 4.7BDetails87.5%240
–Qwen3.5 4B off · multi-constraintOpen · 4.7BDetails95.8%240
–Qwen3.5 4B off · branchingOpen · 4.7BDetails100.0%240
–Qwen3.5 4B off · cascading-recoveryOpen · 4.7BDetails91.7%240
–Qwen3.5 4B off · long-outputOpen · 4.7BDetails62.5%240
–Qwen3.5 4B on · long-chainOpen · 4.7BDetails87.5%240
–Qwen3.5 4B on · computed-argsOpen · 4.7BDetails91.7%240
–Qwen3.5 4B on · multi-constraintOpen · 4.7BDetails87.5%240
–Qwen3.5 4B on · branchingOpen · 4.7BDetails95.8%240
–Qwen3.5 4B on · cascading-recoveryOpen · 4.7BDetails83.3%240
–Qwen3.5 4B on · long-outputOpen · 4.7BDetails37.5%240
–Raon-1 nano off · long-chainOpen · 3.1BDetails4.2%240
–Raon-1 nano off · computed-argsOpen · 3.1BDetails4.2%240
–Raon-1 nano off · multi-constraintOpen · 3.1BDetails8.3%240
–Raon-1 nano off · branchingOpen · 3.1BDetails45.8%240
–Raon-1 nano off · cascading-recoveryOpen · 3.1BDetails87.5%240
–Raon-1 nano off · long-outputOpen · 3.1BDetails0.0%240
–Raon-1 nano on · long-chainOpen · 3.1BDetails66.7%240
–Raon-1 nano on · computed-argsOpen · 3.1BDetails45.8%240
–Raon-1 nano on · multi-constraintOpen · 3.1BDetails95.8%240
–Raon-1 nano on · branchingOpen · 3.1BDetails100.0%240
–Raon-1 nano on · cascading-recoveryOpen · 3.1BDetails87.5%240
–Raon-1 nano on · long-outputOpen · 3.1BDetails29.2%240
–Gemma 4 E2B (tool-result cue template) off · long-chainOpen · 5.1BDetails75.0%240
–Gemma 4 E2B (tool-result cue template) off · computed-argsOpen · 5.1BDetails70.8%240
–Gemma 4 E2B (tool-result cue template) off · multi-constraintOpen · 5.1BDetails100.0%240
–Gemma 4 E2B (tool-result cue template) off · branchingOpen · 5.1BDetails100.0%240
–Gemma 4 E2B (tool-result cue template) off · cascading-recoveryOpen · 5.1BDetails95.8%240
–Gemma 4 E2B (tool-result cue template) off · long-outputOpen · 5.1BDetails20.8%240
–Granite 4.2 3B off · long-chainOpen · 3.0BDetails20.8%240
–Granite 4.2 3B off · computed-argsOpen · 3.0BDetails20.8%240
–Granite 4.2 3B off · multi-constraintOpen · 3.0BDetails58.3%240
–Granite 4.2 3B off · branchingOpen · 3.0BDetails66.7%240
–Granite 4.2 3B off · cascading-recoveryOpen · 3.0BDetails62.5%240
–Granite 4.2 3B off · long-outputOpen · 3.0BDetails12.5%240
–Granite 4.2 3B on · long-chainOpen · 3.0BDetails100.0%240
–Granite 4.2 3B on · computed-argsOpen · 3.0BDetails87.5%240
–Granite 4.2 3B on · multi-constraintOpen · 3.0BDetails87.5%240
–Granite 4.2 3B on · branchingOpen · 3.0BDetails100.0%240
–Granite 4.2 3B on · cascading-recoveryOpen · 3.0BDetails75.0%240
–Granite 4.2 3B on · long-outputOpen · 3.0BDetails79.2%240
–Granite 4.2 8B off · long-chainOpen · 8.0BDetails33.3%240
–Granite 4.2 8B off · computed-argsOpen · 8.0BDetails20.8%240
–Granite 4.2 8B off · multi-constraintOpen · 8.0BDetails37.5%240
–Granite 4.2 8B off · branchingOpen · 8.0BDetails75.0%240
–Granite 4.2 8B off · cascading-recoveryOpen · 8.0BDetails79.2%240
–Granite 4.2 8B off · long-outputOpen · 8.0BDetails8.3%240
–Granite 4.2 8B on · long-chainOpen · 8.0BDetails100.0%240
–Granite 4.2 8B on · computed-argsOpen · 8.0BDetails79.2%240
–Granite 4.2 8B on · multi-constraintOpen · 8.0BDetails100.0%240
–Granite 4.2 8B on · branchingOpen · 8.0BDetails100.0%240
–Granite 4.2 8B on · cascading-recoveryOpen · 8.0BDetails95.8%240
–Granite 4.2 8B on · long-outputOpen · 8.0BDetails73.9%231
–Gemma 4 12B off · long-chainOpen · 12BDetails100.0%240
–Gemma 4 12B off · computed-argsOpen · 12BDetails83.3%240
–Gemma 4 12B off · multi-constraintOpen · 12BDetails70.8%240
–Gemma 4 12B off · branchingOpen · 12BDetails75.0%240
–Gemma 4 12B off · cascading-recoveryOpen · 12BDetails100.0%240
–Gemma 4 12B off · long-outputOpen · 12BDetails0.0%240
–Gemma 4 12B on · long-chainOpen · 12BDetails100.0%240
–Gemma 4 12B on · computed-argsOpen · 12BDetails100.0%240
–Gemma 4 12B on · multi-constraintOpen · 12BDetails100.0%240
–Gemma 4 12B on · branchingOpen · 12BDetails100.0%240
–Gemma 4 12B on · cascading-recoveryOpen · 12BDetails100.0%240
–Gemma 4 12B on · long-outputOpen · 12BDetails70.8%240
–LFM2.5-24B-A2B on · long-chainOpen · 24B MoE · 2.5B activeDetails95.8%240
–LFM2.5-24B-A2B on · computed-argsOpen · 24B MoE · 2.5B activeDetails100.0%240
–LFM2.5-24B-A2B on · multi-constraintOpen · 24B MoE · 2.5B activeDetails91.7%240
–LFM2.5-24B-A2B on · branchingOpen · 24B MoE · 2.5B activeDetails100.0%240
–LFM2.5-24B-A2B on · cascading-recoveryOpen · 24B MoE · 2.5B activeDetails87.5%240
–LFM2.5-24B-A2B on · long-outputOpen · 24B MoE · 2.5B activeDetails66.7%240
–Qwen3.8-27B off · long-chainOpen · 27BDetails75.0%240
–Qwen3.8-27B off · computed-argsOpen · 27BDetails70.8%240
–Qwen3.8-27B off · multi-constraintOpen · 27BDetails100.0%240
–Qwen3.8-27B off · branchingOpen · 27BDetails75.0%240
–Qwen3.8-27B off · cascading-recoveryOpen · 27BDetails100.0%240
–Qwen3.8-27B off · long-outputOpen · 27BDetails25.0%240
–Qwen3.8-27B on · long-chainOpen · 27BDetails87.5%240
–Qwen3.8-27B on · computed-argsOpen · 27BDetails87.5%240
–Qwen3.8-27B on · multi-constraintOpen · 27BDetails100.0%240
–Qwen3.8-27B on · branchingOpen · 27BDetails95.8%240
–Qwen3.8-27B on · cascading-recoveryOpen · 27BDetails95.8%240
–Qwen3.8-27B on · long-outputOpen · 27BDetails62.5%240
–Gemma 4 31B off · long-chainOpen · 31BDetails75.0%240
–Gemma 4 31B off · computed-argsOpen · 31BDetails62.5%240
–Gemma 4 31B off · multi-constraintOpen · 31BDetails79.2%240
–Gemma 4 31B off · branchingOpen · 31BDetails75.0%240
–Gemma 4 31B off · cascading-recoveryOpen · 31BDetails83.3%240
–Gemma 4 31B off · long-outputOpen · 31BDetails12.5%240
–Gemma 4 31B on · long-chainOpen · 31BDetails100.0%240
–Gemma 4 31B on · computed-argsOpen · 31BDetails100.0%240
–Gemma 4 31B on · multi-constraintOpen · 31BDetails75.0%240
–Gemma 4 31B on · branchingOpen · 31BDetails100.0%240
–Gemma 4 31B on · cascading-recoveryOpen · 31BDetails95.8%240
–Gemma 4 31B on · long-outputOpen · 31BDetails75.0%240

No row of this kind yet.

Task pass rates shown separately for single, selection, arguments, multi-step, restraint and recovery tasks. These are categories of the same task bank, not independent benchmark banks.

Not tiered. Serving and template failures are unscored, not model zeroes.

Failure reasons Failed tasks by reason — lower is better.

ModelFailure reasonsScored attemptsUnscored attempts
–LFM3-1.7B offOpen · 1.7BDetails392520
–LFM3-1.7B onOpen · 1.7BDetails212520
–Gemma 4 E2B onOpen · 5.1BDetails152520
–MiniCPM5-2B offOpen · 2.5BDetails472520
–MiniCPM5-2B onOpen · 2.5BDetails242520
–Qwen3.5-2B offOpen · 2.0BDetails672520
–Qwen3.5-2B onOpen · 2.0BDetails322520
–Qwen3.5 4B offOpen · 4.7BDetails222520
–Qwen3.5 4B onOpen · 4.7BDetails262520
–Raon-1 nano offOpen · 3.1BDetails512520
–Raon-1 nano onOpen · 3.1BDetails282520
–Gemma 4 E2B (tool-result cue template) offOpen · 5.1BDetails272520
–Granite 4.2 3B offOpen · 3.0BDetails522520
–Granite 4.2 3B onOpen · 3.0BDetails182520
–Granite 4.2 8B offOpen · 8.0BDetails412520
–Granite 4.2 8B onOpen · 8.0BDetails172520
–Gemma 4 12B offOpen · 12BDetails142520
–Gemma 4 12B onOpen · 12BDetails182520
–LFM2.5-24B-A2B onOpen · 24B MoE · 2.5B activeDetails182520
–Qwen3.8-27B offOpen · 27BDetails82520
–Qwen3.8-27B onOpen · 27BDetails202520
–Gemma 4 31B offOpen · 31BDetails02520
–Gemma 4 31B onOpen · 31BDetails32520
–Gemma 4 E2B offOpen · 5.1BDetailsNot qualifiedNot scoredNot scored

No row of this kind yet.

Counts or shares of failed tasks by the recorded failure reason: missing, malformed, wrong or extra calls, invented observations, or a loop. Verified infrastructure failures are kept separate and unscored.

Not tiered. Serving and template failures are unscored, not model zeroes.

Failure reasons Failed tasks by reason — lower is better.

ModelFailure reasonsScored attemptsUnscored attempts
–LFM3-1.7B offOpen · 1.7BDetails1021440
–LFM3-1.7B onOpen · 1.7BDetails511440
–Gemma 4 E2B onOpen · 5.1BDetails381440
–MiniCPM5-2B offOpen · 2.5BDetails1021440
–MiniCPM5-2B onOpen · 2.5BDetails241440
–Qwen3.5-2B offOpen · 2.0BDetails611440
–Qwen3.5-2B onOpen · 2.0BDetails461440
–Qwen3.5 4B offOpen · 4.7BDetails151440
–Qwen3.5 4B onOpen · 4.7BDetails281440
–Raon-1 nano offOpen · 3.1BDetails1081440
–Raon-1 nano onOpen · 3.1BDetails421440
–Gemma 4 E2B (tool-result cue template) offOpen · 5.1BDetails331440
–Granite 4.2 3B offOpen · 3.0BDetails861440
–Granite 4.2 3B onOpen · 3.0BDetails171440
–Granite 4.2 8B offOpen · 8.0BDetails831440
–Granite 4.2 8B onOpen · 8.0BDetails121431
–Gemma 4 12B offOpen · 12BDetails411440
–Gemma 4 12B onOpen · 12BDetails71440
–LFM2.5-24B-A2B onOpen · 24B MoE · 2.5B activeDetails141440
–Qwen3.8-27B offOpen · 27BDetails371440
–Qwen3.8-27B onOpen · 27BDetails171440
–Gemma 4 31B offOpen · 31BDetails511440
–Gemma 4 31B onOpen · 31BDetails131440

No row of this kind yet.

Counts or shares of failed tasks by the recorded failure reason: missing, malformed, wrong or extra calls, invented observations, or a loop. Verified infrastructure failures are kept separate and unscored.

Not tiered. Serving and template failures are unscored, not model zeroes.

Renamed tools Original and renamed task pass rates — higher is better.

Tiers within each size class: for each pair of setups, every task (42 per track, public and private) is one paired item, won by the setup that passed more of its 3 repetitions; ties drop out. Exact two-sided sign test, Holm-corrected over all pairs of the class, p < 0.05. Rows in accuracy order: the top row starts Tier 1, and a row starts the next tier when the test separates it from its tier's top row. Accuracy: both tracks (84 tasks); Renamed tools: each track (42 tasks). – : not tiered (class not verified, incomplete or not qualified).ModelRenamed toolsScored attemptsUnscored attempts
Original tools · Tiny
1Gemma 4 E2B on · originalOpen · 5.1BDetails93.7%1260
1Qwen3.5 4B off · originalOpen · 4.7BDetails92.1%1260
1Granite 4.2 3B on · originalOpen · 3.0BDetails92.1%1260
1LFM3-1.7B on · originalOpen · 1.7BDetails91.3%1260
1Gemma 4 E2B (tool-result cue template) off · originalOpen · 5.1BDetails90.5%1260
1Qwen3.5-2B on · originalOpen · 2.0BDetails89.7%1260
1Qwen3.5 4B on · originalOpen · 4.7BDetails88.9%1260
1MiniCPM5-2B on · originalOpen · 2.5BDetails88.1%1260
1Raon-1 nano on · originalOpen · 3.1BDetails88.1%1260
1LFM3-1.7B off · originalOpen · 1.7BDetails86.5%1260
1Granite 4.2 3B off · originalOpen · 3.0BDetails78.6%1260
1MiniCPM5-2B off · originalOpen · 2.5BDetails77.8%1260
1Raon-1 nano off · originalOpen · 3.1BDetails77.0%1260
2Qwen3.5-2B off · originalOpen · 2.0BDetails73.8%1260
–Gemma 4 E2B offOpen · 5.1BDetailsNot qualifiedNot scoredNot scored
Original tools · Small
1Gemma 4 12B off · originalOpen · 12BDetails94.4%1260
1Granite 4.2 8B on · originalOpen · 8.0BDetails92.9%1260
1Gemma 4 12B on · originalOpen · 12BDetails92.9%1260
1Granite 4.2 8B off · originalOpen · 8.0BDetails84.9%1260
Original tools · Medium
1Gemma 4 31B off · originalOpen · 31BDetails100.0%1260
1Gemma 4 31B on · originalOpen · 31BDetails99.2%1260
1Qwen3.8-27B off · originalOpen · 27BDetails96.0%1260
1Qwen3.8-27B on · originalOpen · 27BDetails92.9%1260
1LFM2.5-24B-A2B on · originalOpen · 24B MoE · 2.5B activeDetails92.1%1260
Renamed tools · Tiny
1Gemma 4 E2B on · renamedOpen · 5.1BDetails94.4%1260
1Granite 4.2 3B on · renamedOpen · 3.0BDetails93.7%1260
1MiniCPM5-2B on · renamedOpen · 2.5BDetails92.9%1260
1LFM3-1.7B on · renamedOpen · 1.7BDetails92.1%1260
1Qwen3.5 4B off · renamedOpen · 4.7BDetails90.5%1260
1Qwen3.5 4B on · renamedOpen · 4.7BDetails90.5%1260
1Raon-1 nano on · renamedOpen · 3.1BDetails89.7%1260
1Gemma 4 E2B (tool-result cue template) off · renamedOpen · 5.1BDetails88.1%1260
1MiniCPM5-2B off · renamedOpen · 2.5BDetails84.9%1260
1Qwen3.5-2B on · renamedOpen · 2.0BDetails84.9%1260
1LFM3-1.7B off · renamedOpen · 1.7BDetails82.5%1260
1Raon-1 nano off · renamedOpen · 3.1BDetails82.5%1260
1Granite 4.2 3B off · renamedOpen · 3.0BDetails80.2%1260
2Qwen3.5-2B off · renamedOpen · 2.0BDetails73.0%1260
–Gemma 4 E2B offOpen · 5.1BDetailsNot qualifiedNot scoredNot scored
Renamed tools · Small
1Gemma 4 12B off · renamedOpen · 12BDetails94.4%1260
1Granite 4.2 8B on · renamedOpen · 8.0BDetails93.7%1260
1Gemma 4 12B on · renamedOpen · 12BDetails92.9%1260
1Granite 4.2 8B off · renamedOpen · 8.0BDetails82.5%1260
Renamed tools · Medium
1Gemma 4 31B off · renamedOpen · 31BDetails100.0%1260
1Gemma 4 31B on · renamedOpen · 31BDetails98.4%1260
1Qwen3.8-27B off · renamedOpen · 27BDetails97.6%1260
1LFM2.5-24B-A2B on · renamedOpen · 24B MoE · 2.5B activeDetails93.7%1260
1Qwen3.8-27B on · renamedOpen · 27BDetails91.3%1260

No row of this kind yet.

Paired original-tool and renamed-tool conditions use the same task meaning and mock state. Compare their pass rates without treating the paired versions as independent tasks.

Tiers within each size class: for each pair of setups, every task (42 per track, public and private) is one paired item, won by the setup that passed more of its 3 repetitions; ties drop out. Exact two-sided sign test, Holm-corrected over all pairs of the class, p < 0.05. Rows in accuracy order: the top row starts Tier 1, and a row starts the next tier when the test separates it from its tier's top row. Accuracy: both tracks (84 tasks); Renamed tools: each track (42 tasks). – : not tiered (class not verified, incomplete or not qualified). Serving and template failures are unscored, not model zeroes.

Renamed tools Original and renamed task pass rates — higher is better.

Tiers within each size class: for each pair of setups, every task (24 per track, public and private) is one paired item, won by the setup that passed more of its 3 repetitions; ties drop out. Exact two-sided sign test, Holm-corrected over all pairs of the class, p < 0.05. Rows in accuracy order: the top row starts Tier 1, and a row starts the next tier when the test separates it from its tier's top row. Accuracy: both tracks (48 tasks); Renamed tools: each track (24 tasks). – : not tiered (class not verified, incomplete or not qualified).ModelRenamed toolsScored attemptsUnscored attempts
Original tools · Tiny
1Qwen3.5 4B off · originalOpen · 4.7BDetails94.4%720
1Granite 4.2 3B on · originalOpen · 3.0BDetails84.7%720
1MiniCPM5-2B on · originalOpen · 2.5BDetails83.3%720
1Qwen3.5 4B on · originalOpen · 4.7BDetails77.8%720
1Gemma 4 E2B (tool-result cue template) off · originalOpen · 5.1BDetails76.4%720
1Gemma 4 E2B on · originalOpen · 5.1BDetails70.8%720
1Raon-1 nano on · originalOpen · 3.1BDetails70.8%720
2Qwen3.5-2B on · originalOpen · 2.0BDetails66.7%720
2LFM3-1.7B on · originalOpen · 1.7BDetails65.3%720
2Qwen3.5-2B off · originalOpen · 2.0BDetails58.3%720
2Granite 4.2 3B off · originalOpen · 3.0BDetails38.9%720
2MiniCPM5-2B off · originalOpen · 2.5BDetails36.1%720
2LFM3-1.7B off · originalOpen · 1.7BDetails33.3%720
2Raon-1 nano off · originalOpen · 3.1BDetails26.4%720
Original tools · Small
–Granite 4.2 8B on · originalOpen · 8.0BDetails95.8%711
1Gemma 4 12B on · originalOpen · 12BDetails94.4%720
2Gemma 4 12B off · originalOpen · 12BDetails70.8%720
3Granite 4.2 8B off · originalOpen · 8.0BDetails41.7%720
Original tools · Medium
1LFM2.5-24B-A2B on · originalOpen · 24B MoE · 2.5B activeDetails91.7%720
1Gemma 4 31B on · originalOpen · 31BDetails90.3%720
1Qwen3.8-27B on · originalOpen · 27BDetails88.9%720
1Qwen3.8-27B off · originalOpen · 27BDetails75.0%720
1Gemma 4 31B off · originalOpen · 31BDetails62.5%720
Renamed tools · Tiny
1Granite 4.2 3B on · renamedOpen · 3.0BDetails91.7%720
1Qwen3.5 4B off · renamedOpen · 4.7BDetails84.7%720
1MiniCPM5-2B on · renamedOpen · 2.5BDetails83.3%720
1Qwen3.5 4B on · renamedOpen · 4.7BDetails83.3%720
1Gemma 4 E2B (tool-result cue template) off · renamedOpen · 5.1BDetails77.8%720
1Gemma 4 E2B on · renamedOpen · 5.1BDetails76.4%720
1Raon-1 nano on · renamedOpen · 3.1BDetails70.8%720
1Qwen3.5-2B on · renamedOpen · 2.0BDetails69.4%720
1LFM3-1.7B on · renamedOpen · 1.7BDetails63.9%720
2Qwen3.5-2B off · renamedOpen · 2.0BDetails56.9%720
2Granite 4.2 3B off · renamedOpen · 3.0BDetails41.7%720
2LFM3-1.7B off · renamedOpen · 1.7BDetails25.0%720
3Raon-1 nano off · renamedOpen · 3.1BDetails23.6%720
3MiniCPM5-2B off · renamedOpen · 2.5BDetails22.2%720
Renamed tools · Small
1Gemma 4 12B on · renamedOpen · 12BDetails95.8%720
–Granite 4.2 8B on · renamedOpen · 8.0BDetails87.5%720
2Gemma 4 12B off · renamedOpen · 12BDetails72.2%720
3Granite 4.2 8B off · renamedOpen · 8.0BDetails43.1%720
Renamed tools · Medium
1Gemma 4 31B on · renamedOpen · 31BDetails91.7%720
1LFM2.5-24B-A2B on · renamedOpen · 24B MoE · 2.5B activeDetails88.9%720
1Qwen3.8-27B on · renamedOpen · 27BDetails87.5%720
1Qwen3.8-27B off · renamedOpen · 27BDetails73.6%720
1Gemma 4 31B off · renamedOpen · 31BDetails66.7%720

No row of this kind yet.

Paired original-tool and renamed-tool conditions use the same task meaning and mock state. Compare their pass rates without treating the paired versions as independent tasks.

Tiers within each size class: for each pair of setups, every task (24 per track, public and private) is one paired item, won by the setup that passed more of its 3 repetitions; ties drop out. Exact two-sided sign test, Holm-corrected over all pairs of the class, p < 0.05. Rows in accuracy order: the top row starts Tier 1, and a row starts the next tier when the test separates it from its tier's top row. Accuracy: both tracks (48 tasks); Renamed tools: each track (24 tasks). – : not tiered (class not verified, incomplete or not qualified). Serving and template failures are unscored, not model zeroes.

Latency Seconds per task — lower is better.

ModelLatencyScored attemptsUnscored attempts
–LFM3-1.7B offOpen · 1.7BDetails0.530 s/task2520
–LFM3-1.7B onOpen · 1.7BDetails1.271 s/task2520
–Gemma 4 E2B onOpen · 5.1BDetails4.309 s/task2520
–MiniCPM5-2B offOpen · 2.5BDetails0.746 s/task2520
–MiniCPM5-2B onOpen · 2.5BDetails2.655 s/task2520
–Qwen3.5-2B offOpen · 2.0BDetails1.343 s/task2520
–Qwen3.5-2B onOpen · 2.0BDetails2.917 s/task2520
–Qwen3.5 4B offOpen · 4.7BDetails1.878 s/task2520
–Qwen3.5 4B onOpen · 4.7BDetails4.507 s/task2520
–Raon-1 nano offOpen · 3.1BDetails2.320 s/task2520
–Raon-1 nano onOpen · 3.1BDetails4.257 s/task2520
–Gemma 4 E2B (tool-result cue template) offOpen · 5.1BDetails1.711 s/task2520
–Granite 4.2 3B offOpen · 3.0BDetails1.514 s/task2520
–Granite 4.2 3B onOpen · 3.0BDetails8.037 s/task2520
–Granite 4.2 8B offOpen · 8.0BDetails3.324 s/task2520
–Granite 4.2 8B onOpen · 8.0BDetails19.213 s/task2520
–Gemma 4 12B offOpen · 12BDetails2.284 s/task2520
–Gemma 4 12B onOpen · 12BDetails7.256 s/task2520
–LFM2.5-24B-A2B onOpen · 24B MoE · 2.5B activeDetails8.233 s/task2520
–Qwen3.8-27B offOpen · 27BDetails8.800 s/task2520
–Qwen3.8-27B onOpen · 27BDetails18.498 s/task2520
–Gemma 4 31B offOpen · 31BDetails5.150 s/task2520
–Gemma 4 31B onOpen · 31BDetails15.245 s/task2520
–Gemma 4 E2B offOpen · 5.1BDetailsNot qualifiedNot scoredNot scored

No row of this kind yet.

Observed end-to-end seconds per evaluated task with engine, precision, server and concurrency recorded. Compare like-for-like setups; synthetic fixture timings are not model latency.

Not tiered. Serving and template failures are unscored, not model zeroes.

Latency Seconds per task — lower is better.

ModelLatencyScored attemptsUnscored attempts
–LFM3-1.7B offOpen · 1.7BDetails2.145 s/task1440
–LFM3-1.7B onOpen · 1.7BDetails4.835 s/task1440
–Gemma 4 E2B onOpen · 5.1BDetails10.974 s/task1440
–MiniCPM5-2B offOpen · 2.5BDetails2.353 s/task1440
–MiniCPM5-2B onOpen · 2.5BDetails8.338 s/task1440
–Qwen3.5-2B offOpen · 2.0BDetails5.866 s/task1440
–Qwen3.5-2B onOpen · 2.0BDetails11.172 s/task1440
–Qwen3.5 4B offOpen · 4.7BDetails8.048 s/task1440
–Qwen3.5 4B onOpen · 4.7BDetails15.285 s/task1440
–Raon-1 nano offOpen · 3.1BDetails8.829 s/task1440
–Raon-1 nano onOpen · 3.1BDetails22.222 s/task1440
–Gemma 4 E2B (tool-result cue template) offOpen · 5.1BDetails6.976 s/task1440
–Granite 4.2 3B offOpen · 3.0BDetails12.157 s/task1440
–Granite 4.2 3B onOpen · 3.0BDetails61.376 s/task1440
–Granite 4.2 8B offOpen · 8.0BDetails23.233 s/task1440
–Granite 4.2 8B onOpen · 8.0BDetails94.753 s/task1431
–Gemma 4 12B offOpen · 12BDetails8.252 s/task1440
–Gemma 4 12B onOpen · 12BDetails23.041 s/task1440
–LFM2.5-24B-A2B onOpen · 24B MoE · 2.5B activeDetails21.228 s/task1440
–Qwen3.8-27B offOpen · 27BDetails25.824 s/task1440
–Qwen3.8-27B onOpen · 27BDetails53.048 s/task1440
–Gemma 4 31B offOpen · 31BDetails29.182 s/task1440
–Gemma 4 31B onOpen · 31BDetails63.261 s/task1440

No row of this kind yet.

Observed end-to-end seconds per evaluated task with engine, precision, server and concurrency recorded. Compare like-for-like setups; synthetic fixture timings are not model latency.

Not tiered. Serving and template failures are unscored, not model zeroes.

Cost USD per evaluated task — lower is better.

ModelCostGPU time · cost per runRequests in flight
–LFM3-1.7B offOpen · 1.7BDetails$0.000026/task10 s · $0.0066/run16
–LFM3-1.7B onOpen · 1.7BDetails$0.000064/task23 s · $0.0162/run16
–Gemma 4 E2B onOpen · 5.1BDetails$0.000195/task71 s · $0.0492/run16
–MiniCPM5-2B offOpen · 2.5BDetails$0.000034/task12 s · $0.0085/run16
–MiniCPM5-2B onOpen · 2.5BDetails$0.000158/task57 s · $0.0397/run16
–Qwen3.5-2B offOpen · 2.0BDetails$0.000071/task26 s · $0.0180/run16
–Qwen3.5-2B onOpen · 2.0BDetails$0.000163/task59 s · $0.0411/run16
–Qwen3.5 4B offOpen · 4.7BDetails$0.000086/task31 s · $0.0216/run16
–Qwen3.5 4B onOpen · 4.7BDetails$0.000214/task78 s · $0.0539/run16
–Raon-1 nano offOpen · 3.1BDetails$0.000146/task53 s · $0.0369/run16
–Raon-1 nano onOpen · 3.1BDetails$0.000238/task87 s · $0.0600/run16
–Gemma 4 E2B (tool-result cue template) offOpen · 5.1BDetails$0.000086/task31 s · $0.0216/run16
–Granite 4.2 3B offOpen · 3.0BDetails$0.000099/task36 s · $0.0249/run16
–Granite 4.2 3B onOpen · 3.0BDetails$0.000432/task157 s · $0.1089/run16
–Granite 4.2 8B offOpen · 8.0BDetails$0.000224/task82 s · $0.0564/run16
–Granite 4.2 8B onOpen · 8.0BDetails$0.000910/task332 s · $0.2294/run16
–Gemma 4 12B offOpen · 12BDetails$0.000105/task38 s · $0.0266/run16
–Gemma 4 12B onOpen · 12BDetails$0.000332/task121 s · $0.0836/run16
–LFM2.5-24B-A2B onOpen · 24B MoE · 2.5B activeDetails$0.000370/task135 s · $0.0932/run16
–Qwen3.8-27B offOpen · 27BDetails$0.000394/task144 s · $0.0993/run16
–Qwen3.8-27B onOpen · 27BDetails$0.000864/task315 s · $0.2177/run16
–Gemma 4 31B offOpen · 31BDetails$0.000236/task86 s · $0.0594/run16
–Gemma 4 31B onOpen · 31BDetails$0.000691/task252 s · $0.1740/run16
–Gemma 4 E2B offOpen · 5.1BDetailsNot qualifiedNot scoredNot scored

No row of this kind yet.

Documented charge per evaluated task from actual usage and applicable prices or verified local accounting. Unknown cost stays unreported, never zero. Different-engine cost and speed are not presented as like-for-like.

Not tiered. Serving and template failures are unscored, not model zeroes.

Cost USD per evaluated task — lower is better.

ModelCostGPU time · cost per runRequests in flight
–LFM3-1.7B offOpen · 1.7BDetails$0.000108/task22 s · $0.0156/run16
–LFM3-1.7B onOpen · 1.7BDetails$0.000246/task51 s · $0.0355/run16
–Gemma 4 E2B onOpen · 5.1BDetails$0.000510/task106 s · $0.0734/run16
–MiniCPM5-2B offOpen · 2.5BDetails$0.000114/task24 s · $0.0165/run16
–MiniCPM5-2B onOpen · 2.5BDetails$0.000442/task92 s · $0.0637/run16
–Qwen3.5-2B offOpen · 2.0BDetails$0.000316/task66 s · $0.0455/run16
–Qwen3.5-2B onOpen · 2.0BDetails$0.000537/task112 s · $0.0774/run16
–Qwen3.5 4B offOpen · 4.7BDetails$0.000367/task76 s · $0.0528/run16
–Qwen3.5 4B onOpen · 4.7BDetails$0.000719/task150 s · $0.1035/run16
–Raon-1 nano offOpen · 3.1BDetails$0.000451/task94 s · $0.0649/run16
–Raon-1 nano onOpen · 3.1BDetails$0.001319/task275 s · $0.1899/run16
–Gemma 4 E2B (tool-result cue template) offOpen · 5.1BDetails$0.000326/task68 s · $0.0469/run16
–Granite 4.2 3B offOpen · 3.0BDetails$0.001283/task267 s · $0.1847/run16
–Granite 4.2 3B onOpen · 3.0BDetails$0.003479/task724 s · $0.5010/run16
–Granite 4.2 8B offOpen · 8.0BDetails$0.001391/task290 s · $0.2003/run16
–Granite 4.2 8B onOpen · 8.0BDetails$0.005874/task1214 s · $0.8400/run16
–Gemma 4 12B offOpen · 12BDetails$0.000398/task83 s · $0.0573/run16
–Gemma 4 12B onOpen · 12BDetails$0.001100/task229 s · $0.1585/run16
–LFM2.5-24B-A2B onOpen · 24B MoE · 2.5B activeDetails$0.000965/task201 s · $0.1390/run16
–Qwen3.8-27B offOpen · 27BDetails$0.001161/task242 s · $0.1672/run16
–Qwen3.8-27B onOpen · 27BDetails$0.002477/task516 s · $0.3566/run16
–Gemma 4 31B offOpen · 31BDetails$0.001349/task281 s · $0.1943/run16
–Gemma 4 31B onOpen · 31BDetails$0.002921/task608 s · $0.4207/run16

No row of this kind yet.

Documented charge per evaluated task from actual usage and applicable prices or verified local accounting. Unknown cost stays unreported, never zero. Different-engine cost and speed are not presented as like-for-like.

Not tiered. Serving and template failures are unscored, not model zeroes.

Details of each setup

Provider: local vLLM served by @logic on bariton (#9534) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4

237/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"gave_up_or_loop": 5, "wrong_args": 7, "wrong_tool": 3}

Tokens: 1034078

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 71 s · $0.0492 for the run, $0.000195 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

234/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"no_call": 1, "wrong_args": 6, "wrong_tool": 11}

Tokens: 1560191

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 157 s · $0.1089 for the run, $0.000432 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

231/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 10, "gave_up_or_loop": 5, "wrong_args": 5, "wrong_tool": 1}

Tokens: 888087

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 23 s · $0.0162 for the run, $0.000064 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

230/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 1, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 11, "wrong_tool": 7}

Tokens: 1026831

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

228/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 4, "wrong_args": 18, "wrong_tool": 2}

Tokens: 1133799

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 57 s · $0.0397 for the run, $0.000158 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

226/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 5, "wrong_args": 15, "wrong_tool": 6}

Tokens: 1138207

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 78 s · $0.0539 for the run, $0.000214 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the run, identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4

Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.

225/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 4, "gave_up_or_loop": 1, "wrong_args": 22}

Tokens: 797158

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

224/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 15, "wrong_tool": 4}

Tokens: 1023064

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 87 s · $0.0600 for the run, $0.000238 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

220/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 7, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 13, "wrong_tool": 9}

Tokens: 1267484

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 59 s · $0.0411 for the run, $0.000163 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

213/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"bad_format": 5, "extra_call": 7, "gave_up_or_loop": 2, "wrong_args": 19, "wrong_tool": 6}

Tokens: 773212

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 10 s · $0.0066 for the run, $0.000026 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

205/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"bad_format": 2, "extra_call": 7, "gave_up_or_loop": 14, "wrong_args": 21, "wrong_tool": 3}

Tokens: 894011

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 12 s · $0.0085 for the run, $0.000034 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

201/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"bad_format": 3, "extra_call": 22, "wrong_args": 20, "wrong_tool": 6}

Tokens: 1179082

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 53 s · $0.0369 for the run, $0.000146 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

200/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"gave_up_or_loop": 5, "made_up_result": 1, "wrong_args": 39, "wrong_tool": 7}

Tokens: 1058883

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 36 s · $0.0249 for the run, $0.000099 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

185/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 14, "gave_up_or_loop": 11, "made_up_result": 2, "wrong_args": 33, "wrong_tool": 7}

Tokens: 1173418

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 26 s · $0.0180 for the run, $0.000071 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Not qualified: Qualification failed on public task single-01: after a correct get_price call and its mock result, the model's next token was <|tool_response> (an end-of-generation id), so the reply was empty. With thinking off, neither the maker's chat template nor vLLM's example template adds a continuation cue after a tool result. One observation; cause not proven.

No score: the setup did not complete the qualification tool round trip, so its full run was not admitted.

Qualification evidence sha256: 11b2effa6e9f1f57d2fd1a76001d65b3f2a5f497ee815666bc2a8b6e8d41a18a

Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

238/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"gave_up_or_loop": 2, "wrong_tool": 12}

Tokens: 728093

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 38 s · $0.0266 for the run, $0.000105 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

235/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"gave_up_or_loop": 2, "wrong_args": 8, "wrong_tool": 7}

Tokens: 1580185

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 332 s · $0.2294 for the run, $0.000910 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

234/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 2, "wrong_args": 4, "wrong_tool": 12}

Tokens: 870626

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 121 s · $0.0836 for the run, $0.000332 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

211/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 3, "gave_up_or_loop": 3, "wrong_args": 21, "wrong_tool": 14}

Tokens: 1026531

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 82 s · $0.0564 for the run, $0.000224 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.

252/252 scored passed; 0 unscored; 252 attempted

Failure tags: {}

Tokens: 691572

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 86 s · $0.0594 for the run, $0.000236 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.

249/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"bad_format": 2, "wrong_tool": 1}

Tokens: 783015

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 252 s · $0.1740 for the run, $0.000691 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

244/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 5, "wrong_args": 1, "wrong_tool": 2}

Tokens: 948341

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 144 s · $0.0993 for the run, $0.000394 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183

Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2

234/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 2, "wrong_tool": 7}

Tokens: 998331

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 135 s · $0.0932 for the run, $0.000370 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

232/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"bad_format": 5, "extra_call": 10, "wrong_args": 1, "wrong_tool": 4}

Tokens: 1098479

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 315 s · $0.2177 for the run, $0.000864 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

129/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 1, "wrong_args": 14}

Tokens: 2194487

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 76 s · $0.0528 for the run, $0.000367 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

127/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 4, "no_call": 1, "wrong_args": 12}

Tokens: 4304813

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 724 s · $0.5010 for the run, $0.003479 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

120/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 6, "wrong_args": 18}

Tokens: 2288473

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 92 s · $0.0637 for the run, $0.000442 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

116/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 3, "wrong_args": 25}

Tokens: 2457570

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 150 s · $0.1035 for the run, $0.000719 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the standard run; #10423 GPU 1 for the hard set; identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4

Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.

111/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 1, "made_up_result": 5, "wrong_args": 27}

Tokens: 1836873

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 68 s · $0.0469 for the run, $0.000326 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9534 standard set; #10411 hard set, identical argv) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4

106/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 11, "wrong_args": 27}

Tokens: 2048615

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 106 s · $0.0734 for the run, $0.000510 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

102/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 12, "wrong_args": 30}

Tokens: 2137047

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 275 s · $0.1899 for the run, $0.001319 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

98/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 5, "wrong_args": 40, "wrong_tool": 1}

Tokens: 2603860

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 112 s · $0.0774 for the run, $0.000537 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

93/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 25, "extra_call": 5, "wrong_args": 21}

Tokens: 1790981

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 51 s · $0.0355 for the run, $0.000246 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

83/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 3, "gave_up_or_loop": 8, "made_up_result": 1, "wrong_args": 47, "wrong_tool": 2}

Tokens: 2473838

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 66 s · $0.0455 for the run, $0.000316 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

58/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 12, "gave_up_or_loop": 5, "wrong_args": 68, "wrong_tool": 1}

Tokens: 2601502

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 267 s · $0.1847 for the run, $0.001283 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

42/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 18, "extra_call": 6, "gave_up_or_loop": 4, "wrong_args": 73, "wrong_tool": 1}

Tokens: 1340110

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 22 s · $0.0156 for the run, $0.000108 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

42/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 18, "extra_call": 7, "gave_up_or_loop": 6, "wrong_args": 71}

Tokens: 1586690

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 24 s · $0.0165 for the run, $0.000114 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

36/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 1, "extra_call": 15, "gave_up_or_loop": 6, "wrong_args": 81, "wrong_tool": 5}

Tokens: 2355910

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 94 s · $0.0649 for the run, $0.000451 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

137/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 1, "wrong_args": 6}

Tokens: 2017522

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 229 s · $0.1585 for the run, $0.001100 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

Run note: Not tiered: one of its 144 task runs is unscored. On a long-output task the server refused the request because the prompt plus the fixed 32,768-token answer budget exceeded its 69,632-token context (HTTP 400). Serving and configuration errors are not counted as model failures; the score is over the scored runs.

131/143 scored passed; 1 unscored; 144 attempted

Failure tags: {"wrong_args": 12}

Tokens: Not reported

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 1214 s · $0.8400 for the run, $0.005874 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

103/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 7, "wrong_args": 34}

Tokens: 1798075

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 83 s · $0.0573 for the run, $0.000398 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

61/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 14, "wrong_args": 68, "wrong_tool": 1}

Tokens: 2353976

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 290 s · $0.2003 for the run, $0.001391 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

131/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 13}

Tokens: 1830591

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 608 s · $0.4207 for the run, $0.002921 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10561 hard set on GPU 1, identical argv) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183

Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2

130/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 8, "extra_call": 1, "wrong_args": 5}

Tokens: 2110323

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 201 s · $0.1390 for the run, $0.000965 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

127/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 9, "extra_call": 3, "wrong_args": 5}

Tokens: 2204967

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 516 s · $0.3566 for the run, $0.002477 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

107/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 1, "extra_call": 6, "wrong_args": 30}

Tokens: 2144354

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 242 s · $0.1672 for the run, $0.001161 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

93/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 20, "extra_call": 6, "wrong_args": 25}

Tokens: 1565256

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 281 s · $0.1943 for the run, $0.001349 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"bad_format": 5, "extra_call": 7, "gave_up_or_loop": 2, "wrong_args": 19, "wrong_tool": 6}

Tokens: 773212

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 10 s · $0.0066 for the run, $0.000026 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

27/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"bad_format": 5, "extra_call": 7, "gave_up_or_loop": 2, "wrong_args": 19, "wrong_tool": 6}

Tokens: 773212

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 10 s · $0.0066 for the run, $0.000026 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"bad_format": 5, "extra_call": 7, "gave_up_or_loop": 2, "wrong_args": 19, "wrong_tool": 6}

Tokens: 773212

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 10 s · $0.0066 for the run, $0.000026 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

25/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"bad_format": 5, "extra_call": 7, "gave_up_or_loop": 2, "wrong_args": 19, "wrong_tool": 6}

Tokens: 773212

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 10 s · $0.0066 for the run, $0.000026 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"bad_format": 5, "extra_call": 7, "gave_up_or_loop": 2, "wrong_args": 19, "wrong_tool": 6}

Tokens: 773212

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 10 s · $0.0066 for the run, $0.000026 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

35/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"bad_format": 5, "extra_call": 7, "gave_up_or_loop": 2, "wrong_args": 19, "wrong_tool": 6}

Tokens: 773212

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 10 s · $0.0066 for the run, $0.000026 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 10, "gave_up_or_loop": 5, "wrong_args": 5, "wrong_tool": 1}

Tokens: 888087

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 23 s · $0.0162 for the run, $0.000064 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

41/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 10, "gave_up_or_loop": 5, "wrong_args": 5, "wrong_tool": 1}

Tokens: 888087

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 23 s · $0.0162 for the run, $0.000064 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 10, "gave_up_or_loop": 5, "wrong_args": 5, "wrong_tool": 1}

Tokens: 888087

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 23 s · $0.0162 for the run, $0.000064 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

32/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 10, "gave_up_or_loop": 5, "wrong_args": 5, "wrong_tool": 1}

Tokens: 888087

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 23 s · $0.0162 for the run, $0.000064 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

38/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 10, "gave_up_or_loop": 5, "wrong_args": 5, "wrong_tool": 1}

Tokens: 888087

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 23 s · $0.0162 for the run, $0.000064 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

36/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 10, "gave_up_or_loop": 5, "wrong_args": 5, "wrong_tool": 1}

Tokens: 888087

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 23 s · $0.0162 for the run, $0.000064 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9534) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"gave_up_or_loop": 5, "wrong_args": 7, "wrong_tool": 3}

Tokens: 1034078

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 71 s · $0.0492 for the run, $0.000195 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9534) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4

36/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"gave_up_or_loop": 5, "wrong_args": 7, "wrong_tool": 3}

Tokens: 1034078

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 71 s · $0.0492 for the run, $0.000195 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9534) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"gave_up_or_loop": 5, "wrong_args": 7, "wrong_tool": 3}

Tokens: 1034078

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 71 s · $0.0492 for the run, $0.000195 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9534) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4

39/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"gave_up_or_loop": 5, "wrong_args": 7, "wrong_tool": 3}

Tokens: 1034078

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 71 s · $0.0492 for the run, $0.000195 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9534) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"gave_up_or_loop": 5, "wrong_args": 7, "wrong_tool": 3}

Tokens: 1034078

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 71 s · $0.0492 for the run, $0.000195 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9534) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4

36/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"gave_up_or_loop": 5, "wrong_args": 7, "wrong_tool": 3}

Tokens: 1034078

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 71 s · $0.0492 for the run, $0.000195 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"bad_format": 2, "extra_call": 7, "gave_up_or_loop": 14, "wrong_args": 21, "wrong_tool": 3}

Tokens: 894011

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 12 s · $0.0085 for the run, $0.000034 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

30/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"bad_format": 2, "extra_call": 7, "gave_up_or_loop": 14, "wrong_args": 21, "wrong_tool": 3}

Tokens: 894011

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 12 s · $0.0085 for the run, $0.000034 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"bad_format": 2, "extra_call": 7, "gave_up_or_loop": 14, "wrong_args": 21, "wrong_tool": 3}

Tokens: 894011

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 12 s · $0.0085 for the run, $0.000034 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

16/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"bad_format": 2, "extra_call": 7, "gave_up_or_loop": 14, "wrong_args": 21, "wrong_tool": 3}

Tokens: 894011

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 12 s · $0.0085 for the run, $0.000034 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

41/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"bad_format": 2, "extra_call": 7, "gave_up_or_loop": 14, "wrong_args": 21, "wrong_tool": 3}

Tokens: 894011

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 12 s · $0.0085 for the run, $0.000034 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

34/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"bad_format": 2, "extra_call": 7, "gave_up_or_loop": 14, "wrong_args": 21, "wrong_tool": 3}

Tokens: 894011

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 12 s · $0.0085 for the run, $0.000034 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 4, "wrong_args": 18, "wrong_tool": 2}

Tokens: 1133799

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 57 s · $0.0397 for the run, $0.000158 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

30/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 4, "wrong_args": 18, "wrong_tool": 2}

Tokens: 1133799

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 57 s · $0.0397 for the run, $0.000158 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 4, "wrong_args": 18, "wrong_tool": 2}

Tokens: 1133799

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 57 s · $0.0397 for the run, $0.000158 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

32/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 4, "wrong_args": 18, "wrong_tool": 2}

Tokens: 1133799

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 57 s · $0.0397 for the run, $0.000158 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 4, "wrong_args": 18, "wrong_tool": 2}

Tokens: 1133799

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 57 s · $0.0397 for the run, $0.000158 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

40/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 4, "wrong_args": 18, "wrong_tool": 2}

Tokens: 1133799

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 57 s · $0.0397 for the run, $0.000158 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

40/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 14, "gave_up_or_loop": 11, "made_up_result": 2, "wrong_args": 33, "wrong_tool": 7}

Tokens: 1173418

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 26 s · $0.0180 for the run, $0.000071 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

26/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 14, "gave_up_or_loop": 11, "made_up_result": 2, "wrong_args": 33, "wrong_tool": 7}

Tokens: 1173418

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 26 s · $0.0180 for the run, $0.000071 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

37/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 14, "gave_up_or_loop": 11, "made_up_result": 2, "wrong_args": 33, "wrong_tool": 7}

Tokens: 1173418

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 26 s · $0.0180 for the run, $0.000071 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

19/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 14, "gave_up_or_loop": 11, "made_up_result": 2, "wrong_args": 33, "wrong_tool": 7}

Tokens: 1173418

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 26 s · $0.0180 for the run, $0.000071 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

35/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 14, "gave_up_or_loop": 11, "made_up_result": 2, "wrong_args": 33, "wrong_tool": 7}

Tokens: 1173418

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 26 s · $0.0180 for the run, $0.000071 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

28/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 14, "gave_up_or_loop": 11, "made_up_result": 2, "wrong_args": 33, "wrong_tool": 7}

Tokens: 1173418

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 26 s · $0.0180 for the run, $0.000071 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 7, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 13, "wrong_tool": 9}

Tokens: 1267484

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 59 s · $0.0411 for the run, $0.000163 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

34/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 7, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 13, "wrong_tool": 9}

Tokens: 1267484

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 59 s · $0.0411 for the run, $0.000163 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 7, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 13, "wrong_tool": 9}

Tokens: 1267484

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 59 s · $0.0411 for the run, $0.000163 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

36/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 7, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 13, "wrong_tool": 9}

Tokens: 1267484

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 59 s · $0.0411 for the run, $0.000163 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

37/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 7, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 13, "wrong_tool": 9}

Tokens: 1267484

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 59 s · $0.0411 for the run, $0.000163 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

29/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 7, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 13, "wrong_tool": 9}

Tokens: 1267484

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 59 s · $0.0411 for the run, $0.000163 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 1, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 11, "wrong_tool": 7}

Tokens: 1026831

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

37/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 1, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 11, "wrong_tool": 7}

Tokens: 1026831

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 1, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 11, "wrong_tool": 7}

Tokens: 1026831

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 1, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 11, "wrong_tool": 7}

Tokens: 1026831

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

40/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 1, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 11, "wrong_tool": 7}

Tokens: 1026831

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

27/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 1, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 11, "wrong_tool": 7}

Tokens: 1026831

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 5, "wrong_args": 15, "wrong_tool": 6}

Tokens: 1138207

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 78 s · $0.0539 for the run, $0.000214 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

40/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 5, "wrong_args": 15, "wrong_tool": 6}

Tokens: 1138207

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 78 s · $0.0539 for the run, $0.000214 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

41/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 5, "wrong_args": 15, "wrong_tool": 6}

Tokens: 1138207

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 78 s · $0.0539 for the run, $0.000214 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

30/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 5, "wrong_args": 15, "wrong_tool": 6}

Tokens: 1138207

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 78 s · $0.0539 for the run, $0.000214 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

40/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 5, "wrong_args": 15, "wrong_tool": 6}

Tokens: 1138207

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 78 s · $0.0539 for the run, $0.000214 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

33/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 5, "wrong_args": 15, "wrong_tool": 6}

Tokens: 1138207

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 78 s · $0.0539 for the run, $0.000214 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"bad_format": 3, "extra_call": 22, "wrong_args": 20, "wrong_tool": 6}

Tokens: 1179082

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 53 s · $0.0369 for the run, $0.000146 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

30/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"bad_format": 3, "extra_call": 22, "wrong_args": 20, "wrong_tool": 6}

Tokens: 1179082

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 53 s · $0.0369 for the run, $0.000146 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

40/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"bad_format": 3, "extra_call": 22, "wrong_args": 20, "wrong_tool": 6}

Tokens: 1179082

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 53 s · $0.0369 for the run, $0.000146 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

24/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"bad_format": 3, "extra_call": 22, "wrong_args": 20, "wrong_tool": 6}

Tokens: 1179082

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 53 s · $0.0369 for the run, $0.000146 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

32/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"bad_format": 3, "extra_call": 22, "wrong_args": 20, "wrong_tool": 6}

Tokens: 1179082

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 53 s · $0.0369 for the run, $0.000146 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

33/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"bad_format": 3, "extra_call": 22, "wrong_args": 20, "wrong_tool": 6}

Tokens: 1179082

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 53 s · $0.0369 for the run, $0.000146 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

41/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 15, "wrong_tool": 4}

Tokens: 1023064

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 87 s · $0.0600 for the run, $0.000238 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

36/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 15, "wrong_tool": 4}

Tokens: 1023064

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 87 s · $0.0600 for the run, $0.000238 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 15, "wrong_tool": 4}

Tokens: 1023064

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 87 s · $0.0600 for the run, $0.000238 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

26/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 15, "wrong_tool": 4}

Tokens: 1023064

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 87 s · $0.0600 for the run, $0.000238 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 15, "wrong_tool": 4}

Tokens: 1023064

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 87 s · $0.0600 for the run, $0.000238 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

37/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 15, "wrong_tool": 4}

Tokens: 1023064

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 87 s · $0.0600 for the run, $0.000238 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the run, identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4

Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 4, "gave_up_or_loop": 1, "wrong_args": 22}

Tokens: 797158

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the run, identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4

Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.

30/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 4, "gave_up_or_loop": 1, "wrong_args": 22}

Tokens: 797158

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the run, identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4

Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 4, "gave_up_or_loop": 1, "wrong_args": 22}

Tokens: 797158

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the run, identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4

Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.

31/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 4, "gave_up_or_loop": 1, "wrong_args": 22}

Tokens: 797158

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the run, identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4

Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.

39/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 4, "gave_up_or_loop": 1, "wrong_args": 22}

Tokens: 797158

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the run, identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4

Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.

41/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 4, "gave_up_or_loop": 1, "wrong_args": 22}

Tokens: 797158

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"gave_up_or_loop": 5, "made_up_result": 1, "wrong_args": 39, "wrong_tool": 7}

Tokens: 1058883

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 36 s · $0.0249 for the run, $0.000099 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

25/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"gave_up_or_loop": 5, "made_up_result": 1, "wrong_args": 39, "wrong_tool": 7}

Tokens: 1058883

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 36 s · $0.0249 for the run, $0.000099 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"gave_up_or_loop": 5, "made_up_result": 1, "wrong_args": 39, "wrong_tool": 7}

Tokens: 1058883

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 36 s · $0.0249 for the run, $0.000099 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

26/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"gave_up_or_loop": 5, "made_up_result": 1, "wrong_args": 39, "wrong_tool": 7}

Tokens: 1058883

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 36 s · $0.0249 for the run, $0.000099 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"gave_up_or_loop": 5, "made_up_result": 1, "wrong_args": 39, "wrong_tool": 7}

Tokens: 1058883

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 36 s · $0.0249 for the run, $0.000099 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

23/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"gave_up_or_loop": 5, "made_up_result": 1, "wrong_args": 39, "wrong_tool": 7}

Tokens: 1058883

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 36 s · $0.0249 for the run, $0.000099 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

41/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"no_call": 1, "wrong_args": 6, "wrong_tool": 11}

Tokens: 1560191

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 157 s · $0.1089 for the run, $0.000432 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

36/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"no_call": 1, "wrong_args": 6, "wrong_tool": 11}

Tokens: 1560191

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 157 s · $0.1089 for the run, $0.000432 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"no_call": 1, "wrong_args": 6, "wrong_tool": 11}

Tokens: 1560191

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 157 s · $0.1089 for the run, $0.000432 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"no_call": 1, "wrong_args": 6, "wrong_tool": 11}

Tokens: 1560191

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 157 s · $0.1089 for the run, $0.000432 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"no_call": 1, "wrong_args": 6, "wrong_tool": 11}

Tokens: 1560191

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 157 s · $0.1089 for the run, $0.000432 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

31/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"no_call": 1, "wrong_args": 6, "wrong_tool": 11}

Tokens: 1560191

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 157 s · $0.1089 for the run, $0.000432 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 3, "gave_up_or_loop": 3, "wrong_args": 21, "wrong_tool": 14}

Tokens: 1026531

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 82 s · $0.0564 for the run, $0.000224 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

32/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 3, "gave_up_or_loop": 3, "wrong_args": 21, "wrong_tool": 14}

Tokens: 1026531

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 82 s · $0.0564 for the run, $0.000224 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

41/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 3, "gave_up_or_loop": 3, "wrong_args": 21, "wrong_tool": 14}

Tokens: 1026531

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 82 s · $0.0564 for the run, $0.000224 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

27/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 3, "gave_up_or_loop": 3, "wrong_args": 21, "wrong_tool": 14}

Tokens: 1026531

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 82 s · $0.0564 for the run, $0.000224 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 3, "gave_up_or_loop": 3, "wrong_args": 21, "wrong_tool": 14}

Tokens: 1026531

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 82 s · $0.0564 for the run, $0.000224 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

27/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 3, "gave_up_or_loop": 3, "wrong_args": 21, "wrong_tool": 14}

Tokens: 1026531

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 82 s · $0.0564 for the run, $0.000224 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"gave_up_or_loop": 2, "wrong_args": 8, "wrong_tool": 7}

Tokens: 1580185

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 332 s · $0.2294 for the run, $0.000910 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

35/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"gave_up_or_loop": 2, "wrong_args": 8, "wrong_tool": 7}

Tokens: 1580185

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 332 s · $0.2294 for the run, $0.000910 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"gave_up_or_loop": 2, "wrong_args": 8, "wrong_tool": 7}

Tokens: 1580185

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 332 s · $0.2294 for the run, $0.000910 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

40/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"gave_up_or_loop": 2, "wrong_args": 8, "wrong_tool": 7}

Tokens: 1580185

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 332 s · $0.2294 for the run, $0.000910 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"gave_up_or_loop": 2, "wrong_args": 8, "wrong_tool": 7}

Tokens: 1580185

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 332 s · $0.2294 for the run, $0.000910 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

34/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"gave_up_or_loop": 2, "wrong_args": 8, "wrong_tool": 7}

Tokens: 1580185

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 332 s · $0.2294 for the run, $0.000910 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"gave_up_or_loop": 2, "wrong_tool": 12}

Tokens: 728093

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 38 s · $0.0266 for the run, $0.000105 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"gave_up_or_loop": 2, "wrong_tool": 12}

Tokens: 728093

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 38 s · $0.0266 for the run, $0.000105 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"gave_up_or_loop": 2, "wrong_tool": 12}

Tokens: 728093

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 38 s · $0.0266 for the run, $0.000105 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"gave_up_or_loop": 2, "wrong_tool": 12}

Tokens: 728093

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 38 s · $0.0266 for the run, $0.000105 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"gave_up_or_loop": 2, "wrong_tool": 12}

Tokens: 728093

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 38 s · $0.0266 for the run, $0.000105 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

28/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"gave_up_or_loop": 2, "wrong_tool": 12}

Tokens: 728093

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 38 s · $0.0266 for the run, $0.000105 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 2, "wrong_args": 4, "wrong_tool": 12}

Tokens: 870626

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 121 s · $0.0836 for the run, $0.000332 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

38/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 2, "wrong_args": 4, "wrong_tool": 12}

Tokens: 870626

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 121 s · $0.0836 for the run, $0.000332 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 2, "wrong_args": 4, "wrong_tool": 12}

Tokens: 870626

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 121 s · $0.0836 for the run, $0.000332 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

41/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 2, "wrong_args": 4, "wrong_tool": 12}

Tokens: 870626

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 121 s · $0.0836 for the run, $0.000332 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 2, "wrong_args": 4, "wrong_tool": 12}

Tokens: 870626

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 121 s · $0.0836 for the run, $0.000332 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

29/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 2, "wrong_args": 4, "wrong_tool": 12}

Tokens: 870626

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 121 s · $0.0836 for the run, $0.000332 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183

Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 2, "wrong_tool": 7}

Tokens: 998331

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 135 s · $0.0932 for the run, $0.000370 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183

Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2

37/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 2, "wrong_tool": 7}

Tokens: 998331

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 135 s · $0.0932 for the run, $0.000370 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183

Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 2, "wrong_tool": 7}

Tokens: 998331

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 135 s · $0.0932 for the run, $0.000370 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183

Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 2, "wrong_tool": 7}

Tokens: 998331

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 135 s · $0.0932 for the run, $0.000370 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183

Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2

37/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 2, "wrong_tool": 7}

Tokens: 998331

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 135 s · $0.0932 for the run, $0.000370 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183

Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2

34/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 2, "wrong_tool": 7}

Tokens: 998331

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 135 s · $0.0932 for the run, $0.000370 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 5, "wrong_args": 1, "wrong_tool": 2}

Tokens: 948341

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 144 s · $0.0993 for the run, $0.000394 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 5, "wrong_args": 1, "wrong_tool": 2}

Tokens: 948341

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 144 s · $0.0993 for the run, $0.000394 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 5, "wrong_args": 1, "wrong_tool": 2}

Tokens: 948341

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 144 s · $0.0993 for the run, $0.000394 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

37/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 5, "wrong_args": 1, "wrong_tool": 2}

Tokens: 948341

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 144 s · $0.0993 for the run, $0.000394 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 5, "wrong_args": 1, "wrong_tool": 2}

Tokens: 948341

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 144 s · $0.0993 for the run, $0.000394 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

39/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"extra_call": 5, "wrong_args": 1, "wrong_tool": 2}

Tokens: 948341

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 144 s · $0.0993 for the run, $0.000394 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"bad_format": 5, "extra_call": 10, "wrong_args": 1, "wrong_tool": 4}

Tokens: 1098479

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 315 s · $0.2177 for the run, $0.000864 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

41/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"bad_format": 5, "extra_call": 10, "wrong_args": 1, "wrong_tool": 4}

Tokens: 1098479

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 315 s · $0.2177 for the run, $0.000864 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

37/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"bad_format": 5, "extra_call": 10, "wrong_args": 1, "wrong_tool": 4}

Tokens: 1098479

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 315 s · $0.2177 for the run, $0.000864 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

36/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"bad_format": 5, "extra_call": 10, "wrong_args": 1, "wrong_tool": 4}

Tokens: 1098479

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 315 s · $0.2177 for the run, $0.000864 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"bad_format": 5, "extra_call": 10, "wrong_args": 1, "wrong_tool": 4}

Tokens: 1098479

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 315 s · $0.2177 for the run, $0.000864 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

34/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"bad_format": 5, "extra_call": 10, "wrong_args": 1, "wrong_tool": 4}

Tokens: 1098479

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 315 s · $0.2177 for the run, $0.000864 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {}

Tokens: 691572

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 86 s · $0.0594 for the run, $0.000236 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {}

Tokens: 691572

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 86 s · $0.0594 for the run, $0.000236 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {}

Tokens: 691572

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 86 s · $0.0594 for the run, $0.000236 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {}

Tokens: 691572

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 86 s · $0.0594 for the run, $0.000236 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {}

Tokens: 691572

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 86 s · $0.0594 for the run, $0.000236 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {}

Tokens: 691572

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 86 s · $0.0594 for the run, $0.000236 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"bad_format": 2, "wrong_tool": 1}

Tokens: 783015

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 252 s · $0.1740 for the run, $0.000691 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"bad_format": 2, "wrong_tool": 1}

Tokens: 783015

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 252 s · $0.1740 for the run, $0.000691 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"bad_format": 2, "wrong_tool": 1}

Tokens: 783015

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 252 s · $0.1740 for the run, $0.000691 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.

40/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"bad_format": 2, "wrong_tool": 1}

Tokens: 783015

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 252 s · $0.1740 for the run, $0.000691 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.

42/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"bad_format": 2, "wrong_tool": 1}

Tokens: 783015

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 252 s · $0.1740 for the run, $0.000691 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.

41/42 scored passed; 0 unscored; 42 attempted

Failure tags: {"bad_format": 2, "wrong_tool": 1}

Tokens: 783015

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 252 s · $0.1740 for the run, $0.000691 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Not qualified: Qualification failed on public task single-01: after a correct get_price call and its mock result, the model's next token was <|tool_response> (an end-of-generation id), so the reply was empty. With thinking off, neither the maker's chat template nor vLLM's example template adds a continuation cue after a tool result. One observation; cause not proven.

No score: the setup did not complete the qualification tool round trip, so its full run was not admitted.

Qualification evidence sha256: 11b2effa6e9f1f57d2fd1a76001d65b3f2a5f497ee815666bc2a8b6e8d41a18a

Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

0/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 18, "extra_call": 6, "gave_up_or_loop": 4, "wrong_args": 73, "wrong_tool": 1}

Tokens: 1340110

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 22 s · $0.0156 for the run, $0.000108 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

2/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 18, "extra_call": 6, "gave_up_or_loop": 4, "wrong_args": 73, "wrong_tool": 1}

Tokens: 1340110

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 22 s · $0.0156 for the run, $0.000108 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

10/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 18, "extra_call": 6, "gave_up_or_loop": 4, "wrong_args": 73, "wrong_tool": 1}

Tokens: 1340110

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 22 s · $0.0156 for the run, $0.000108 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

18/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 18, "extra_call": 6, "gave_up_or_loop": 4, "wrong_args": 73, "wrong_tool": 1}

Tokens: 1340110

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 22 s · $0.0156 for the run, $0.000108 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

9/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 18, "extra_call": 6, "gave_up_or_loop": 4, "wrong_args": 73, "wrong_tool": 1}

Tokens: 1340110

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 22 s · $0.0156 for the run, $0.000108 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

3/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 18, "extra_call": 6, "gave_up_or_loop": 4, "wrong_args": 73, "wrong_tool": 1}

Tokens: 1340110

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 22 s · $0.0156 for the run, $0.000108 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

12/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 25, "extra_call": 5, "wrong_args": 21}

Tokens: 1790981

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 51 s · $0.0355 for the run, $0.000246 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

18/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 25, "extra_call": 5, "wrong_args": 21}

Tokens: 1790981

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 51 s · $0.0355 for the run, $0.000246 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

19/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 25, "extra_call": 5, "wrong_args": 21}

Tokens: 1790981

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 51 s · $0.0355 for the run, $0.000246 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

24/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 25, "extra_call": 5, "wrong_args": 21}

Tokens: 1790981

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 51 s · $0.0355 for the run, $0.000246 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

14/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 25, "extra_call": 5, "wrong_args": 21}

Tokens: 1790981

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 51 s · $0.0355 for the run, $0.000246 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

6/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 25, "extra_call": 5, "wrong_args": 21}

Tokens: 1790981

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 51 s · $0.0355 for the run, $0.000246 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9534 standard set; #10411 hard set, identical argv) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4

19/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 11, "wrong_args": 27}

Tokens: 2048615

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 106 s · $0.0734 for the run, $0.000510 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9534 standard set; #10411 hard set, identical argv) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4

15/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 11, "wrong_args": 27}

Tokens: 2048615

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 106 s · $0.0734 for the run, $0.000510 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9534 standard set; #10411 hard set, identical argv) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4

20/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 11, "wrong_args": 27}

Tokens: 2048615

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 106 s · $0.0734 for the run, $0.000510 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9534 standard set; #10411 hard set, identical argv) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4

24/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 11, "wrong_args": 27}

Tokens: 2048615

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 106 s · $0.0734 for the run, $0.000510 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9534 standard set; #10411 hard set, identical argv) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4

21/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 11, "wrong_args": 27}

Tokens: 2048615

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 106 s · $0.0734 for the run, $0.000510 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9534 standard set; #10411 hard set, identical argv) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4

7/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 11, "wrong_args": 27}

Tokens: 2048615

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 106 s · $0.0734 for the run, $0.000510 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

2/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 18, "extra_call": 7, "gave_up_or_loop": 6, "wrong_args": 71}

Tokens: 1586690

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 24 s · $0.0165 for the run, $0.000114 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

4/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 18, "extra_call": 7, "gave_up_or_loop": 6, "wrong_args": 71}

Tokens: 1586690

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 24 s · $0.0165 for the run, $0.000114 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

2/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 18, "extra_call": 7, "gave_up_or_loop": 6, "wrong_args": 71}

Tokens: 1586690

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 24 s · $0.0165 for the run, $0.000114 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

16/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 18, "extra_call": 7, "gave_up_or_loop": 6, "wrong_args": 71}

Tokens: 1586690

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 24 s · $0.0165 for the run, $0.000114 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

17/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 18, "extra_call": 7, "gave_up_or_loop": 6, "wrong_args": 71}

Tokens: 1586690

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 24 s · $0.0165 for the run, $0.000114 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

1/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 18, "extra_call": 7, "gave_up_or_loop": 6, "wrong_args": 71}

Tokens: 1586690

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 24 s · $0.0165 for the run, $0.000114 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

21/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 6, "wrong_args": 18}

Tokens: 2288473

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 92 s · $0.0637 for the run, $0.000442 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

17/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 6, "wrong_args": 18}

Tokens: 2288473

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 92 s · $0.0637 for the run, $0.000442 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

22/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 6, "wrong_args": 18}

Tokens: 2288473

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 92 s · $0.0637 for the run, $0.000442 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

24/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 6, "wrong_args": 18}

Tokens: 2288473

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 92 s · $0.0637 for the run, $0.000442 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

23/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 6, "wrong_args": 18}

Tokens: 2288473

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 92 s · $0.0637 for the run, $0.000442 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

13/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 6, "wrong_args": 18}

Tokens: 2288473

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 92 s · $0.0637 for the run, $0.000442 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

18/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 3, "gave_up_or_loop": 8, "made_up_result": 1, "wrong_args": 47, "wrong_tool": 2}

Tokens: 2473838

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 66 s · $0.0455 for the run, $0.000316 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

7/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 3, "gave_up_or_loop": 8, "made_up_result": 1, "wrong_args": 47, "wrong_tool": 2}

Tokens: 2473838

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 66 s · $0.0455 for the run, $0.000316 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

15/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 3, "gave_up_or_loop": 8, "made_up_result": 1, "wrong_args": 47, "wrong_tool": 2}

Tokens: 2473838

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 66 s · $0.0455 for the run, $0.000316 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

20/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 3, "gave_up_or_loop": 8, "made_up_result": 1, "wrong_args": 47, "wrong_tool": 2}

Tokens: 2473838

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 66 s · $0.0455 for the run, $0.000316 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

21/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 3, "gave_up_or_loop": 8, "made_up_result": 1, "wrong_args": 47, "wrong_tool": 2}

Tokens: 2473838

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 66 s · $0.0455 for the run, $0.000316 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

2/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 3, "gave_up_or_loop": 8, "made_up_result": 1, "wrong_args": 47, "wrong_tool": 2}

Tokens: 2473838

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 66 s · $0.0455 for the run, $0.000316 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

18/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 5, "wrong_args": 40, "wrong_tool": 1}

Tokens: 2603860

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 112 s · $0.0774 for the run, $0.000537 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

11/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 5, "wrong_args": 40, "wrong_tool": 1}

Tokens: 2603860

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 112 s · $0.0774 for the run, $0.000537 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

20/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 5, "wrong_args": 40, "wrong_tool": 1}

Tokens: 2603860

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 112 s · $0.0774 for the run, $0.000537 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

24/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 5, "wrong_args": 40, "wrong_tool": 1}

Tokens: 2603860

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 112 s · $0.0774 for the run, $0.000537 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

20/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 5, "wrong_args": 40, "wrong_tool": 1}

Tokens: 2603860

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 112 s · $0.0774 for the run, $0.000537 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

5/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 5, "wrong_args": 40, "wrong_tool": 1}

Tokens: 2603860

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 112 s · $0.0774 for the run, $0.000537 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

24/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 1, "wrong_args": 14}

Tokens: 2194487

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 76 s · $0.0528 for the run, $0.000367 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

21/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 1, "wrong_args": 14}

Tokens: 2194487

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 76 s · $0.0528 for the run, $0.000367 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

23/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 1, "wrong_args": 14}

Tokens: 2194487

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 76 s · $0.0528 for the run, $0.000367 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

24/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 1, "wrong_args": 14}

Tokens: 2194487

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 76 s · $0.0528 for the run, $0.000367 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

22/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 1, "wrong_args": 14}

Tokens: 2194487

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 76 s · $0.0528 for the run, $0.000367 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

15/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 1, "wrong_args": 14}

Tokens: 2194487

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 76 s · $0.0528 for the run, $0.000367 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

21/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 3, "wrong_args": 25}

Tokens: 2457570

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 150 s · $0.1035 for the run, $0.000719 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

22/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 3, "wrong_args": 25}

Tokens: 2457570

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 150 s · $0.1035 for the run, $0.000719 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

21/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 3, "wrong_args": 25}

Tokens: 2457570

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 150 s · $0.1035 for the run, $0.000719 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

23/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 3, "wrong_args": 25}

Tokens: 2457570

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 150 s · $0.1035 for the run, $0.000719 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

20/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 3, "wrong_args": 25}

Tokens: 2457570

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 150 s · $0.1035 for the run, $0.000719 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

9/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 3, "wrong_args": 25}

Tokens: 2457570

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 150 s · $0.1035 for the run, $0.000719 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

1/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 1, "extra_call": 15, "gave_up_or_loop": 6, "wrong_args": 81, "wrong_tool": 5}

Tokens: 2355910

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 94 s · $0.0649 for the run, $0.000451 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

1/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 1, "extra_call": 15, "gave_up_or_loop": 6, "wrong_args": 81, "wrong_tool": 5}

Tokens: 2355910

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 94 s · $0.0649 for the run, $0.000451 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

2/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 1, "extra_call": 15, "gave_up_or_loop": 6, "wrong_args": 81, "wrong_tool": 5}

Tokens: 2355910

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 94 s · $0.0649 for the run, $0.000451 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

11/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 1, "extra_call": 15, "gave_up_or_loop": 6, "wrong_args": 81, "wrong_tool": 5}

Tokens: 2355910

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 94 s · $0.0649 for the run, $0.000451 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

21/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 1, "extra_call": 15, "gave_up_or_loop": 6, "wrong_args": 81, "wrong_tool": 5}

Tokens: 2355910

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 94 s · $0.0649 for the run, $0.000451 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

0/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 1, "extra_call": 15, "gave_up_or_loop": 6, "wrong_args": 81, "wrong_tool": 5}

Tokens: 2355910

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 94 s · $0.0649 for the run, $0.000451 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

16/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 12, "wrong_args": 30}

Tokens: 2137047

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 275 s · $0.1899 for the run, $0.001319 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

11/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 12, "wrong_args": 30}

Tokens: 2137047

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 275 s · $0.1899 for the run, $0.001319 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

23/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 12, "wrong_args": 30}

Tokens: 2137047

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 275 s · $0.1899 for the run, $0.001319 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

24/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 12, "wrong_args": 30}

Tokens: 2137047

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 275 s · $0.1899 for the run, $0.001319 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

21/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 12, "wrong_args": 30}

Tokens: 2137047

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 275 s · $0.1899 for the run, $0.001319 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

7/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 12, "wrong_args": 30}

Tokens: 2137047

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 275 s · $0.1899 for the run, $0.001319 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the standard run; #10423 GPU 1 for the hard set; identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4

Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.

18/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 1, "made_up_result": 5, "wrong_args": 27}

Tokens: 1836873

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 68 s · $0.0469 for the run, $0.000326 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the standard run; #10423 GPU 1 for the hard set; identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4

Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.

17/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 1, "made_up_result": 5, "wrong_args": 27}

Tokens: 1836873

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 68 s · $0.0469 for the run, $0.000326 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the standard run; #10423 GPU 1 for the hard set; identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4

Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.

24/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 1, "made_up_result": 5, "wrong_args": 27}

Tokens: 1836873

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 68 s · $0.0469 for the run, $0.000326 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the standard run; #10423 GPU 1 for the hard set; identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4

Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.

24/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 1, "made_up_result": 5, "wrong_args": 27}

Tokens: 1836873

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 68 s · $0.0469 for the run, $0.000326 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the standard run; #10423 GPU 1 for the hard set; identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4

Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.

23/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 1, "made_up_result": 5, "wrong_args": 27}

Tokens: 1836873

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 68 s · $0.0469 for the run, $0.000326 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the standard run; #10423 GPU 1 for the hard set; identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4

Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.

5/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 1, "made_up_result": 5, "wrong_args": 27}

Tokens: 1836873

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 68 s · $0.0469 for the run, $0.000326 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

5/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 12, "gave_up_or_loop": 5, "wrong_args": 68, "wrong_tool": 1}

Tokens: 2601502

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 267 s · $0.1847 for the run, $0.001283 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

5/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 12, "gave_up_or_loop": 5, "wrong_args": 68, "wrong_tool": 1}

Tokens: 2601502

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 267 s · $0.1847 for the run, $0.001283 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

14/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 12, "gave_up_or_loop": 5, "wrong_args": 68, "wrong_tool": 1}

Tokens: 2601502

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 267 s · $0.1847 for the run, $0.001283 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

16/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 12, "gave_up_or_loop": 5, "wrong_args": 68, "wrong_tool": 1}

Tokens: 2601502

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 267 s · $0.1847 for the run, $0.001283 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

15/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 12, "gave_up_or_loop": 5, "wrong_args": 68, "wrong_tool": 1}

Tokens: 2601502

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 267 s · $0.1847 for the run, $0.001283 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

3/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 12, "gave_up_or_loop": 5, "wrong_args": 68, "wrong_tool": 1}

Tokens: 2601502

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 267 s · $0.1847 for the run, $0.001283 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

24/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 4, "no_call": 1, "wrong_args": 12}

Tokens: 4304813

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 724 s · $0.5010 for the run, $0.003479 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

21/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 4, "no_call": 1, "wrong_args": 12}

Tokens: 4304813

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 724 s · $0.5010 for the run, $0.003479 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

21/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 4, "no_call": 1, "wrong_args": 12}

Tokens: 4304813

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 724 s · $0.5010 for the run, $0.003479 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

24/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 4, "no_call": 1, "wrong_args": 12}

Tokens: 4304813

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 724 s · $0.5010 for the run, $0.003479 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

18/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 4, "no_call": 1, "wrong_args": 12}

Tokens: 4304813

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 724 s · $0.5010 for the run, $0.003479 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

19/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 4, "no_call": 1, "wrong_args": 12}

Tokens: 4304813

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 724 s · $0.5010 for the run, $0.003479 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

8/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 14, "wrong_args": 68, "wrong_tool": 1}

Tokens: 2353976

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 290 s · $0.2003 for the run, $0.001391 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

5/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 14, "wrong_args": 68, "wrong_tool": 1}

Tokens: 2353976

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 290 s · $0.2003 for the run, $0.001391 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

9/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 14, "wrong_args": 68, "wrong_tool": 1}

Tokens: 2353976

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 290 s · $0.2003 for the run, $0.001391 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

18/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 14, "wrong_args": 68, "wrong_tool": 1}

Tokens: 2353976

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 290 s · $0.2003 for the run, $0.001391 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

19/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 14, "wrong_args": 68, "wrong_tool": 1}

Tokens: 2353976

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 290 s · $0.2003 for the run, $0.001391 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

2/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 14, "wrong_args": 68, "wrong_tool": 1}

Tokens: 2353976

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 290 s · $0.2003 for the run, $0.001391 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

Run note: Not tiered: one of its 144 task runs is unscored. On a long-output task the server refused the request because the prompt plus the fixed 32,768-token answer budget exceeded its 69,632-token context (HTTP 400). Serving and configuration errors are not counted as model failures; the score is over the scored runs.

24/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"wrong_args": 12}

Tokens: Not reported

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 1214 s · $0.8400 for the run, $0.005874 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

Run note: Not tiered: one of its 144 task runs is unscored. On a long-output task the server refused the request because the prompt plus the fixed 32,768-token answer budget exceeded its 69,632-token context (HTTP 400). Serving and configuration errors are not counted as model failures; the score is over the scored runs.

19/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"wrong_args": 12}

Tokens: Not reported

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 1214 s · $0.8400 for the run, $0.005874 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

Run note: Not tiered: one of its 144 task runs is unscored. On a long-output task the server refused the request because the prompt plus the fixed 32,768-token answer budget exceeded its 69,632-token context (HTTP 400). Serving and configuration errors are not counted as model failures; the score is over the scored runs.

24/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"wrong_args": 12}

Tokens: Not reported

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 1214 s · $0.8400 for the run, $0.005874 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

Run note: Not tiered: one of its 144 task runs is unscored. On a long-output task the server refused the request because the prompt plus the fixed 32,768-token answer budget exceeded its 69,632-token context (HTTP 400). Serving and configuration errors are not counted as model failures; the score is over the scored runs.

24/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"wrong_args": 12}

Tokens: Not reported

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 1214 s · $0.8400 for the run, $0.005874 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

Run note: Not tiered: one of its 144 task runs is unscored. On a long-output task the server refused the request because the prompt plus the fixed 32,768-token answer budget exceeded its 69,632-token context (HTTP 400). Serving and configuration errors are not counted as model failures; the score is over the scored runs.

23/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"wrong_args": 12}

Tokens: Not reported

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 1214 s · $0.8400 for the run, $0.005874 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

Run note: Not tiered: one of its 144 task runs is unscored. On a long-output task the server refused the request because the prompt plus the fixed 32,768-token answer budget exceeded its 69,632-token context (HTTP 400). Serving and configuration errors are not counted as model failures; the score is over the scored runs.

17/23 scored passed; 1 unscored; 24 attempted

Failure tags: {"wrong_args": 12}

Tokens: Not reported

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 1214 s · $0.8400 for the run, $0.005874 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

24/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 7, "wrong_args": 34}

Tokens: 1798075

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 83 s · $0.0573 for the run, $0.000398 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

20/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 7, "wrong_args": 34}

Tokens: 1798075

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 83 s · $0.0573 for the run, $0.000398 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

17/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 7, "wrong_args": 34}

Tokens: 1798075

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 83 s · $0.0573 for the run, $0.000398 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

18/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 7, "wrong_args": 34}

Tokens: 1798075

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 83 s · $0.0573 for the run, $0.000398 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

24/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 7, "wrong_args": 34}

Tokens: 1798075

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 83 s · $0.0573 for the run, $0.000398 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

0/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 7, "wrong_args": 34}

Tokens: 1798075

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 83 s · $0.0573 for the run, $0.000398 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

24/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 1, "wrong_args": 6}

Tokens: 2017522

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 229 s · $0.1585 for the run, $0.001100 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

24/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 1, "wrong_args": 6}

Tokens: 2017522

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 229 s · $0.1585 for the run, $0.001100 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

24/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 1, "wrong_args": 6}

Tokens: 2017522

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 229 s · $0.1585 for the run, $0.001100 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

24/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 1, "wrong_args": 6}

Tokens: 2017522

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 229 s · $0.1585 for the run, $0.001100 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

24/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 1, "wrong_args": 6}

Tokens: 2017522

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 229 s · $0.1585 for the run, $0.001100 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

17/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"extra_call": 1, "wrong_args": 6}

Tokens: 2017522

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 229 s · $0.1585 for the run, $0.001100 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10561 hard set on GPU 1, identical argv) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183

Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2

23/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 8, "extra_call": 1, "wrong_args": 5}

Tokens: 2110323

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 201 s · $0.1390 for the run, $0.000965 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10561 hard set on GPU 1, identical argv) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183

Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2

24/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 8, "extra_call": 1, "wrong_args": 5}

Tokens: 2110323

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 201 s · $0.1390 for the run, $0.000965 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10561 hard set on GPU 1, identical argv) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183

Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2

22/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 8, "extra_call": 1, "wrong_args": 5}

Tokens: 2110323

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 201 s · $0.1390 for the run, $0.000965 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10561 hard set on GPU 1, identical argv) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183

Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2

24/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 8, "extra_call": 1, "wrong_args": 5}

Tokens: 2110323

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 201 s · $0.1390 for the run, $0.000965 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10561 hard set on GPU 1, identical argv) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183

Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2

21/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 8, "extra_call": 1, "wrong_args": 5}

Tokens: 2110323

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 201 s · $0.1390 for the run, $0.000965 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10561 hard set on GPU 1, identical argv) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183

Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2

16/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 8, "extra_call": 1, "wrong_args": 5}

Tokens: 2110323

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 201 s · $0.1390 for the run, $0.000965 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

18/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 1, "extra_call": 6, "wrong_args": 30}

Tokens: 2144354

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 242 s · $0.1672 for the run, $0.001161 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

17/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 1, "extra_call": 6, "wrong_args": 30}

Tokens: 2144354

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 242 s · $0.1672 for the run, $0.001161 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

24/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 1, "extra_call": 6, "wrong_args": 30}

Tokens: 2144354

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 242 s · $0.1672 for the run, $0.001161 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

18/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 1, "extra_call": 6, "wrong_args": 30}

Tokens: 2144354

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 242 s · $0.1672 for the run, $0.001161 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

24/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 1, "extra_call": 6, "wrong_args": 30}

Tokens: 2144354

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 242 s · $0.1672 for the run, $0.001161 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

6/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 1, "extra_call": 6, "wrong_args": 30}

Tokens: 2144354

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 242 s · $0.1672 for the run, $0.001161 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

21/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 9, "extra_call": 3, "wrong_args": 5}

Tokens: 2204967

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 516 s · $0.3566 for the run, $0.002477 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

21/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 9, "extra_call": 3, "wrong_args": 5}

Tokens: 2204967

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 516 s · $0.3566 for the run, $0.002477 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

24/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 9, "extra_call": 3, "wrong_args": 5}

Tokens: 2204967

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 516 s · $0.3566 for the run, $0.002477 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

23/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 9, "extra_call": 3, "wrong_args": 5}

Tokens: 2204967

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 516 s · $0.3566 for the run, $0.002477 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

23/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 9, "extra_call": 3, "wrong_args": 5}

Tokens: 2204967

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 516 s · $0.3566 for the run, $0.002477 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

15/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 9, "extra_call": 3, "wrong_args": 5}

Tokens: 2204967

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 516 s · $0.3566 for the run, $0.002477 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

18/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 20, "extra_call": 6, "wrong_args": 25}

Tokens: 1565256

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 281 s · $0.1943 for the run, $0.001349 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

15/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 20, "extra_call": 6, "wrong_args": 25}

Tokens: 1565256

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 281 s · $0.1943 for the run, $0.001349 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

19/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 20, "extra_call": 6, "wrong_args": 25}

Tokens: 1565256

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 281 s · $0.1943 for the run, $0.001349 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

18/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 20, "extra_call": 6, "wrong_args": 25}

Tokens: 1565256

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 281 s · $0.1943 for the run, $0.001349 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

20/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 20, "extra_call": 6, "wrong_args": 25}

Tokens: 1565256

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 281 s · $0.1943 for the run, $0.001349 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

3/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 20, "extra_call": 6, "wrong_args": 25}

Tokens: 1565256

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 281 s · $0.1943 for the run, $0.001349 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

24/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 13}

Tokens: 1830591

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 608 s · $0.4207 for the run, $0.002921 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

24/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 13}

Tokens: 1830591

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 608 s · $0.4207 for the run, $0.002921 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

18/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 13}

Tokens: 1830591

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 608 s · $0.4207 for the run, $0.002921 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

24/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 13}

Tokens: 1830591

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 608 s · $0.4207 for the run, $0.002921 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

23/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 13}

Tokens: 1830591

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 608 s · $0.4207 for the run, $0.002921 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

18/24 scored passed; 0 unscored; 24 attempted

Failure tags: {"bad_format": 13}

Tokens: 1830591

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 608 s · $0.4207 for the run, $0.002921 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

213/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"bad_format": 5, "extra_call": 7, "gave_up_or_loop": 2, "wrong_args": 19, "wrong_tool": 6}

Tokens: 773212

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 10 s · $0.0066 for the run, $0.000026 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

231/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 10, "gave_up_or_loop": 5, "wrong_args": 5, "wrong_tool": 1}

Tokens: 888087

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 23 s · $0.0162 for the run, $0.000064 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9534) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4

237/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"gave_up_or_loop": 5, "wrong_args": 7, "wrong_tool": 3}

Tokens: 1034078

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 71 s · $0.0492 for the run, $0.000195 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

205/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"bad_format": 2, "extra_call": 7, "gave_up_or_loop": 14, "wrong_args": 21, "wrong_tool": 3}

Tokens: 894011

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 12 s · $0.0085 for the run, $0.000034 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

228/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 4, "wrong_args": 18, "wrong_tool": 2}

Tokens: 1133799

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 57 s · $0.0397 for the run, $0.000158 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

185/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 14, "gave_up_or_loop": 11, "made_up_result": 2, "wrong_args": 33, "wrong_tool": 7}

Tokens: 1173418

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 26 s · $0.0180 for the run, $0.000071 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

220/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 7, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 13, "wrong_tool": 9}

Tokens: 1267484

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 59 s · $0.0411 for the run, $0.000163 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

230/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 1, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 11, "wrong_tool": 7}

Tokens: 1026831

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

226/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 5, "wrong_args": 15, "wrong_tool": 6}

Tokens: 1138207

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 78 s · $0.0539 for the run, $0.000214 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

201/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"bad_format": 3, "extra_call": 22, "wrong_args": 20, "wrong_tool": 6}

Tokens: 1179082

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 53 s · $0.0369 for the run, $0.000146 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

224/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 15, "wrong_tool": 4}

Tokens: 1023064

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 87 s · $0.0600 for the run, $0.000238 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the run, identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4

Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.

225/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 4, "gave_up_or_loop": 1, "wrong_args": 22}

Tokens: 797158

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

200/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"gave_up_or_loop": 5, "made_up_result": 1, "wrong_args": 39, "wrong_tool": 7}

Tokens: 1058883

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 36 s · $0.0249 for the run, $0.000099 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

234/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"no_call": 1, "wrong_args": 6, "wrong_tool": 11}

Tokens: 1560191

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 157 s · $0.1089 for the run, $0.000432 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

211/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 3, "gave_up_or_loop": 3, "wrong_args": 21, "wrong_tool": 14}

Tokens: 1026531

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 82 s · $0.0564 for the run, $0.000224 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

235/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"gave_up_or_loop": 2, "wrong_args": 8, "wrong_tool": 7}

Tokens: 1580185

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 332 s · $0.2294 for the run, $0.000910 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

238/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"gave_up_or_loop": 2, "wrong_tool": 12}

Tokens: 728093

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 38 s · $0.0266 for the run, $0.000105 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

234/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 2, "wrong_args": 4, "wrong_tool": 12}

Tokens: 870626

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 121 s · $0.0836 for the run, $0.000332 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183

Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2

234/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 2, "wrong_tool": 7}

Tokens: 998331

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 135 s · $0.0932 for the run, $0.000370 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

244/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 5, "wrong_args": 1, "wrong_tool": 2}

Tokens: 948341

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 144 s · $0.0993 for the run, $0.000394 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

232/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"bad_format": 5, "extra_call": 10, "wrong_args": 1, "wrong_tool": 4}

Tokens: 1098479

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 315 s · $0.2177 for the run, $0.000864 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.

252/252 scored passed; 0 unscored; 252 attempted

Failure tags: {}

Tokens: 691572

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 86 s · $0.0594 for the run, $0.000236 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.

249/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"bad_format": 2, "wrong_tool": 1}

Tokens: 783015

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 252 s · $0.1740 for the run, $0.000691 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Not qualified: Qualification failed on public task single-01: after a correct get_price call and its mock result, the model's next token was <|tool_response> (an end-of-generation id), so the reply was empty. With thinking off, neither the maker's chat template nor vLLM's example template adds a continuation cue after a tool result. One observation; cause not proven.

No score: the setup did not complete the qualification tool round trip, so its full run was not admitted.

Qualification evidence sha256: 11b2effa6e9f1f57d2fd1a76001d65b3f2a5f497ee815666bc2a8b6e8d41a18a

Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

42/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 18, "extra_call": 6, "gave_up_or_loop": 4, "wrong_args": 73, "wrong_tool": 1}

Tokens: 1340110

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 22 s · $0.0156 for the run, $0.000108 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

93/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 25, "extra_call": 5, "wrong_args": 21}

Tokens: 1790981

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 51 s · $0.0355 for the run, $0.000246 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9534 standard set; #10411 hard set, identical argv) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4

106/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 11, "wrong_args": 27}

Tokens: 2048615

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 106 s · $0.0734 for the run, $0.000510 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

42/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 18, "extra_call": 7, "gave_up_or_loop": 6, "wrong_args": 71}

Tokens: 1586690

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 24 s · $0.0165 for the run, $0.000114 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

120/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 6, "wrong_args": 18}

Tokens: 2288473

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 92 s · $0.0637 for the run, $0.000442 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

83/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 3, "gave_up_or_loop": 8, "made_up_result": 1, "wrong_args": 47, "wrong_tool": 2}

Tokens: 2473838

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 66 s · $0.0455 for the run, $0.000316 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

98/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 5, "wrong_args": 40, "wrong_tool": 1}

Tokens: 2603860

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 112 s · $0.0774 for the run, $0.000537 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

129/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 1, "wrong_args": 14}

Tokens: 2194487

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 76 s · $0.0528 for the run, $0.000367 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

116/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 3, "wrong_args": 25}

Tokens: 2457570

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 150 s · $0.1035 for the run, $0.000719 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

36/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 1, "extra_call": 15, "gave_up_or_loop": 6, "wrong_args": 81, "wrong_tool": 5}

Tokens: 2355910

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 94 s · $0.0649 for the run, $0.000451 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

102/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 12, "wrong_args": 30}

Tokens: 2137047

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 275 s · $0.1899 for the run, $0.001319 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the standard run; #10423 GPU 1 for the hard set; identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4

Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.

111/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 1, "made_up_result": 5, "wrong_args": 27}

Tokens: 1836873

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 68 s · $0.0469 for the run, $0.000326 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

58/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 12, "gave_up_or_loop": 5, "wrong_args": 68, "wrong_tool": 1}

Tokens: 2601502

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 267 s · $0.1847 for the run, $0.001283 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

127/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 4, "no_call": 1, "wrong_args": 12}

Tokens: 4304813

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 724 s · $0.5010 for the run, $0.003479 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

61/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 14, "wrong_args": 68, "wrong_tool": 1}

Tokens: 2353976

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 290 s · $0.2003 for the run, $0.001391 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

Run note: Not tiered: one of its 144 task runs is unscored. On a long-output task the server refused the request because the prompt plus the fixed 32,768-token answer budget exceeded its 69,632-token context (HTTP 400). Serving and configuration errors are not counted as model failures; the score is over the scored runs.

131/143 scored passed; 1 unscored; 144 attempted

Failure tags: {"wrong_args": 12}

Tokens: Not reported

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 1214 s · $0.8400 for the run, $0.005874 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

103/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 7, "wrong_args": 34}

Tokens: 1798075

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 83 s · $0.0573 for the run, $0.000398 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

137/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 1, "wrong_args": 6}

Tokens: 2017522

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 229 s · $0.1585 for the run, $0.001100 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10561 hard set on GPU 1, identical argv) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183

Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2

130/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 8, "extra_call": 1, "wrong_args": 5}

Tokens: 2110323

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 201 s · $0.1390 for the run, $0.000965 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

107/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 1, "extra_call": 6, "wrong_args": 30}

Tokens: 2144354

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 242 s · $0.1672 for the run, $0.001161 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

127/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 9, "extra_call": 3, "wrong_args": 5}

Tokens: 2204967

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 516 s · $0.3566 for the run, $0.002477 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

93/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 20, "extra_call": 6, "wrong_args": 25}

Tokens: 1565256

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 281 s · $0.1943 for the run, $0.001349 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

131/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 13}

Tokens: 1830591

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 608 s · $0.4207 for the run, $0.002921 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9534) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4

118/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"gave_up_or_loop": 5, "wrong_args": 7, "wrong_tool": 3}

Tokens: 1034078

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 71 s · $0.0492 for the run, $0.000195 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

116/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"extra_call": 1, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 11, "wrong_tool": 7}

Tokens: 1026831

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

116/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"no_call": 1, "wrong_args": 6, "wrong_tool": 11}

Tokens: 1560191

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 157 s · $0.1089 for the run, $0.000432 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

115/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"extra_call": 10, "gave_up_or_loop": 5, "wrong_args": 5, "wrong_tool": 1}

Tokens: 888087

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 23 s · $0.0162 for the run, $0.000064 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the run, identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4

Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.

114/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"extra_call": 4, "gave_up_or_loop": 1, "wrong_args": 22}

Tokens: 797158

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

113/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"extra_call": 7, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 13, "wrong_tool": 9}

Tokens: 1267484

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 59 s · $0.0411 for the run, $0.000163 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

112/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"extra_call": 5, "wrong_args": 15, "wrong_tool": 6}

Tokens: 1138207

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 78 s · $0.0539 for the run, $0.000214 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

111/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"extra_call": 4, "wrong_args": 18, "wrong_tool": 2}

Tokens: 1133799

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 57 s · $0.0397 for the run, $0.000158 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

111/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 15, "wrong_tool": 4}

Tokens: 1023064

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 87 s · $0.0600 for the run, $0.000238 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

109/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"bad_format": 5, "extra_call": 7, "gave_up_or_loop": 2, "wrong_args": 19, "wrong_tool": 6}

Tokens: 773212

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 10 s · $0.0066 for the run, $0.000026 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

99/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"gave_up_or_loop": 5, "made_up_result": 1, "wrong_args": 39, "wrong_tool": 7}

Tokens: 1058883

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 36 s · $0.0249 for the run, $0.000099 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

98/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"bad_format": 2, "extra_call": 7, "gave_up_or_loop": 14, "wrong_args": 21, "wrong_tool": 3}

Tokens: 894011

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 12 s · $0.0085 for the run, $0.000034 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

97/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"bad_format": 3, "extra_call": 22, "wrong_args": 20, "wrong_tool": 6}

Tokens: 1179082

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 53 s · $0.0369 for the run, $0.000146 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

93/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"extra_call": 14, "gave_up_or_loop": 11, "made_up_result": 2, "wrong_args": 33, "wrong_tool": 7}

Tokens: 1173418

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 26 s · $0.0180 for the run, $0.000071 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Not qualified: Qualification failed on public task single-01: after a correct get_price call and its mock result, the model's next token was <|tool_response> (an end-of-generation id), so the reply was empty. With thinking off, neither the maker's chat template nor vLLM's example template adds a continuation cue after a tool result. One observation; cause not proven.

No score: the setup did not complete the qualification tool round trip, so its full run was not admitted.

Qualification evidence sha256: 11b2effa6e9f1f57d2fd1a76001d65b3f2a5f497ee815666bc2a8b6e8d41a18a

Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

119/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"gave_up_or_loop": 2, "wrong_tool": 12}

Tokens: 728093

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 38 s · $0.0266 for the run, $0.000105 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

117/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"gave_up_or_loop": 2, "wrong_args": 8, "wrong_tool": 7}

Tokens: 1580185

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 332 s · $0.2294 for the run, $0.000910 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

117/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"extra_call": 2, "wrong_args": 4, "wrong_tool": 12}

Tokens: 870626

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 121 s · $0.0836 for the run, $0.000332 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

107/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"extra_call": 3, "gave_up_or_loop": 3, "wrong_args": 21, "wrong_tool": 14}

Tokens: 1026531

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 82 s · $0.0564 for the run, $0.000224 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.

126/126 scored passed; 0 unscored; 126 attempted

Failure tags: {}

Tokens: 691572

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 86 s · $0.0594 for the run, $0.000236 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.

125/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"bad_format": 2, "wrong_tool": 1}

Tokens: 783015

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 252 s · $0.1740 for the run, $0.000691 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

121/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"extra_call": 5, "wrong_args": 1, "wrong_tool": 2}

Tokens: 948341

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 144 s · $0.0993 for the run, $0.000394 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

117/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"bad_format": 5, "extra_call": 10, "wrong_args": 1, "wrong_tool": 4}

Tokens: 1098479

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 315 s · $0.2177 for the run, $0.000864 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183

Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2

116/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 2, "wrong_tool": 7}

Tokens: 998331

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 135 s · $0.0932 for the run, $0.000370 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9534) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4

119/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"gave_up_or_loop": 5, "wrong_args": 7, "wrong_tool": 3}

Tokens: 1034078

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 71 s · $0.0492 for the run, $0.000195 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

118/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"no_call": 1, "wrong_args": 6, "wrong_tool": 11}

Tokens: 1560191

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 157 s · $0.1089 for the run, $0.000432 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

117/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"extra_call": 4, "wrong_args": 18, "wrong_tool": 2}

Tokens: 1133799

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 57 s · $0.0397 for the run, $0.000158 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

116/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"extra_call": 10, "gave_up_or_loop": 5, "wrong_args": 5, "wrong_tool": 1}

Tokens: 888087

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 23 s · $0.0162 for the run, $0.000064 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

114/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"extra_call": 1, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 11, "wrong_tool": 7}

Tokens: 1026831

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

114/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"extra_call": 5, "wrong_args": 15, "wrong_tool": 6}

Tokens: 1138207

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 78 s · $0.0539 for the run, $0.000214 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

113/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 15, "wrong_tool": 4}

Tokens: 1023064

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 87 s · $0.0600 for the run, $0.000238 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the run, identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4

Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.

111/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"extra_call": 4, "gave_up_or_loop": 1, "wrong_args": 22}

Tokens: 797158

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

107/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"bad_format": 2, "extra_call": 7, "gave_up_or_loop": 14, "wrong_args": 21, "wrong_tool": 3}

Tokens: 894011

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 12 s · $0.0085 for the run, $0.000034 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

107/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"extra_call": 7, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 13, "wrong_tool": 9}

Tokens: 1267484

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 59 s · $0.0411 for the run, $0.000163 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

104/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"bad_format": 5, "extra_call": 7, "gave_up_or_loop": 2, "wrong_args": 19, "wrong_tool": 6}

Tokens: 773212

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 10 s · $0.0066 for the run, $0.000026 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

104/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"bad_format": 3, "extra_call": 22, "wrong_args": 20, "wrong_tool": 6}

Tokens: 1179082

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 53 s · $0.0369 for the run, $0.000146 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

101/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"gave_up_or_loop": 5, "made_up_result": 1, "wrong_args": 39, "wrong_tool": 7}

Tokens: 1058883

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 36 s · $0.0249 for the run, $0.000099 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

92/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"extra_call": 14, "gave_up_or_loop": 11, "made_up_result": 2, "wrong_args": 33, "wrong_tool": 7}

Tokens: 1173418

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 26 s · $0.0180 for the run, $0.000071 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

119/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"gave_up_or_loop": 2, "wrong_tool": 12}

Tokens: 728093

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 38 s · $0.0266 for the run, $0.000105 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

118/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"gave_up_or_loop": 2, "wrong_args": 8, "wrong_tool": 7}

Tokens: 1580185

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 332 s · $0.2294 for the run, $0.000910 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

117/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"extra_call": 2, "wrong_args": 4, "wrong_tool": 12}

Tokens: 870626

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 121 s · $0.0836 for the run, $0.000332 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

104/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"extra_call": 3, "gave_up_or_loop": 3, "wrong_args": 21, "wrong_tool": 14}

Tokens: 1026531

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 82 s · $0.0564 for the run, $0.000224 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.

126/126 scored passed; 0 unscored; 126 attempted

Failure tags: {}

Tokens: 691572

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 86 s · $0.0594 for the run, $0.000236 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.

124/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"bad_format": 2, "wrong_tool": 1}

Tokens: 783015

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 252 s · $0.1740 for the run, $0.000691 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

123/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"extra_call": 5, "wrong_args": 1, "wrong_tool": 2}

Tokens: 948341

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 144 s · $0.0993 for the run, $0.000394 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183

Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2

118/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 2, "wrong_tool": 7}

Tokens: 998331

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 135 s · $0.0932 for the run, $0.000370 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

115/126 scored passed; 0 unscored; 126 attempted

Failure tags: {"bad_format": 5, "extra_call": 10, "wrong_args": 1, "wrong_tool": 4}

Tokens: 1098479

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 315 s · $0.2177 for the run, $0.000864 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

68/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"bad_format": 1, "wrong_args": 14}

Tokens: 2194487

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 76 s · $0.0528 for the run, $0.000367 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

61/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"extra_call": 4, "no_call": 1, "wrong_args": 12}

Tokens: 4304813

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 724 s · $0.5010 for the run, $0.003479 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

60/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"bad_format": 6, "wrong_args": 18}

Tokens: 2288473

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 92 s · $0.0637 for the run, $0.000442 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

56/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"bad_format": 3, "wrong_args": 25}

Tokens: 2457570

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 150 s · $0.1035 for the run, $0.000719 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the standard run; #10423 GPU 1 for the hard set; identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4

Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.

55/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"extra_call": 1, "made_up_result": 5, "wrong_args": 27}

Tokens: 1836873

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 68 s · $0.0469 for the run, $0.000326 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9534 standard set; #10411 hard set, identical argv) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4

51/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"bad_format": 11, "wrong_args": 27}

Tokens: 2048615

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 106 s · $0.0734 for the run, $0.000510 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

51/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"extra_call": 12, "wrong_args": 30}

Tokens: 2137047

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 275 s · $0.1899 for the run, $0.001319 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

48/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"extra_call": 5, "wrong_args": 40, "wrong_tool": 1}

Tokens: 2603860

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 112 s · $0.0774 for the run, $0.000537 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

47/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"bad_format": 25, "extra_call": 5, "wrong_args": 21}

Tokens: 1790981

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 51 s · $0.0355 for the run, $0.000246 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

42/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"extra_call": 3, "gave_up_or_loop": 8, "made_up_result": 1, "wrong_args": 47, "wrong_tool": 2}

Tokens: 2473838

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 66 s · $0.0455 for the run, $0.000316 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

28/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"extra_call": 12, "gave_up_or_loop": 5, "wrong_args": 68, "wrong_tool": 1}

Tokens: 2601502

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 267 s · $0.1847 for the run, $0.001283 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

26/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"bad_format": 18, "extra_call": 7, "gave_up_or_loop": 6, "wrong_args": 71}

Tokens: 1586690

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 24 s · $0.0165 for the run, $0.000114 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

24/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"bad_format": 18, "extra_call": 6, "gave_up_or_loop": 4, "wrong_args": 73, "wrong_tool": 1}

Tokens: 1340110

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 22 s · $0.0156 for the run, $0.000108 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

19/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"bad_format": 1, "extra_call": 15, "gave_up_or_loop": 6, "wrong_args": 81, "wrong_tool": 5}

Tokens: 2355910

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 94 s · $0.0649 for the run, $0.000451 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

Run note: Not tiered: one of its 144 task runs is unscored. On a long-output task the server refused the request because the prompt plus the fixed 32,768-token answer budget exceeded its 69,632-token context (HTTP 400). Serving and configuration errors are not counted as model failures; the score is over the scored runs.

68/71 scored passed; 1 unscored; 72 attempted

Failure tags: {"wrong_args": 12}

Tokens: Not reported

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 1214 s · $0.8400 for the run, $0.005874 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

68/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"extra_call": 1, "wrong_args": 6}

Tokens: 2017522

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 229 s · $0.1585 for the run, $0.001100 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

51/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"extra_call": 7, "wrong_args": 34}

Tokens: 1798075

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 83 s · $0.0573 for the run, $0.000398 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

30/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"extra_call": 14, "wrong_args": 68, "wrong_tool": 1}

Tokens: 2353976

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 290 s · $0.2003 for the run, $0.001391 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10561 hard set on GPU 1, identical argv) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183

Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2

66/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"bad_format": 8, "extra_call": 1, "wrong_args": 5}

Tokens: 2110323

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 201 s · $0.1390 for the run, $0.000965 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

65/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"bad_format": 13}

Tokens: 1830591

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 608 s · $0.4207 for the run, $0.002921 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

64/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"bad_format": 9, "extra_call": 3, "wrong_args": 5}

Tokens: 2204967

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 516 s · $0.3566 for the run, $0.002477 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

54/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"bad_format": 1, "extra_call": 6, "wrong_args": 30}

Tokens: 2144354

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 242 s · $0.1672 for the run, $0.001161 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

45/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"bad_format": 20, "extra_call": 6, "wrong_args": 25}

Tokens: 1565256

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 281 s · $0.1943 for the run, $0.001349 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

66/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"extra_call": 4, "no_call": 1, "wrong_args": 12}

Tokens: 4304813

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 724 s · $0.5010 for the run, $0.003479 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

61/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"bad_format": 1, "wrong_args": 14}

Tokens: 2194487

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 76 s · $0.0528 for the run, $0.000367 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

60/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"bad_format": 6, "wrong_args": 18}

Tokens: 2288473

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 92 s · $0.0637 for the run, $0.000442 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

60/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"bad_format": 3, "wrong_args": 25}

Tokens: 2457570

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 150 s · $0.1035 for the run, $0.000719 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the standard run; #10423 GPU 1 for the hard set; identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4

Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.

56/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"extra_call": 1, "made_up_result": 5, "wrong_args": 27}

Tokens: 1836873

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 68 s · $0.0469 for the run, $0.000326 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9534 standard set; #10411 hard set, identical argv) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4

55/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"bad_format": 11, "wrong_args": 27}

Tokens: 2048615

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 106 s · $0.0734 for the run, $0.000510 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

51/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"extra_call": 12, "wrong_args": 30}

Tokens: 2137047

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 275 s · $0.1899 for the run, $0.001319 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

50/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"extra_call": 5, "wrong_args": 40, "wrong_tool": 1}

Tokens: 2603860

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 112 s · $0.0774 for the run, $0.000537 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

46/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"bad_format": 25, "extra_call": 5, "wrong_args": 21}

Tokens: 1790981

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 51 s · $0.0355 for the run, $0.000246 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

41/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"extra_call": 3, "gave_up_or_loop": 8, "made_up_result": 1, "wrong_args": 47, "wrong_tool": 2}

Tokens: 2473838

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 66 s · $0.0455 for the run, $0.000316 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

30/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"extra_call": 12, "gave_up_or_loop": 5, "wrong_args": 68, "wrong_tool": 1}

Tokens: 2601502

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 267 s · $0.1847 for the run, $0.001283 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

18/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"bad_format": 18, "extra_call": 6, "gave_up_or_loop": 4, "wrong_args": 73, "wrong_tool": 1}

Tokens: 1340110

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 22 s · $0.0156 for the run, $0.000108 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

17/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"bad_format": 1, "extra_call": 15, "gave_up_or_loop": 6, "wrong_args": 81, "wrong_tool": 5}

Tokens: 2355910

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 94 s · $0.0649 for the run, $0.000451 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

16/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"bad_format": 18, "extra_call": 7, "gave_up_or_loop": 6, "wrong_args": 71}

Tokens: 1586690

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 24 s · $0.0165 for the run, $0.000114 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

69/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"extra_call": 1, "wrong_args": 6}

Tokens: 2017522

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 229 s · $0.1585 for the run, $0.001100 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

Run note: Not tiered: one of its 144 task runs is unscored. On a long-output task the server refused the request because the prompt plus the fixed 32,768-token answer budget exceeded its 69,632-token context (HTTP 400). Serving and configuration errors are not counted as model failures; the score is over the scored runs.

63/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"wrong_args": 12}

Tokens: Not reported

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 1214 s · $0.8400 for the run, $0.005874 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

52/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"extra_call": 7, "wrong_args": 34}

Tokens: 1798075

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 83 s · $0.0573 for the run, $0.000398 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

31/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"extra_call": 14, "wrong_args": 68, "wrong_tool": 1}

Tokens: 2353976

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 290 s · $0.2003 for the run, $0.001391 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

66/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"bad_format": 13}

Tokens: 1830591

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 608 s · $0.4207 for the run, $0.002921 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10561 hard set on GPU 1, identical argv) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183

Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2

64/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"bad_format": 8, "extra_call": 1, "wrong_args": 5}

Tokens: 2110323

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 201 s · $0.1390 for the run, $0.000965 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

63/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"bad_format": 9, "extra_call": 3, "wrong_args": 5}

Tokens: 2204967

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 516 s · $0.3566 for the run, $0.002477 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

53/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"bad_format": 1, "extra_call": 6, "wrong_args": 30}

Tokens: 2144354

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 242 s · $0.1672 for the run, $0.001161 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

48/72 scored passed; 0 unscored; 72 attempted

Failure tags: {"bad_format": 20, "extra_call": 6, "wrong_args": 25}

Tokens: 1565256

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 281 s · $0.1943 for the run, $0.001349 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

213/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"bad_format": 5, "extra_call": 7, "gave_up_or_loop": 2, "wrong_args": 19, "wrong_tool": 6}

Tokens: 773212

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 10 s · $0.0066 for the run, $0.000026 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

231/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 10, "gave_up_or_loop": 5, "wrong_args": 5, "wrong_tool": 1}

Tokens: 888087

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 23 s · $0.0162 for the run, $0.000064 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9534) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4

237/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"gave_up_or_loop": 5, "wrong_args": 7, "wrong_tool": 3}

Tokens: 1034078

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 71 s · $0.0492 for the run, $0.000195 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

205/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"bad_format": 2, "extra_call": 7, "gave_up_or_loop": 14, "wrong_args": 21, "wrong_tool": 3}

Tokens: 894011

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 12 s · $0.0085 for the run, $0.000034 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

228/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 4, "wrong_args": 18, "wrong_tool": 2}

Tokens: 1133799

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 57 s · $0.0397 for the run, $0.000158 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

185/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 14, "gave_up_or_loop": 11, "made_up_result": 2, "wrong_args": 33, "wrong_tool": 7}

Tokens: 1173418

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 26 s · $0.0180 for the run, $0.000071 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

220/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 7, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 13, "wrong_tool": 9}

Tokens: 1267484

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 59 s · $0.0411 for the run, $0.000163 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

230/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 1, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 11, "wrong_tool": 7}

Tokens: 1026831

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

226/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 5, "wrong_args": 15, "wrong_tool": 6}

Tokens: 1138207

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 78 s · $0.0539 for the run, $0.000214 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

201/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"bad_format": 3, "extra_call": 22, "wrong_args": 20, "wrong_tool": 6}

Tokens: 1179082

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 53 s · $0.0369 for the run, $0.000146 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

224/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 15, "wrong_tool": 4}

Tokens: 1023064

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 87 s · $0.0600 for the run, $0.000238 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the run, identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4

Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.

225/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 4, "gave_up_or_loop": 1, "wrong_args": 22}

Tokens: 797158

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

200/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"gave_up_or_loop": 5, "made_up_result": 1, "wrong_args": 39, "wrong_tool": 7}

Tokens: 1058883

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 36 s · $0.0249 for the run, $0.000099 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

234/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"no_call": 1, "wrong_args": 6, "wrong_tool": 11}

Tokens: 1560191

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 157 s · $0.1089 for the run, $0.000432 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

211/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 3, "gave_up_or_loop": 3, "wrong_args": 21, "wrong_tool": 14}

Tokens: 1026531

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 82 s · $0.0564 for the run, $0.000224 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

235/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"gave_up_or_loop": 2, "wrong_args": 8, "wrong_tool": 7}

Tokens: 1580185

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 332 s · $0.2294 for the run, $0.000910 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

238/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"gave_up_or_loop": 2, "wrong_tool": 12}

Tokens: 728093

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 38 s · $0.0266 for the run, $0.000105 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

234/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 2, "wrong_args": 4, "wrong_tool": 12}

Tokens: 870626

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 121 s · $0.0836 for the run, $0.000332 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183

Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2

234/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 2, "wrong_tool": 7}

Tokens: 998331

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 135 s · $0.0932 for the run, $0.000370 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

244/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 5, "wrong_args": 1, "wrong_tool": 2}

Tokens: 948341

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 144 s · $0.0993 for the run, $0.000394 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

232/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"bad_format": 5, "extra_call": 10, "wrong_args": 1, "wrong_tool": 4}

Tokens: 1098479

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 315 s · $0.2177 for the run, $0.000864 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.

252/252 scored passed; 0 unscored; 252 attempted

Failure tags: {}

Tokens: 691572

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 86 s · $0.0594 for the run, $0.000236 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.

249/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"bad_format": 2, "wrong_tool": 1}

Tokens: 783015

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 252 s · $0.1740 for the run, $0.000691 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Not qualified: Qualification failed on public task single-01: after a correct get_price call and its mock result, the model's next token was <|tool_response> (an end-of-generation id), so the reply was empty. With thinking off, neither the maker's chat template nor vLLM's example template adds a continuation cue after a tool result. One observation; cause not proven.

No score: the setup did not complete the qualification tool round trip, so its full run was not admitted.

Qualification evidence sha256: 11b2effa6e9f1f57d2fd1a76001d65b3f2a5f497ee815666bc2a8b6e8d41a18a

Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

42/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 18, "extra_call": 6, "gave_up_or_loop": 4, "wrong_args": 73, "wrong_tool": 1}

Tokens: 1340110

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 22 s · $0.0156 for the run, $0.000108 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

93/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 25, "extra_call": 5, "wrong_args": 21}

Tokens: 1790981

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 51 s · $0.0355 for the run, $0.000246 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9534 standard set; #10411 hard set, identical argv) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4

106/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 11, "wrong_args": 27}

Tokens: 2048615

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 106 s · $0.0734 for the run, $0.000510 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

42/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 18, "extra_call": 7, "gave_up_or_loop": 6, "wrong_args": 71}

Tokens: 1586690

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 24 s · $0.0165 for the run, $0.000114 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

120/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 6, "wrong_args": 18}

Tokens: 2288473

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 92 s · $0.0637 for the run, $0.000442 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

83/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 3, "gave_up_or_loop": 8, "made_up_result": 1, "wrong_args": 47, "wrong_tool": 2}

Tokens: 2473838

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 66 s · $0.0455 for the run, $0.000316 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

98/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 5, "wrong_args": 40, "wrong_tool": 1}

Tokens: 2603860

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 112 s · $0.0774 for the run, $0.000537 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

129/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 1, "wrong_args": 14}

Tokens: 2194487

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 76 s · $0.0528 for the run, $0.000367 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

116/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 3, "wrong_args": 25}

Tokens: 2457570

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 150 s · $0.1035 for the run, $0.000719 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

36/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 1, "extra_call": 15, "gave_up_or_loop": 6, "wrong_args": 81, "wrong_tool": 5}

Tokens: 2355910

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 94 s · $0.0649 for the run, $0.000451 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

102/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 12, "wrong_args": 30}

Tokens: 2137047

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 275 s · $0.1899 for the run, $0.001319 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the standard run; #10423 GPU 1 for the hard set; identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4

Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.

111/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 1, "made_up_result": 5, "wrong_args": 27}

Tokens: 1836873

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 68 s · $0.0469 for the run, $0.000326 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

58/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 12, "gave_up_or_loop": 5, "wrong_args": 68, "wrong_tool": 1}

Tokens: 2601502

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 267 s · $0.1847 for the run, $0.001283 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

127/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 4, "no_call": 1, "wrong_args": 12}

Tokens: 4304813

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 724 s · $0.5010 for the run, $0.003479 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

61/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 14, "wrong_args": 68, "wrong_tool": 1}

Tokens: 2353976

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 290 s · $0.2003 for the run, $0.001391 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

Run note: Not tiered: one of its 144 task runs is unscored. On a long-output task the server refused the request because the prompt plus the fixed 32,768-token answer budget exceeded its 69,632-token context (HTTP 400). Serving and configuration errors are not counted as model failures; the score is over the scored runs.

131/143 scored passed; 1 unscored; 144 attempted

Failure tags: {"wrong_args": 12}

Tokens: Not reported

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 1214 s · $0.8400 for the run, $0.005874 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

103/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 7, "wrong_args": 34}

Tokens: 1798075

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 83 s · $0.0573 for the run, $0.000398 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

137/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 1, "wrong_args": 6}

Tokens: 2017522

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 229 s · $0.1585 for the run, $0.001100 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10561 hard set on GPU 1, identical argv) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183

Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2

130/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 8, "extra_call": 1, "wrong_args": 5}

Tokens: 2110323

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 201 s · $0.1390 for the run, $0.000965 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

107/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 1, "extra_call": 6, "wrong_args": 30}

Tokens: 2144354

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 242 s · $0.1672 for the run, $0.001161 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

127/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 9, "extra_call": 3, "wrong_args": 5}

Tokens: 2204967

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 516 s · $0.3566 for the run, $0.002477 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

93/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 20, "extra_call": 6, "wrong_args": 25}

Tokens: 1565256

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 281 s · $0.1943 for the run, $0.001349 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

131/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 13}

Tokens: 1830591

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 608 s · $0.4207 for the run, $0.002921 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

213/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"bad_format": 5, "extra_call": 7, "gave_up_or_loop": 2, "wrong_args": 19, "wrong_tool": 6}

Tokens: 773212

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 10 s · $0.0066 for the run, $0.000026 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

231/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 10, "gave_up_or_loop": 5, "wrong_args": 5, "wrong_tool": 1}

Tokens: 888087

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 23 s · $0.0162 for the run, $0.000064 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9534) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4

237/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"gave_up_or_loop": 5, "wrong_args": 7, "wrong_tool": 3}

Tokens: 1034078

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 71 s · $0.0492 for the run, $0.000195 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

205/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"bad_format": 2, "extra_call": 7, "gave_up_or_loop": 14, "wrong_args": 21, "wrong_tool": 3}

Tokens: 894011

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 12 s · $0.0085 for the run, $0.000034 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

228/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 4, "wrong_args": 18, "wrong_tool": 2}

Tokens: 1133799

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 57 s · $0.0397 for the run, $0.000158 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

185/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 14, "gave_up_or_loop": 11, "made_up_result": 2, "wrong_args": 33, "wrong_tool": 7}

Tokens: 1173418

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 26 s · $0.0180 for the run, $0.000071 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

220/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 7, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 13, "wrong_tool": 9}

Tokens: 1267484

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 59 s · $0.0411 for the run, $0.000163 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

230/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 1, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 11, "wrong_tool": 7}

Tokens: 1026831

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

226/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 5, "wrong_args": 15, "wrong_tool": 6}

Tokens: 1138207

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 78 s · $0.0539 for the run, $0.000214 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

201/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"bad_format": 3, "extra_call": 22, "wrong_args": 20, "wrong_tool": 6}

Tokens: 1179082

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 53 s · $0.0369 for the run, $0.000146 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

224/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 15, "wrong_tool": 4}

Tokens: 1023064

GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 87 s · $0.0600 for the run, $0.000238 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the run, identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4

Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.

225/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 4, "gave_up_or_loop": 1, "wrong_args": 22}

Tokens: 797158

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

200/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"gave_up_or_loop": 5, "made_up_result": 1, "wrong_args": 39, "wrong_tool": 7}

Tokens: 1058883

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 36 s · $0.0249 for the run, $0.000099 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

234/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"no_call": 1, "wrong_args": 6, "wrong_tool": 11}

Tokens: 1560191

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 157 s · $0.1089 for the run, $0.000432 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

211/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 3, "gave_up_or_loop": 3, "wrong_args": 21, "wrong_tool": 14}

Tokens: 1026531

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 82 s · $0.0564 for the run, $0.000224 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

235/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"gave_up_or_loop": 2, "wrong_args": 8, "wrong_tool": 7}

Tokens: 1580185

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 332 s · $0.2294 for the run, $0.000910 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

238/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"gave_up_or_loop": 2, "wrong_tool": 12}

Tokens: 728093

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 38 s · $0.0266 for the run, $0.000105 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

234/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 2, "wrong_args": 4, "wrong_tool": 12}

Tokens: 870626

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 121 s · $0.0836 for the run, $0.000332 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183

Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2

234/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 2, "wrong_tool": 7}

Tokens: 998331

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 135 s · $0.0932 for the run, $0.000370 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

244/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"extra_call": 5, "wrong_args": 1, "wrong_tool": 2}

Tokens: 948341

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 144 s · $0.0993 for the run, $0.000394 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

232/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"bad_format": 5, "extra_call": 10, "wrong_args": 1, "wrong_tool": 4}

Tokens: 1098479

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 315 s · $0.2177 for the run, $0.000864 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.

252/252 scored passed; 0 unscored; 252 attempted

Failure tags: {}

Tokens: 691572

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 86 s · $0.0594 for the run, $0.000236 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.

249/252 scored passed; 0 unscored; 252 attempted

Failure tags: {"bad_format": 2, "wrong_tool": 1}

Tokens: 783015

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 252 s · $0.1740 for the run, $0.000691 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Not qualified: Qualification failed on public task single-01: after a correct get_price call and its mock result, the model's next token was <|tool_response> (an end-of-generation id), so the reply was empty. With thinking off, neither the maker's chat template nor vLLM's example template adds a continuation cue after a tool result. One observation; cause not proven.

No score: the setup did not complete the qualification tool round trip, so its full run was not admitted.

Qualification evidence sha256: 11b2effa6e9f1f57d2fd1a76001d65b3f2a5f497ee815666bc2a8b6e8d41a18a

Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

42/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 18, "extra_call": 6, "gave_up_or_loop": 4, "wrong_args": 73, "wrong_tool": 1}

Tokens: 1340110

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 22 s · $0.0156 for the run, $0.000108 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a

Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml

93/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 25, "extra_call": 5, "wrong_args": 21}

Tokens: 1790981

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 51 s · $0.0355 for the run, $0.000246 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9534 standard set; #10411 hard set, identical argv) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4

106/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 11, "wrong_args": 27}

Tokens: 2048615

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 106 s · $0.0734 for the run, $0.000510 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

42/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 18, "extra_call": 7, "gave_up_or_loop": 6, "wrong_args": 71}

Tokens: 1586690

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 24 s · $0.0165 for the run, $0.000114 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556

Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5

120/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 6, "wrong_args": 18}

Tokens: 2288473

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 92 s · $0.0637 for the run, $0.000442 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

83/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 3, "gave_up_or_loop": 8, "made_up_result": 1, "wrong_args": 47, "wrong_tool": 2}

Tokens: 2473838

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 66 s · $0.0455 for the run, $0.000316 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc

Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder

98/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 5, "wrong_args": 40, "wrong_tool": 1}

Tokens: 2603860

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 112 s · $0.0774 for the run, $0.000537 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

129/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 1, "wrong_args": 14}

Tokens: 2194487

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 76 s · $0.0528 for the run, $0.000367 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder

116/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 3, "wrong_args": 25}

Tokens: 2457570

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 150 s · $0.1035 for the run, $0.000719 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

36/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 1, "extra_call": 15, "gave_up_or_loop": 6, "wrong_args": 81, "wrong_tool": 5}

Tokens: 2355910

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 94 s · $0.0649 for the run, $0.000451 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee

Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1

102/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 12, "wrong_args": 30}

Tokens: 2137047

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 275 s · $0.1899 for the run, $0.001319 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the standard run; #10423 GPU 1 for the hard set; identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37

Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4

Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.

111/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 1, "made_up_result": 5, "wrong_args": 27}

Tokens: 1836873

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 68 s · $0.0469 for the run, $0.000326 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

58/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 12, "gave_up_or_loop": 5, "wrong_args": 68, "wrong_tool": 1}

Tokens: 2601502

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 267 s · $0.1847 for the run, $0.001283 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06

Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder

127/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 4, "no_call": 1, "wrong_args": 12}

Tokens: 4304813

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 724 s · $0.5010 for the run, $0.003479 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

61/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 14, "wrong_args": 68, "wrong_tool": 1}

Tokens: 2353976

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 290 s · $0.2003 for the run, $0.001391 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a

Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder

Run note: Not tiered: one of its 144 task runs is unscored. On a long-output task the server refused the request because the prompt plus the fixed 32,768-token answer budget exceeded its 69,632-token context (HTTP 400). Serving and configuration errors are not counted as model failures; the score is over the scored runs.

131/143 scored passed; 1 unscored; 144 attempted

Failure tags: {"wrong_args": 12}

Tokens: Not reported

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 1214 s · $0.8400 for the run, $0.005874 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

103/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 7, "wrong_args": 34}

Tokens: 1798075

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 83 s · $0.0573 for the run, $0.000398 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d

Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4

137/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"extra_call": 1, "wrong_args": 6}

Tokens: 2017522

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 229 s · $0.1585 for the run, $0.001100 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10561 hard set on GPU 1, identical argv) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183

Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2

130/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 8, "extra_call": 1, "wrong_args": 5}

Tokens: 2110323

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 201 s · $0.1390 for the run, $0.000965 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

107/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 1, "extra_call": 6, "wrong_args": 30}

Tokens: 2144354

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 242 s · $0.1672 for the run, $0.001161 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder

127/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 9, "extra_call": 3, "wrong_args": 5}

Tokens: 2204967

GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 516 s · $0.3566 for the run, $0.002477 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

93/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 20, "extra_call": 6, "wrong_args": 25}

Tokens: 1565256

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 281 s · $0.1943 for the run, $0.001349 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475

Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4

131/144 scored passed; 0 unscored; 144 attempted

Failure tags: {"bad_format": 13}

Tokens: 1830591

GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 608 s · $0.4207 for the run, $0.002921 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)

Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md

Public per-task results (pass/fail by task id)

Methodology

All tool observations use fixed synthetic finance fixtures, never real trades. Model answers are measured independently; tool-calling scores in model cards are maker-reported.

Original and renamed tracks are paired. Three repetitions per task; serving/configuration and verified template failures remain unscored. Public and private scores are separate; only public per-task details leave local storage.

Size class by total parameters, the same figures as the suite's other benches (from the maker's card, checked against the exact weights run): Tiny under 6B, Small 6B–15B, Medium over 15B up to 40B, Large over 40B.

No chance correction is applied. Tiers within each size class: for each pair of setups, every task (42 per track, public and private) is one paired item, won by the setup that passed more of its 3 repetitions; ties drop out. Exact two-sided sign test, Holm-corrected over all pairs of the class, p < 0.05. Rows in accuracy order: the top row starts Tier 1, and a row starts the next tier when the test separates it from its tier's top row. Accuracy: both tracks (84 tasks); Renamed tools: each track (42 tasks). – : not tiered (class not verified, incomplete or not qualified). Every pair's wins, losses, ties and p-values. Hard set: the same test over its 24 tasks per track; its pairs. Unknown cost and tokens stay unreported.

Public results · Common feed · Run coverage