Tasks
Each task is a short request from a user. The model sees the request and a list of tools, decides whether and how to call them, gets the tool results back and answers. A task passes only when the calls, their arguments and order, and the resulting account state are right; on restraint tasks, only when no tool is called. 42 tasks per track, 7 of each type, 3 repetitions each.
The 8 finance tools
get_price: The current price of a stock or coin (AAPL, MSFT, TSLA, BTC, ETH) in a chosen currency.get_ohlc: Daily open, high, low and close prices for a date range.get_balance: How much cash the paper account holds, per currency.get_positions: How much of each stock or coin the paper account holds.convert: Converts an amount between USD, EUR, GBP and JPY at fixed rates.place_order: Places a dry-run buy or sell order (market or limit) on the paper account; nothing is traded.cancel_order: Cancels an open dry-run limit order and releases the money or coins it held back.search_news: Searches a fixed set of made-up headlines by text and date.
Decoy tools
22 tools that have nothing to do with finance (get_weather, list_calendar, search_email, translate_text, set_alarm, find_recipe, …). Some selection and restraint tasks offer all 30 tools, the 8 finance tools and the decoys together. They test whether the model still picks the right tool from a long list, and leaves an unrelated tool alone just because it is there.
The six task types
- Single call: One tool call with the right arguments, then the answer.
- Tool selection: Pick the right tool from a long list that includes unrelated decoys.
- Arguments: Exact argument values and formats: dates, tickers, currencies, empty or default fields.
- Multi-step: Several calls in a row, where later calls use earlier results and the account state must end right.
- Restraint: No tool fits, or the request asks for something the tools cannot do: the right answer calls nothing and invents nothing.
- Recovery: A deliberate error (an unknown ticker, too little cash, a rate limit) that must be handled as instructed.
Renamed tools
Every task also exists in a renamed version: the same request and the same tools, but every tool and argument has a neutral name (get_price becomes fetch_quote, symbol becomes ticker). A model that only recognises familiar names fails there; a model that reads the tool descriptions does not. Both versions are scored, and the Renamed tools tab shows them side by side.
Synthetic only
All data are fixed synthetic fixtures (prices, balances, news), "today" is 2026-11-12, and orders are dry runs on a paper account. Nothing touches a market or a real account, and no real trade is ever made.
Not published
The tasks themselves, their expected answers and the models' replies are not published, so models cannot be trained on them. The tool definitions and each task's pass or fail (by task id) are. The "Public" and "Private" columns split the scores by an older public/private task split. Tool definitions · Tool definitions, renamed · Per-task results (pass/fail by task id)
Hard set
A second, harder set of 24 tasks per track (also in a renamed version) tests longer chains and stricter instruction following. A task that says "in this order" is graded in that order; an order call sent together with other calls in one message counts as a format failure, because the model must see each order result first. Same tools, fixtures and grading; 4 tasks of each type:
- Long chain: 4 to 6 dependent calls; each argument comes from earlier results.
- Computed arguments: Date arithmetic, currency conversion, rounding to whole units, totals across holdings.
- Multi-constraint: Two or three rules at once: order, size cap, forbidden tool or currency, "only if".
- Branching: The next step depends on fetched values, including cases where the right action is to do nothing.
- Cascading recovery: Two different errors in one task (rate limit, then too little cash, ...), each handled as instructed.
- Long output: Find the right value inside a long price history or a crowded account.
Toolbench
Results · 42 tasks per track · 3 repetitions · original/renamed are pairedHard set results · 24 tasks per track · 3 repetitions · original/renamed are paired
Accuracy Task pass rate — higher is better.
| Tiers within each size class: for each pair of setups, every task (42 per track, public and private) is one paired item, won by the setup that passed more of its 3 repetitions; ties drop out. Exact two-sided sign test, Holm-corrected over all pairs of the class, p < 0.05. Rows in accuracy order: the top row starts Tier 1, and a row starts the next tier when the test separates it from its tier's top row. Accuracy: both tracks (84 tasks); Renamed tools: each track (42 tasks). – : not tiered (class not verified, incomplete or not qualified). | Model | Accuracy | Public | Private |
|---|---|---|---|---|
| Tiny | ||||
| 1 | Gemma 4 E2B onOpen · 5.1BDetails | 94.0% | 94.1% | 93.9% |
| 1 | Granite 4.2 3B onOpen · 3.0BDetails | 92.9% | 90.3% | 100.0% |
| 1 | LFM3-1.7B onOpen · 1.7BDetails | 91.7% | 91.4% | 92.4% |
| 1 | Qwen3.5 4B offOpen · 4.7BDetails | 91.3% | 88.7% | 98.5% |
| 1 | MiniCPM5-2B onOpen · 2.5BDetails | 90.5% | 87.1% | 100.0% |
| 1 | Qwen3.5 4B onOpen · 4.7BDetails | 89.7% | 86.0% | 100.0% |
| 1 | Gemma 4 E2B (tool-result cue template) offOpen · 5.1BDetails | 89.3% | 85.5% | 100.0% |
| 1 | Raon-1 nano onOpen · 3.1BDetails | 88.9% | 85.5% | 98.5% |
| 1 | Qwen3.5-2B onOpen · 2.0BDetails | 87.3% | 83.9% | 97.0% |
| 1 | LFM3-1.7B offOpen · 1.7BDetails | 84.5% | 79.0% | 100.0% |
| 2 | MiniCPM5-2B offOpen · 2.5BDetails | 81.3% | 76.3% | 95.5% |
| 2 | Raon-1 nano offOpen · 3.1BDetails | 79.8% | 72.6% | 100.0% |
| 2 | Granite 4.2 3B offOpen · 3.0BDetails | 79.4% | 75.8% | 89.4% |
| 2 | Qwen3.5-2B offOpen · 2.0BDetails | 73.4% | 69.9% | 83.3% |
| – | Gemma 4 E2B offOpen · 5.1BDetails | Not qualified | Not scored | Not scored |
| Small | ||||
| 1 | Gemma 4 12B offOpen · 12BDetails | 94.4% | 93.5% | 97.0% |
| 1 | Granite 4.2 8B onOpen · 8.0BDetails | 93.3% | 91.4% | 98.5% |
| 1 | Gemma 4 12B onOpen · 12BDetails | 92.9% | 90.3% | 100.0% |
| 2 | Granite 4.2 8B offOpen · 8.0BDetails | 83.7% | 83.3% | 84.8% |
| Medium | ||||
| 1 | Gemma 4 31B offOpen · 31BDetails | 100.0% | 100.0% | 100.0% |
| 1 | Gemma 4 31B onOpen · 31BDetails | 98.8% | 99.5% | 97.0% |
| 1 | Qwen3.8-27B offOpen · 27BDetails | 96.8% | 95.7% | 100.0% |
| 1 | LFM2.5-24B-A2B onOpen · 24B MoE · 2.5B activeDetails | 92.9% | 90.9% | 98.5% |
| 2 | Qwen3.8-27B onOpen · 27BDetails | 92.1% | 89.8% | 98.5% |
No row of this kind yet.
Task pass rate averaged over the preregistered repetitions. A pass requires the expected tool calls, arguments, order and mock-state checks. Verified serving or template failures remain unscored, not model zeroes.
Tiers within each size class: for each pair of setups, every task (42 per track, public and private) is one paired item, won by the setup that passed more of its 3 repetitions; ties drop out. Exact two-sided sign test, Holm-corrected over all pairs of the class, p < 0.05. Rows in accuracy order: the top row starts Tier 1, and a row starts the next tier when the test separates it from its tier's top row. Accuracy: both tracks (84 tasks); Renamed tools: each track (42 tasks). – : not tiered (class not verified, incomplete or not qualified). Serving and template failures are unscored, not model zeroes.
Accuracy Task pass rate — higher is better.
| Tiers within each size class: for each pair of setups, every task (24 per track, public and private) is one paired item, won by the setup that passed more of its 3 repetitions; ties drop out. Exact two-sided sign test, Holm-corrected over all pairs of the class, p < 0.05. Rows in accuracy order: the top row starts Tier 1, and a row starts the next tier when the test separates it from its tier's top row. Accuracy: both tracks (48 tasks); Renamed tools: each track (24 tasks). – : not tiered (class not verified, incomplete or not qualified). | Model | Accuracy | Public | Private |
|---|---|---|---|---|
| Tiny | ||||
| 1 | Qwen3.5 4B offOpen · 4.7BDetails | 89.6% | 88.0% | 94.4% |
| 1 | Granite 4.2 3B onOpen · 3.0BDetails | 88.2% | 85.2% | 97.2% |
| 1 | MiniCPM5-2B onOpen · 2.5BDetails | 83.3% | 84.3% | 80.6% |
| 1 | Qwen3.5 4B onOpen · 4.7BDetails | 80.6% | 81.5% | 77.8% |
| 1 | Gemma 4 E2B (tool-result cue template) offOpen · 5.1BDetails | 77.1% | 78.7% | 72.2% |
| 1 | Gemma 4 E2B onOpen · 5.1BDetails | 73.6% | 69.4% | 86.1% |
| 2 | Raon-1 nano onOpen · 3.1BDetails | 70.8% | 67.6% | 80.6% |
| 2 | Qwen3.5-2B onOpen · 2.0BDetails | 68.1% | 66.7% | 72.2% |
| 2 | LFM3-1.7B onOpen · 1.7BDetails | 64.6% | 59.3% | 80.6% |
| 2 | Qwen3.5-2B offOpen · 2.0BDetails | 57.6% | 54.6% | 66.7% |
| 3 | Granite 4.2 3B offOpen · 3.0BDetails | 40.3% | 43.5% | 30.6% |
| 3 | LFM3-1.7B offOpen · 1.7BDetails | 29.2% | 31.5% | 22.2% |
| 3 | MiniCPM5-2B offOpen · 2.5BDetails | 29.2% | 33.3% | 16.7% |
| 3 | Raon-1 nano offOpen · 3.1BDetails | 25.0% | 26.9% | 19.4% |
| Small | ||||
| 1 | Gemma 4 12B onOpen · 12BDetails | 95.1% | 96.3% | 91.7% |
| – | Granite 4.2 8B onOpen · 8.0BDetails | 91.6% | 91.6% | 91.7% |
| 2 | Gemma 4 12B offOpen · 12BDetails | 71.5% | 73.1% | 66.7% |
| 3 | Granite 4.2 8B offOpen · 8.0BDetails | 42.4% | 42.6% | 41.7% |
| Medium | ||||
| 1 | Gemma 4 31B onOpen · 31BDetails | 91.0% | 88.0% | 100.0% |
| 1 | LFM2.5-24B-A2B onOpen · 24B MoE · 2.5B activeDetails | 90.3% | 88.9% | 94.4% |
| 1 | Qwen3.8-27B onOpen · 27BDetails | 88.2% | 88.0% | 88.9% |
| 1 | Qwen3.8-27B offOpen · 27BDetails | 74.3% | 70.4% | 86.1% |
| 2 | Gemma 4 31B offOpen · 31BDetails | 64.6% | 58.3% | 83.3% |
No row of this kind yet.
Task pass rate averaged over the preregistered repetitions. A pass requires the expected tool calls, arguments, order and mock-state checks. Verified serving or template failures remain unscored, not model zeroes.
Tiers within each size class: for each pair of setups, every task (24 per track, public and private) is one paired item, won by the setup that passed more of its 3 repetitions; ties drop out. Exact two-sided sign test, Holm-corrected over all pairs of the class, p < 0.05. Rows in accuracy order: the top row starts Tier 1, and a row starts the next tier when the test separates it from its tier's top row. Accuracy: both tracks (48 tasks); Renamed tools: each track (24 tasks). – : not tiered (class not verified, incomplete or not qualified). Serving and template failures are unscored, not model zeroes.
Accuracy by category Per-category pass rate — higher is better.
| Model | Accuracy by category | Scored attempts | Unscored attempts | |
|---|---|---|---|---|
| – | LFM3-1.7B off · singleOpen · 1.7BDetails | 100.0% | 42 | 0 |
| – | LFM3-1.7B off · selectionOpen · 1.7BDetails | 64.3% | 42 | 0 |
| – | LFM3-1.7B off · argsOpen · 1.7BDetails | 100.0% | 42 | 0 |
| – | LFM3-1.7B off · multi-stepOpen · 1.7BDetails | 59.5% | 42 | 0 |
| – | LFM3-1.7B off · restraintOpen · 1.7BDetails | 100.0% | 42 | 0 |
| – | LFM3-1.7B off · recoveryOpen · 1.7BDetails | 83.3% | 42 | 0 |
| – | LFM3-1.7B on · singleOpen · 1.7BDetails | 100.0% | 42 | 0 |
| – | LFM3-1.7B on · selectionOpen · 1.7BDetails | 97.6% | 42 | 0 |
| – | LFM3-1.7B on · argsOpen · 1.7BDetails | 100.0% | 42 | 0 |
| – | LFM3-1.7B on · multi-stepOpen · 1.7BDetails | 76.2% | 42 | 0 |
| – | LFM3-1.7B on · restraintOpen · 1.7BDetails | 90.5% | 42 | 0 |
| – | LFM3-1.7B on · recoveryOpen · 1.7BDetails | 85.7% | 42 | 0 |
| – | Gemma 4 E2B on · singleOpen · 5.1BDetails | 100.0% | 42 | 0 |
| – | Gemma 4 E2B on · selectionOpen · 5.1BDetails | 85.7% | 42 | 0 |
| – | Gemma 4 E2B on · argsOpen · 5.1BDetails | 100.0% | 42 | 0 |
| – | Gemma 4 E2B on · multi-stepOpen · 5.1BDetails | 92.9% | 42 | 0 |
| – | Gemma 4 E2B on · restraintOpen · 5.1BDetails | 100.0% | 42 | 0 |
| – | Gemma 4 E2B on · recoveryOpen · 5.1BDetails | 85.7% | 42 | 0 |
| – | MiniCPM5-2B off · singleOpen · 2.5BDetails | 100.0% | 42 | 0 |
| – | MiniCPM5-2B off · selectionOpen · 2.5BDetails | 71.4% | 42 | 0 |
| – | MiniCPM5-2B off · argsOpen · 2.5BDetails | 100.0% | 42 | 0 |
| – | MiniCPM5-2B off · multi-stepOpen · 2.5BDetails | 38.1% | 42 | 0 |
| – | MiniCPM5-2B off · restraintOpen · 2.5BDetails | 97.6% | 42 | 0 |
| – | MiniCPM5-2B off · recoveryOpen · 2.5BDetails | 81.0% | 42 | 0 |
| – | MiniCPM5-2B on · singleOpen · 2.5BDetails | 100.0% | 42 | 0 |
| – | MiniCPM5-2B on · selectionOpen · 2.5BDetails | 71.4% | 42 | 0 |
| – | MiniCPM5-2B on · argsOpen · 2.5BDetails | 100.0% | 42 | 0 |
| – | MiniCPM5-2B on · multi-stepOpen · 2.5BDetails | 76.2% | 42 | 0 |
| – | MiniCPM5-2B on · restraintOpen · 2.5BDetails | 100.0% | 42 | 0 |
| – | MiniCPM5-2B on · recoveryOpen · 2.5BDetails | 95.2% | 42 | 0 |
| – | Qwen3.5-2B off · singleOpen · 2.0BDetails | 95.2% | 42 | 0 |
| – | Qwen3.5-2B off · selectionOpen · 2.0BDetails | 61.9% | 42 | 0 |
| – | Qwen3.5-2B off · argsOpen · 2.0BDetails | 88.1% | 42 | 0 |
| – | Qwen3.5-2B off · multi-stepOpen · 2.0BDetails | 45.2% | 42 | 0 |
| – | Qwen3.5-2B off · restraintOpen · 2.0BDetails | 83.3% | 42 | 0 |
| – | Qwen3.5-2B off · recoveryOpen · 2.0BDetails | 66.7% | 42 | 0 |
| – | Qwen3.5-2B on · singleOpen · 2.0BDetails | 100.0% | 42 | 0 |
| – | Qwen3.5-2B on · selectionOpen · 2.0BDetails | 81.0% | 42 | 0 |
| – | Qwen3.5-2B on · argsOpen · 2.0BDetails | 100.0% | 42 | 0 |
| – | Qwen3.5-2B on · multi-stepOpen · 2.0BDetails | 85.7% | 42 | 0 |
| – | Qwen3.5-2B on · restraintOpen · 2.0BDetails | 88.1% | 42 | 0 |
| – | Qwen3.5-2B on · recoveryOpen · 2.0BDetails | 69.0% | 42 | 0 |
| – | Qwen3.5 4B off · singleOpen · 4.7BDetails | 100.0% | 42 | 0 |
| – | Qwen3.5 4B off · selectionOpen · 4.7BDetails | 88.1% | 42 | 0 |
| – | Qwen3.5 4B off · argsOpen · 4.7BDetails | 100.0% | 42 | 0 |
| – | Qwen3.5 4B off · multi-stepOpen · 4.7BDetails | 100.0% | 42 | 0 |
| – | Qwen3.5 4B off · restraintOpen · 4.7BDetails | 95.2% | 42 | 0 |
| – | Qwen3.5 4B off · recoveryOpen · 4.7BDetails | 64.3% | 42 | 0 |
| – | Qwen3.5 4B on · singleOpen · 4.7BDetails | 100.0% | 42 | 0 |
| – | Qwen3.5 4B on · selectionOpen · 4.7BDetails | 95.2% | 42 | 0 |
| – | Qwen3.5 4B on · argsOpen · 4.7BDetails | 97.6% | 42 | 0 |
| – | Qwen3.5 4B on · multi-stepOpen · 4.7BDetails | 71.4% | 42 | 0 |
| – | Qwen3.5 4B on · restraintOpen · 4.7BDetails | 95.2% | 42 | 0 |
| – | Qwen3.5 4B on · recoveryOpen · 4.7BDetails | 78.6% | 42 | 0 |
| – | Raon-1 nano off · singleOpen · 3.1BDetails | 100.0% | 42 | 0 |
| – | Raon-1 nano off · selectionOpen · 3.1BDetails | 71.4% | 42 | 0 |
| – | Raon-1 nano off · argsOpen · 3.1BDetails | 95.2% | 42 | 0 |
| – | Raon-1 nano off · multi-stepOpen · 3.1BDetails | 57.1% | 42 | 0 |
| – | Raon-1 nano off · restraintOpen · 3.1BDetails | 76.2% | 42 | 0 |
| – | Raon-1 nano off · recoveryOpen · 3.1BDetails | 78.6% | 42 | 0 |
| – | Raon-1 nano on · singleOpen · 3.1BDetails | 97.6% | 42 | 0 |
| – | Raon-1 nano on · selectionOpen · 3.1BDetails | 85.7% | 42 | 0 |
| – | Raon-1 nano on · argsOpen · 3.1BDetails | 100.0% | 42 | 0 |
| – | Raon-1 nano on · multi-stepOpen · 3.1BDetails | 61.9% | 42 | 0 |
| – | Raon-1 nano on · restraintOpen · 3.1BDetails | 100.0% | 42 | 0 |
| – | Raon-1 nano on · recoveryOpen · 3.1BDetails | 88.1% | 42 | 0 |
| – | Gemma 4 E2B (tool-result cue template) off · singleOpen · 5.1BDetails | 100.0% | 42 | 0 |
| – | Gemma 4 E2B (tool-result cue template) off · selectionOpen · 5.1BDetails | 71.4% | 42 | 0 |
| – | Gemma 4 E2B (tool-result cue template) off · argsOpen · 5.1BDetails | 100.0% | 42 | 0 |
| – | Gemma 4 E2B (tool-result cue template) off · multi-stepOpen · 5.1BDetails | 73.8% | 42 | 0 |
| – | Gemma 4 E2B (tool-result cue template) off · restraintOpen · 5.1BDetails | 92.9% | 42 | 0 |
| – | Gemma 4 E2B (tool-result cue template) off · recoveryOpen · 5.1BDetails | 97.6% | 42 | 0 |
| – | Granite 4.2 3B off · singleOpen · 3.0BDetails | 100.0% | 42 | 0 |
| – | Granite 4.2 3B off · selectionOpen · 3.0BDetails | 59.5% | 42 | 0 |
| – | Granite 4.2 3B off · argsOpen · 3.0BDetails | 100.0% | 42 | 0 |
| – | Granite 4.2 3B off · multi-stepOpen · 3.0BDetails | 61.9% | 42 | 0 |
| – | Granite 4.2 3B off · restraintOpen · 3.0BDetails | 100.0% | 42 | 0 |
| – | Granite 4.2 3B off · recoveryOpen · 3.0BDetails | 54.8% | 42 | 0 |
| – | Granite 4.2 3B on · singleOpen · 3.0BDetails | 97.6% | 42 | 0 |
| – | Granite 4.2 3B on · selectionOpen · 3.0BDetails | 85.7% | 42 | 0 |
| – | Granite 4.2 3B on · argsOpen · 3.0BDetails | 100.0% | 42 | 0 |
| – | Granite 4.2 3B on · multi-stepOpen · 3.0BDetails | 100.0% | 42 | 0 |
| – | Granite 4.2 3B on · restraintOpen · 3.0BDetails | 100.0% | 42 | 0 |
| – | Granite 4.2 3B on · recoveryOpen · 3.0BDetails | 73.8% | 42 | 0 |
| – | Granite 4.2 8B off · singleOpen · 8.0BDetails | 100.0% | 42 | 0 |
| – | Granite 4.2 8B off · selectionOpen · 8.0BDetails | 76.2% | 42 | 0 |
| – | Granite 4.2 8B off · argsOpen · 8.0BDetails | 97.6% | 42 | 0 |
| – | Granite 4.2 8B off · multi-stepOpen · 8.0BDetails | 64.3% | 42 | 0 |
| – | Granite 4.2 8B off · restraintOpen · 8.0BDetails | 100.0% | 42 | 0 |
| – | Granite 4.2 8B off · recoveryOpen · 8.0BDetails | 64.3% | 42 | 0 |
| – | Granite 4.2 8B on · singleOpen · 8.0BDetails | 100.0% | 42 | 0 |
| – | Granite 4.2 8B on · selectionOpen · 8.0BDetails | 83.3% | 42 | 0 |
| – | Granite 4.2 8B on · argsOpen · 8.0BDetails | 100.0% | 42 | 0 |
| – | Granite 4.2 8B on · multi-stepOpen · 8.0BDetails | 95.2% | 42 | 0 |
| – | Granite 4.2 8B on · restraintOpen · 8.0BDetails | 100.0% | 42 | 0 |
| – | Granite 4.2 8B on · recoveryOpen · 8.0BDetails | 81.0% | 42 | 0 |
| – | Gemma 4 12B off · singleOpen · 12BDetails | 100.0% | 42 | 0 |
| – | Gemma 4 12B off · selectionOpen · 12BDetails | 100.0% | 42 | 0 |
| – | Gemma 4 12B off · argsOpen · 12BDetails | 100.0% | 42 | 0 |
| – | Gemma 4 12B off · multi-stepOpen · 12BDetails | 100.0% | 42 | 0 |
| – | Gemma 4 12B off · restraintOpen · 12BDetails | 100.0% | 42 | 0 |
| – | Gemma 4 12B off · recoveryOpen · 12BDetails | 66.7% | 42 | 0 |
| – | Gemma 4 12B on · singleOpen · 12BDetails | 100.0% | 42 | 0 |
| – | Gemma 4 12B on · selectionOpen · 12BDetails | 90.5% | 42 | 0 |
| – | Gemma 4 12B on · argsOpen · 12BDetails | 100.0% | 42 | 0 |
| – | Gemma 4 12B on · multi-stepOpen · 12BDetails | 97.6% | 42 | 0 |
| – | Gemma 4 12B on · restraintOpen · 12BDetails | 100.0% | 42 | 0 |
| – | Gemma 4 12B on · recoveryOpen · 12BDetails | 69.0% | 42 | 0 |
| – | LFM2.5-24B-A2B on · singleOpen · 24B MoE · 2.5B activeDetails | 100.0% | 42 | 0 |
| – | LFM2.5-24B-A2B on · selectionOpen · 24B MoE · 2.5B activeDetails | 88.1% | 42 | 0 |
| – | LFM2.5-24B-A2B on · argsOpen · 24B MoE · 2.5B activeDetails | 100.0% | 42 | 0 |
| – | LFM2.5-24B-A2B on · multi-stepOpen · 24B MoE · 2.5B activeDetails | 100.0% | 42 | 0 |
| – | LFM2.5-24B-A2B on · restraintOpen · 24B MoE · 2.5B activeDetails | 88.1% | 42 | 0 |
| – | LFM2.5-24B-A2B on · recoveryOpen · 24B MoE · 2.5B activeDetails | 81.0% | 42 | 0 |
| – | Qwen3.8-27B off · singleOpen · 27BDetails | 100.0% | 42 | 0 |
| – | Qwen3.8-27B off · selectionOpen · 27BDetails | 100.0% | 42 | 0 |
| – | Qwen3.8-27B off · argsOpen · 27BDetails | 100.0% | 42 | 0 |
| – | Qwen3.8-27B off · multi-stepOpen · 27BDetails | 88.1% | 42 | 0 |
| – | Qwen3.8-27B off · restraintOpen · 27BDetails | 100.0% | 42 | 0 |
| – | Qwen3.8-27B off · recoveryOpen · 27BDetails | 92.9% | 42 | 0 |
| – | Qwen3.8-27B on · singleOpen · 27BDetails | 100.0% | 42 | 0 |
| – | Qwen3.8-27B on · selectionOpen · 27BDetails | 97.6% | 42 | 0 |
| – | Qwen3.8-27B on · argsOpen · 27BDetails | 88.1% | 42 | 0 |
| – | Qwen3.8-27B on · multi-stepOpen · 27BDetails | 85.7% | 42 | 0 |
| – | Qwen3.8-27B on · restraintOpen · 27BDetails | 100.0% | 42 | 0 |
| – | Qwen3.8-27B on · recoveryOpen · 27BDetails | 81.0% | 42 | 0 |
| – | Gemma 4 31B off · singleOpen · 31BDetails | 100.0% | 42 | 0 |
| – | Gemma 4 31B off · selectionOpen · 31BDetails | 100.0% | 42 | 0 |
| – | Gemma 4 31B off · argsOpen · 31BDetails | 100.0% | 42 | 0 |
| – | Gemma 4 31B off · multi-stepOpen · 31BDetails | 100.0% | 42 | 0 |
| – | Gemma 4 31B off · restraintOpen · 31BDetails | 100.0% | 42 | 0 |
| – | Gemma 4 31B off · recoveryOpen · 31BDetails | 100.0% | 42 | 0 |
| – | Gemma 4 31B on · singleOpen · 31BDetails | 100.0% | 42 | 0 |
| – | Gemma 4 31B on · selectionOpen · 31BDetails | 100.0% | 42 | 0 |
| – | Gemma 4 31B on · argsOpen · 31BDetails | 100.0% | 42 | 0 |
| – | Gemma 4 31B on · multi-stepOpen · 31BDetails | 95.2% | 42 | 0 |
| – | Gemma 4 31B on · restraintOpen · 31BDetails | 100.0% | 42 | 0 |
| – | Gemma 4 31B on · recoveryOpen · 31BDetails | 97.6% | 42 | 0 |
| – | Gemma 4 E2B offOpen · 5.1BDetails | Not qualified | Not scored | Not scored |
No row of this kind yet.
Task pass rates shown separately for single, selection, arguments, multi-step, restraint and recovery tasks. These are categories of the same task bank, not independent benchmark banks.
Not tiered. Serving and template failures are unscored, not model zeroes.
Accuracy by category Per-category pass rate — higher is better.
| Model | Accuracy by category | Scored attempts | Unscored attempts | |
|---|---|---|---|---|
| – | LFM3-1.7B off · long-chainOpen · 1.7BDetails | 0.0% | 24 | 0 |
| – | LFM3-1.7B off · computed-argsOpen · 1.7BDetails | 8.3% | 24 | 0 |
| – | LFM3-1.7B off · multi-constraintOpen · 1.7BDetails | 41.7% | 24 | 0 |
| – | LFM3-1.7B off · branchingOpen · 1.7BDetails | 75.0% | 24 | 0 |
| – | LFM3-1.7B off · cascading-recoveryOpen · 1.7BDetails | 37.5% | 24 | 0 |
| – | LFM3-1.7B off · long-outputOpen · 1.7BDetails | 12.5% | 24 | 0 |
| – | LFM3-1.7B on · long-chainOpen · 1.7BDetails | 50.0% | 24 | 0 |
| – | LFM3-1.7B on · computed-argsOpen · 1.7BDetails | 75.0% | 24 | 0 |
| – | LFM3-1.7B on · multi-constraintOpen · 1.7BDetails | 79.2% | 24 | 0 |
| – | LFM3-1.7B on · branchingOpen · 1.7BDetails | 100.0% | 24 | 0 |
| – | LFM3-1.7B on · cascading-recoveryOpen · 1.7BDetails | 58.3% | 24 | 0 |
| – | LFM3-1.7B on · long-outputOpen · 1.7BDetails | 25.0% | 24 | 0 |
| – | Gemma 4 E2B on · long-chainOpen · 5.1BDetails | 79.2% | 24 | 0 |
| – | Gemma 4 E2B on · computed-argsOpen · 5.1BDetails | 62.5% | 24 | 0 |
| – | Gemma 4 E2B on · multi-constraintOpen · 5.1BDetails | 83.3% | 24 | 0 |
| – | Gemma 4 E2B on · branchingOpen · 5.1BDetails | 100.0% | 24 | 0 |
| – | Gemma 4 E2B on · cascading-recoveryOpen · 5.1BDetails | 87.5% | 24 | 0 |
| – | Gemma 4 E2B on · long-outputOpen · 5.1BDetails | 29.2% | 24 | 0 |
| – | MiniCPM5-2B off · long-chainOpen · 2.5BDetails | 8.3% | 24 | 0 |
| – | MiniCPM5-2B off · computed-argsOpen · 2.5BDetails | 16.7% | 24 | 0 |
| – | MiniCPM5-2B off · multi-constraintOpen · 2.5BDetails | 8.3% | 24 | 0 |
| – | MiniCPM5-2B off · branchingOpen · 2.5BDetails | 66.7% | 24 | 0 |
| – | MiniCPM5-2B off · cascading-recoveryOpen · 2.5BDetails | 70.8% | 24 | 0 |
| – | MiniCPM5-2B off · long-outputOpen · 2.5BDetails | 4.2% | 24 | 0 |
| – | MiniCPM5-2B on · long-chainOpen · 2.5BDetails | 87.5% | 24 | 0 |
| – | MiniCPM5-2B on · computed-argsOpen · 2.5BDetails | 70.8% | 24 | 0 |
| – | MiniCPM5-2B on · multi-constraintOpen · 2.5BDetails | 91.7% | 24 | 0 |
| – | MiniCPM5-2B on · branchingOpen · 2.5BDetails | 100.0% | 24 | 0 |
| – | MiniCPM5-2B on · cascading-recoveryOpen · 2.5BDetails | 95.8% | 24 | 0 |
| – | MiniCPM5-2B on · long-outputOpen · 2.5BDetails | 54.2% | 24 | 0 |
| – | Qwen3.5-2B off · long-chainOpen · 2.0BDetails | 75.0% | 24 | 0 |
| – | Qwen3.5-2B off · computed-argsOpen · 2.0BDetails | 29.2% | 24 | 0 |
| – | Qwen3.5-2B off · multi-constraintOpen · 2.0BDetails | 62.5% | 24 | 0 |
| – | Qwen3.5-2B off · branchingOpen · 2.0BDetails | 83.3% | 24 | 0 |
| – | Qwen3.5-2B off · cascading-recoveryOpen · 2.0BDetails | 87.5% | 24 | 0 |
| – | Qwen3.5-2B off · long-outputOpen · 2.0BDetails | 8.3% | 24 | 0 |
| – | Qwen3.5-2B on · long-chainOpen · 2.0BDetails | 75.0% | 24 | 0 |
| – | Qwen3.5-2B on · computed-argsOpen · 2.0BDetails | 45.8% | 24 | 0 |
| – | Qwen3.5-2B on · multi-constraintOpen · 2.0BDetails | 83.3% | 24 | 0 |
| – | Qwen3.5-2B on · branchingOpen · 2.0BDetails | 100.0% | 24 | 0 |
| – | Qwen3.5-2B on · cascading-recoveryOpen · 2.0BDetails | 83.3% | 24 | 0 |
| – | Qwen3.5-2B on · long-outputOpen · 2.0BDetails | 20.8% | 24 | 0 |
| – | Qwen3.5 4B off · long-chainOpen · 4.7BDetails | 100.0% | 24 | 0 |
| – | Qwen3.5 4B off · computed-argsOpen · 4.7BDetails | 87.5% | 24 | 0 |
| – | Qwen3.5 4B off · multi-constraintOpen · 4.7BDetails | 95.8% | 24 | 0 |
| – | Qwen3.5 4B off · branchingOpen · 4.7BDetails | 100.0% | 24 | 0 |
| – | Qwen3.5 4B off · cascading-recoveryOpen · 4.7BDetails | 91.7% | 24 | 0 |
| – | Qwen3.5 4B off · long-outputOpen · 4.7BDetails | 62.5% | 24 | 0 |
| – | Qwen3.5 4B on · long-chainOpen · 4.7BDetails | 87.5% | 24 | 0 |
| – | Qwen3.5 4B on · computed-argsOpen · 4.7BDetails | 91.7% | 24 | 0 |
| – | Qwen3.5 4B on · multi-constraintOpen · 4.7BDetails | 87.5% | 24 | 0 |
| – | Qwen3.5 4B on · branchingOpen · 4.7BDetails | 95.8% | 24 | 0 |
| – | Qwen3.5 4B on · cascading-recoveryOpen · 4.7BDetails | 83.3% | 24 | 0 |
| – | Qwen3.5 4B on · long-outputOpen · 4.7BDetails | 37.5% | 24 | 0 |
| – | Raon-1 nano off · long-chainOpen · 3.1BDetails | 4.2% | 24 | 0 |
| – | Raon-1 nano off · computed-argsOpen · 3.1BDetails | 4.2% | 24 | 0 |
| – | Raon-1 nano off · multi-constraintOpen · 3.1BDetails | 8.3% | 24 | 0 |
| – | Raon-1 nano off · branchingOpen · 3.1BDetails | 45.8% | 24 | 0 |
| – | Raon-1 nano off · cascading-recoveryOpen · 3.1BDetails | 87.5% | 24 | 0 |
| – | Raon-1 nano off · long-outputOpen · 3.1BDetails | 0.0% | 24 | 0 |
| – | Raon-1 nano on · long-chainOpen · 3.1BDetails | 66.7% | 24 | 0 |
| – | Raon-1 nano on · computed-argsOpen · 3.1BDetails | 45.8% | 24 | 0 |
| – | Raon-1 nano on · multi-constraintOpen · 3.1BDetails | 95.8% | 24 | 0 |
| – | Raon-1 nano on · branchingOpen · 3.1BDetails | 100.0% | 24 | 0 |
| – | Raon-1 nano on · cascading-recoveryOpen · 3.1BDetails | 87.5% | 24 | 0 |
| – | Raon-1 nano on · long-outputOpen · 3.1BDetails | 29.2% | 24 | 0 |
| – | Gemma 4 E2B (tool-result cue template) off · long-chainOpen · 5.1BDetails | 75.0% | 24 | 0 |
| – | Gemma 4 E2B (tool-result cue template) off · computed-argsOpen · 5.1BDetails | 70.8% | 24 | 0 |
| – | Gemma 4 E2B (tool-result cue template) off · multi-constraintOpen · 5.1BDetails | 100.0% | 24 | 0 |
| – | Gemma 4 E2B (tool-result cue template) off · branchingOpen · 5.1BDetails | 100.0% | 24 | 0 |
| – | Gemma 4 E2B (tool-result cue template) off · cascading-recoveryOpen · 5.1BDetails | 95.8% | 24 | 0 |
| – | Gemma 4 E2B (tool-result cue template) off · long-outputOpen · 5.1BDetails | 20.8% | 24 | 0 |
| – | Granite 4.2 3B off · long-chainOpen · 3.0BDetails | 20.8% | 24 | 0 |
| – | Granite 4.2 3B off · computed-argsOpen · 3.0BDetails | 20.8% | 24 | 0 |
| – | Granite 4.2 3B off · multi-constraintOpen · 3.0BDetails | 58.3% | 24 | 0 |
| – | Granite 4.2 3B off · branchingOpen · 3.0BDetails | 66.7% | 24 | 0 |
| – | Granite 4.2 3B off · cascading-recoveryOpen · 3.0BDetails | 62.5% | 24 | 0 |
| – | Granite 4.2 3B off · long-outputOpen · 3.0BDetails | 12.5% | 24 | 0 |
| – | Granite 4.2 3B on · long-chainOpen · 3.0BDetails | 100.0% | 24 | 0 |
| – | Granite 4.2 3B on · computed-argsOpen · 3.0BDetails | 87.5% | 24 | 0 |
| – | Granite 4.2 3B on · multi-constraintOpen · 3.0BDetails | 87.5% | 24 | 0 |
| – | Granite 4.2 3B on · branchingOpen · 3.0BDetails | 100.0% | 24 | 0 |
| – | Granite 4.2 3B on · cascading-recoveryOpen · 3.0BDetails | 75.0% | 24 | 0 |
| – | Granite 4.2 3B on · long-outputOpen · 3.0BDetails | 79.2% | 24 | 0 |
| – | Granite 4.2 8B off · long-chainOpen · 8.0BDetails | 33.3% | 24 | 0 |
| – | Granite 4.2 8B off · computed-argsOpen · 8.0BDetails | 20.8% | 24 | 0 |
| – | Granite 4.2 8B off · multi-constraintOpen · 8.0BDetails | 37.5% | 24 | 0 |
| – | Granite 4.2 8B off · branchingOpen · 8.0BDetails | 75.0% | 24 | 0 |
| – | Granite 4.2 8B off · cascading-recoveryOpen · 8.0BDetails | 79.2% | 24 | 0 |
| – | Granite 4.2 8B off · long-outputOpen · 8.0BDetails | 8.3% | 24 | 0 |
| – | Granite 4.2 8B on · long-chainOpen · 8.0BDetails | 100.0% | 24 | 0 |
| – | Granite 4.2 8B on · computed-argsOpen · 8.0BDetails | 79.2% | 24 | 0 |
| – | Granite 4.2 8B on · multi-constraintOpen · 8.0BDetails | 100.0% | 24 | 0 |
| – | Granite 4.2 8B on · branchingOpen · 8.0BDetails | 100.0% | 24 | 0 |
| – | Granite 4.2 8B on · cascading-recoveryOpen · 8.0BDetails | 95.8% | 24 | 0 |
| – | Granite 4.2 8B on · long-outputOpen · 8.0BDetails | 73.9% | 23 | 1 |
| – | Gemma 4 12B off · long-chainOpen · 12BDetails | 100.0% | 24 | 0 |
| – | Gemma 4 12B off · computed-argsOpen · 12BDetails | 83.3% | 24 | 0 |
| – | Gemma 4 12B off · multi-constraintOpen · 12BDetails | 70.8% | 24 | 0 |
| – | Gemma 4 12B off · branchingOpen · 12BDetails | 75.0% | 24 | 0 |
| – | Gemma 4 12B off · cascading-recoveryOpen · 12BDetails | 100.0% | 24 | 0 |
| – | Gemma 4 12B off · long-outputOpen · 12BDetails | 0.0% | 24 | 0 |
| – | Gemma 4 12B on · long-chainOpen · 12BDetails | 100.0% | 24 | 0 |
| – | Gemma 4 12B on · computed-argsOpen · 12BDetails | 100.0% | 24 | 0 |
| – | Gemma 4 12B on · multi-constraintOpen · 12BDetails | 100.0% | 24 | 0 |
| – | Gemma 4 12B on · branchingOpen · 12BDetails | 100.0% | 24 | 0 |
| – | Gemma 4 12B on · cascading-recoveryOpen · 12BDetails | 100.0% | 24 | 0 |
| – | Gemma 4 12B on · long-outputOpen · 12BDetails | 70.8% | 24 | 0 |
| – | LFM2.5-24B-A2B on · long-chainOpen · 24B MoE · 2.5B activeDetails | 95.8% | 24 | 0 |
| – | LFM2.5-24B-A2B on · computed-argsOpen · 24B MoE · 2.5B activeDetails | 100.0% | 24 | 0 |
| – | LFM2.5-24B-A2B on · multi-constraintOpen · 24B MoE · 2.5B activeDetails | 91.7% | 24 | 0 |
| – | LFM2.5-24B-A2B on · branchingOpen · 24B MoE · 2.5B activeDetails | 100.0% | 24 | 0 |
| – | LFM2.5-24B-A2B on · cascading-recoveryOpen · 24B MoE · 2.5B activeDetails | 87.5% | 24 | 0 |
| – | LFM2.5-24B-A2B on · long-outputOpen · 24B MoE · 2.5B activeDetails | 66.7% | 24 | 0 |
| – | Qwen3.8-27B off · long-chainOpen · 27BDetails | 75.0% | 24 | 0 |
| – | Qwen3.8-27B off · computed-argsOpen · 27BDetails | 70.8% | 24 | 0 |
| – | Qwen3.8-27B off · multi-constraintOpen · 27BDetails | 100.0% | 24 | 0 |
| – | Qwen3.8-27B off · branchingOpen · 27BDetails | 75.0% | 24 | 0 |
| – | Qwen3.8-27B off · cascading-recoveryOpen · 27BDetails | 100.0% | 24 | 0 |
| – | Qwen3.8-27B off · long-outputOpen · 27BDetails | 25.0% | 24 | 0 |
| – | Qwen3.8-27B on · long-chainOpen · 27BDetails | 87.5% | 24 | 0 |
| – | Qwen3.8-27B on · computed-argsOpen · 27BDetails | 87.5% | 24 | 0 |
| – | Qwen3.8-27B on · multi-constraintOpen · 27BDetails | 100.0% | 24 | 0 |
| – | Qwen3.8-27B on · branchingOpen · 27BDetails | 95.8% | 24 | 0 |
| – | Qwen3.8-27B on · cascading-recoveryOpen · 27BDetails | 95.8% | 24 | 0 |
| – | Qwen3.8-27B on · long-outputOpen · 27BDetails | 62.5% | 24 | 0 |
| – | Gemma 4 31B off · long-chainOpen · 31BDetails | 75.0% | 24 | 0 |
| – | Gemma 4 31B off · computed-argsOpen · 31BDetails | 62.5% | 24 | 0 |
| – | Gemma 4 31B off · multi-constraintOpen · 31BDetails | 79.2% | 24 | 0 |
| – | Gemma 4 31B off · branchingOpen · 31BDetails | 75.0% | 24 | 0 |
| – | Gemma 4 31B off · cascading-recoveryOpen · 31BDetails | 83.3% | 24 | 0 |
| – | Gemma 4 31B off · long-outputOpen · 31BDetails | 12.5% | 24 | 0 |
| – | Gemma 4 31B on · long-chainOpen · 31BDetails | 100.0% | 24 | 0 |
| – | Gemma 4 31B on · computed-argsOpen · 31BDetails | 100.0% | 24 | 0 |
| – | Gemma 4 31B on · multi-constraintOpen · 31BDetails | 75.0% | 24 | 0 |
| – | Gemma 4 31B on · branchingOpen · 31BDetails | 100.0% | 24 | 0 |
| – | Gemma 4 31B on · cascading-recoveryOpen · 31BDetails | 95.8% | 24 | 0 |
| – | Gemma 4 31B on · long-outputOpen · 31BDetails | 75.0% | 24 | 0 |
No row of this kind yet.
Task pass rates shown separately for single, selection, arguments, multi-step, restraint and recovery tasks. These are categories of the same task bank, not independent benchmark banks.
Not tiered. Serving and template failures are unscored, not model zeroes.
Failure reasons Failed tasks by reason — lower is better.
| Model | Failure reasons | Scored attempts | Unscored attempts | |
|---|---|---|---|---|
| – | LFM3-1.7B offOpen · 1.7BDetails | 39 | 252 | 0 |
| – | LFM3-1.7B onOpen · 1.7BDetails | 21 | 252 | 0 |
| – | Gemma 4 E2B onOpen · 5.1BDetails | 15 | 252 | 0 |
| – | MiniCPM5-2B offOpen · 2.5BDetails | 47 | 252 | 0 |
| – | MiniCPM5-2B onOpen · 2.5BDetails | 24 | 252 | 0 |
| – | Qwen3.5-2B offOpen · 2.0BDetails | 67 | 252 | 0 |
| – | Qwen3.5-2B onOpen · 2.0BDetails | 32 | 252 | 0 |
| – | Qwen3.5 4B offOpen · 4.7BDetails | 22 | 252 | 0 |
| – | Qwen3.5 4B onOpen · 4.7BDetails | 26 | 252 | 0 |
| – | Raon-1 nano offOpen · 3.1BDetails | 51 | 252 | 0 |
| – | Raon-1 nano onOpen · 3.1BDetails | 28 | 252 | 0 |
| – | Gemma 4 E2B (tool-result cue template) offOpen · 5.1BDetails | 27 | 252 | 0 |
| – | Granite 4.2 3B offOpen · 3.0BDetails | 52 | 252 | 0 |
| – | Granite 4.2 3B onOpen · 3.0BDetails | 18 | 252 | 0 |
| – | Granite 4.2 8B offOpen · 8.0BDetails | 41 | 252 | 0 |
| – | Granite 4.2 8B onOpen · 8.0BDetails | 17 | 252 | 0 |
| – | Gemma 4 12B offOpen · 12BDetails | 14 | 252 | 0 |
| – | Gemma 4 12B onOpen · 12BDetails | 18 | 252 | 0 |
| – | LFM2.5-24B-A2B onOpen · 24B MoE · 2.5B activeDetails | 18 | 252 | 0 |
| – | Qwen3.8-27B offOpen · 27BDetails | 8 | 252 | 0 |
| – | Qwen3.8-27B onOpen · 27BDetails | 20 | 252 | 0 |
| – | Gemma 4 31B offOpen · 31BDetails | 0 | 252 | 0 |
| – | Gemma 4 31B onOpen · 31BDetails | 3 | 252 | 0 |
| – | Gemma 4 E2B offOpen · 5.1BDetails | Not qualified | Not scored | Not scored |
No row of this kind yet.
Counts or shares of failed tasks by the recorded failure reason: missing, malformed, wrong or extra calls, invented observations, or a loop. Verified infrastructure failures are kept separate and unscored.
Not tiered. Serving and template failures are unscored, not model zeroes.
Failure reasons Failed tasks by reason — lower is better.
| Model | Failure reasons | Scored attempts | Unscored attempts | |
|---|---|---|---|---|
| – | LFM3-1.7B offOpen · 1.7BDetails | 102 | 144 | 0 |
| – | LFM3-1.7B onOpen · 1.7BDetails | 51 | 144 | 0 |
| – | Gemma 4 E2B onOpen · 5.1BDetails | 38 | 144 | 0 |
| – | MiniCPM5-2B offOpen · 2.5BDetails | 102 | 144 | 0 |
| – | MiniCPM5-2B onOpen · 2.5BDetails | 24 | 144 | 0 |
| – | Qwen3.5-2B offOpen · 2.0BDetails | 61 | 144 | 0 |
| – | Qwen3.5-2B onOpen · 2.0BDetails | 46 | 144 | 0 |
| – | Qwen3.5 4B offOpen · 4.7BDetails | 15 | 144 | 0 |
| – | Qwen3.5 4B onOpen · 4.7BDetails | 28 | 144 | 0 |
| – | Raon-1 nano offOpen · 3.1BDetails | 108 | 144 | 0 |
| – | Raon-1 nano onOpen · 3.1BDetails | 42 | 144 | 0 |
| – | Gemma 4 E2B (tool-result cue template) offOpen · 5.1BDetails | 33 | 144 | 0 |
| – | Granite 4.2 3B offOpen · 3.0BDetails | 86 | 144 | 0 |
| – | Granite 4.2 3B onOpen · 3.0BDetails | 17 | 144 | 0 |
| – | Granite 4.2 8B offOpen · 8.0BDetails | 83 | 144 | 0 |
| – | Granite 4.2 8B onOpen · 8.0BDetails | 12 | 143 | 1 |
| – | Gemma 4 12B offOpen · 12BDetails | 41 | 144 | 0 |
| – | Gemma 4 12B onOpen · 12BDetails | 7 | 144 | 0 |
| – | LFM2.5-24B-A2B onOpen · 24B MoE · 2.5B activeDetails | 14 | 144 | 0 |
| – | Qwen3.8-27B offOpen · 27BDetails | 37 | 144 | 0 |
| – | Qwen3.8-27B onOpen · 27BDetails | 17 | 144 | 0 |
| – | Gemma 4 31B offOpen · 31BDetails | 51 | 144 | 0 |
| – | Gemma 4 31B onOpen · 31BDetails | 13 | 144 | 0 |
No row of this kind yet.
Counts or shares of failed tasks by the recorded failure reason: missing, malformed, wrong or extra calls, invented observations, or a loop. Verified infrastructure failures are kept separate and unscored.
Not tiered. Serving and template failures are unscored, not model zeroes.
Renamed tools Original and renamed task pass rates — higher is better.
| Tiers within each size class: for each pair of setups, every task (42 per track, public and private) is one paired item, won by the setup that passed more of its 3 repetitions; ties drop out. Exact two-sided sign test, Holm-corrected over all pairs of the class, p < 0.05. Rows in accuracy order: the top row starts Tier 1, and a row starts the next tier when the test separates it from its tier's top row. Accuracy: both tracks (84 tasks); Renamed tools: each track (42 tasks). – : not tiered (class not verified, incomplete or not qualified). | Model | Renamed tools | Scored attempts | Unscored attempts |
|---|---|---|---|---|
| Original tools · Tiny | ||||
| 1 | Gemma 4 E2B on · originalOpen · 5.1BDetails | 93.7% | 126 | 0 |
| 1 | Qwen3.5 4B off · originalOpen · 4.7BDetails | 92.1% | 126 | 0 |
| 1 | Granite 4.2 3B on · originalOpen · 3.0BDetails | 92.1% | 126 | 0 |
| 1 | LFM3-1.7B on · originalOpen · 1.7BDetails | 91.3% | 126 | 0 |
| 1 | Gemma 4 E2B (tool-result cue template) off · originalOpen · 5.1BDetails | 90.5% | 126 | 0 |
| 1 | Qwen3.5-2B on · originalOpen · 2.0BDetails | 89.7% | 126 | 0 |
| 1 | Qwen3.5 4B on · originalOpen · 4.7BDetails | 88.9% | 126 | 0 |
| 1 | MiniCPM5-2B on · originalOpen · 2.5BDetails | 88.1% | 126 | 0 |
| 1 | Raon-1 nano on · originalOpen · 3.1BDetails | 88.1% | 126 | 0 |
| 1 | LFM3-1.7B off · originalOpen · 1.7BDetails | 86.5% | 126 | 0 |
| 1 | Granite 4.2 3B off · originalOpen · 3.0BDetails | 78.6% | 126 | 0 |
| 1 | MiniCPM5-2B off · originalOpen · 2.5BDetails | 77.8% | 126 | 0 |
| 1 | Raon-1 nano off · originalOpen · 3.1BDetails | 77.0% | 126 | 0 |
| 2 | Qwen3.5-2B off · originalOpen · 2.0BDetails | 73.8% | 126 | 0 |
| – | Gemma 4 E2B offOpen · 5.1BDetails | Not qualified | Not scored | Not scored |
| Original tools · Small | ||||
| 1 | Gemma 4 12B off · originalOpen · 12BDetails | 94.4% | 126 | 0 |
| 1 | Granite 4.2 8B on · originalOpen · 8.0BDetails | 92.9% | 126 | 0 |
| 1 | Gemma 4 12B on · originalOpen · 12BDetails | 92.9% | 126 | 0 |
| 1 | Granite 4.2 8B off · originalOpen · 8.0BDetails | 84.9% | 126 | 0 |
| Original tools · Medium | ||||
| 1 | Gemma 4 31B off · originalOpen · 31BDetails | 100.0% | 126 | 0 |
| 1 | Gemma 4 31B on · originalOpen · 31BDetails | 99.2% | 126 | 0 |
| 1 | Qwen3.8-27B off · originalOpen · 27BDetails | 96.0% | 126 | 0 |
| 1 | Qwen3.8-27B on · originalOpen · 27BDetails | 92.9% | 126 | 0 |
| 1 | LFM2.5-24B-A2B on · originalOpen · 24B MoE · 2.5B activeDetails | 92.1% | 126 | 0 |
| Renamed tools · Tiny | ||||
| 1 | Gemma 4 E2B on · renamedOpen · 5.1BDetails | 94.4% | 126 | 0 |
| 1 | Granite 4.2 3B on · renamedOpen · 3.0BDetails | 93.7% | 126 | 0 |
| 1 | MiniCPM5-2B on · renamedOpen · 2.5BDetails | 92.9% | 126 | 0 |
| 1 | LFM3-1.7B on · renamedOpen · 1.7BDetails | 92.1% | 126 | 0 |
| 1 | Qwen3.5 4B off · renamedOpen · 4.7BDetails | 90.5% | 126 | 0 |
| 1 | Qwen3.5 4B on · renamedOpen · 4.7BDetails | 90.5% | 126 | 0 |
| 1 | Raon-1 nano on · renamedOpen · 3.1BDetails | 89.7% | 126 | 0 |
| 1 | Gemma 4 E2B (tool-result cue template) off · renamedOpen · 5.1BDetails | 88.1% | 126 | 0 |
| 1 | MiniCPM5-2B off · renamedOpen · 2.5BDetails | 84.9% | 126 | 0 |
| 1 | Qwen3.5-2B on · renamedOpen · 2.0BDetails | 84.9% | 126 | 0 |
| 1 | LFM3-1.7B off · renamedOpen · 1.7BDetails | 82.5% | 126 | 0 |
| 1 | Raon-1 nano off · renamedOpen · 3.1BDetails | 82.5% | 126 | 0 |
| 1 | Granite 4.2 3B off · renamedOpen · 3.0BDetails | 80.2% | 126 | 0 |
| 2 | Qwen3.5-2B off · renamedOpen · 2.0BDetails | 73.0% | 126 | 0 |
| – | Gemma 4 E2B offOpen · 5.1BDetails | Not qualified | Not scored | Not scored |
| Renamed tools · Small | ||||
| 1 | Gemma 4 12B off · renamedOpen · 12BDetails | 94.4% | 126 | 0 |
| 1 | Granite 4.2 8B on · renamedOpen · 8.0BDetails | 93.7% | 126 | 0 |
| 1 | Gemma 4 12B on · renamedOpen · 12BDetails | 92.9% | 126 | 0 |
| 1 | Granite 4.2 8B off · renamedOpen · 8.0BDetails | 82.5% | 126 | 0 |
| Renamed tools · Medium | ||||
| 1 | Gemma 4 31B off · renamedOpen · 31BDetails | 100.0% | 126 | 0 |
| 1 | Gemma 4 31B on · renamedOpen · 31BDetails | 98.4% | 126 | 0 |
| 1 | Qwen3.8-27B off · renamedOpen · 27BDetails | 97.6% | 126 | 0 |
| 1 | LFM2.5-24B-A2B on · renamedOpen · 24B MoE · 2.5B activeDetails | 93.7% | 126 | 0 |
| 1 | Qwen3.8-27B on · renamedOpen · 27BDetails | 91.3% | 126 | 0 |
No row of this kind yet.
Paired original-tool and renamed-tool conditions use the same task meaning and mock state. Compare their pass rates without treating the paired versions as independent tasks.
Tiers within each size class: for each pair of setups, every task (42 per track, public and private) is one paired item, won by the setup that passed more of its 3 repetitions; ties drop out. Exact two-sided sign test, Holm-corrected over all pairs of the class, p < 0.05. Rows in accuracy order: the top row starts Tier 1, and a row starts the next tier when the test separates it from its tier's top row. Accuracy: both tracks (84 tasks); Renamed tools: each track (42 tasks). – : not tiered (class not verified, incomplete or not qualified). Serving and template failures are unscored, not model zeroes.
Renamed tools Original and renamed task pass rates — higher is better.
| Tiers within each size class: for each pair of setups, every task (24 per track, public and private) is one paired item, won by the setup that passed more of its 3 repetitions; ties drop out. Exact two-sided sign test, Holm-corrected over all pairs of the class, p < 0.05. Rows in accuracy order: the top row starts Tier 1, and a row starts the next tier when the test separates it from its tier's top row. Accuracy: both tracks (48 tasks); Renamed tools: each track (24 tasks). – : not tiered (class not verified, incomplete or not qualified). | Model | Renamed tools | Scored attempts | Unscored attempts |
|---|---|---|---|---|
| Original tools · Tiny | ||||
| 1 | Qwen3.5 4B off · originalOpen · 4.7BDetails | 94.4% | 72 | 0 |
| 1 | Granite 4.2 3B on · originalOpen · 3.0BDetails | 84.7% | 72 | 0 |
| 1 | MiniCPM5-2B on · originalOpen · 2.5BDetails | 83.3% | 72 | 0 |
| 1 | Qwen3.5 4B on · originalOpen · 4.7BDetails | 77.8% | 72 | 0 |
| 1 | Gemma 4 E2B (tool-result cue template) off · originalOpen · 5.1BDetails | 76.4% | 72 | 0 |
| 1 | Gemma 4 E2B on · originalOpen · 5.1BDetails | 70.8% | 72 | 0 |
| 1 | Raon-1 nano on · originalOpen · 3.1BDetails | 70.8% | 72 | 0 |
| 2 | Qwen3.5-2B on · originalOpen · 2.0BDetails | 66.7% | 72 | 0 |
| 2 | LFM3-1.7B on · originalOpen · 1.7BDetails | 65.3% | 72 | 0 |
| 2 | Qwen3.5-2B off · originalOpen · 2.0BDetails | 58.3% | 72 | 0 |
| 2 | Granite 4.2 3B off · originalOpen · 3.0BDetails | 38.9% | 72 | 0 |
| 2 | MiniCPM5-2B off · originalOpen · 2.5BDetails | 36.1% | 72 | 0 |
| 2 | LFM3-1.7B off · originalOpen · 1.7BDetails | 33.3% | 72 | 0 |
| 2 | Raon-1 nano off · originalOpen · 3.1BDetails | 26.4% | 72 | 0 |
| Original tools · Small | ||||
| – | Granite 4.2 8B on · originalOpen · 8.0BDetails | 95.8% | 71 | 1 |
| 1 | Gemma 4 12B on · originalOpen · 12BDetails | 94.4% | 72 | 0 |
| 2 | Gemma 4 12B off · originalOpen · 12BDetails | 70.8% | 72 | 0 |
| 3 | Granite 4.2 8B off · originalOpen · 8.0BDetails | 41.7% | 72 | 0 |
| Original tools · Medium | ||||
| 1 | LFM2.5-24B-A2B on · originalOpen · 24B MoE · 2.5B activeDetails | 91.7% | 72 | 0 |
| 1 | Gemma 4 31B on · originalOpen · 31BDetails | 90.3% | 72 | 0 |
| 1 | Qwen3.8-27B on · originalOpen · 27BDetails | 88.9% | 72 | 0 |
| 1 | Qwen3.8-27B off · originalOpen · 27BDetails | 75.0% | 72 | 0 |
| 1 | Gemma 4 31B off · originalOpen · 31BDetails | 62.5% | 72 | 0 |
| Renamed tools · Tiny | ||||
| 1 | Granite 4.2 3B on · renamedOpen · 3.0BDetails | 91.7% | 72 | 0 |
| 1 | Qwen3.5 4B off · renamedOpen · 4.7BDetails | 84.7% | 72 | 0 |
| 1 | MiniCPM5-2B on · renamedOpen · 2.5BDetails | 83.3% | 72 | 0 |
| 1 | Qwen3.5 4B on · renamedOpen · 4.7BDetails | 83.3% | 72 | 0 |
| 1 | Gemma 4 E2B (tool-result cue template) off · renamedOpen · 5.1BDetails | 77.8% | 72 | 0 |
| 1 | Gemma 4 E2B on · renamedOpen · 5.1BDetails | 76.4% | 72 | 0 |
| 1 | Raon-1 nano on · renamedOpen · 3.1BDetails | 70.8% | 72 | 0 |
| 1 | Qwen3.5-2B on · renamedOpen · 2.0BDetails | 69.4% | 72 | 0 |
| 1 | LFM3-1.7B on · renamedOpen · 1.7BDetails | 63.9% | 72 | 0 |
| 2 | Qwen3.5-2B off · renamedOpen · 2.0BDetails | 56.9% | 72 | 0 |
| 2 | Granite 4.2 3B off · renamedOpen · 3.0BDetails | 41.7% | 72 | 0 |
| 2 | LFM3-1.7B off · renamedOpen · 1.7BDetails | 25.0% | 72 | 0 |
| 3 | Raon-1 nano off · renamedOpen · 3.1BDetails | 23.6% | 72 | 0 |
| 3 | MiniCPM5-2B off · renamedOpen · 2.5BDetails | 22.2% | 72 | 0 |
| Renamed tools · Small | ||||
| 1 | Gemma 4 12B on · renamedOpen · 12BDetails | 95.8% | 72 | 0 |
| – | Granite 4.2 8B on · renamedOpen · 8.0BDetails | 87.5% | 72 | 0 |
| 2 | Gemma 4 12B off · renamedOpen · 12BDetails | 72.2% | 72 | 0 |
| 3 | Granite 4.2 8B off · renamedOpen · 8.0BDetails | 43.1% | 72 | 0 |
| Renamed tools · Medium | ||||
| 1 | Gemma 4 31B on · renamedOpen · 31BDetails | 91.7% | 72 | 0 |
| 1 | LFM2.5-24B-A2B on · renamedOpen · 24B MoE · 2.5B activeDetails | 88.9% | 72 | 0 |
| 1 | Qwen3.8-27B on · renamedOpen · 27BDetails | 87.5% | 72 | 0 |
| 1 | Qwen3.8-27B off · renamedOpen · 27BDetails | 73.6% | 72 | 0 |
| 1 | Gemma 4 31B off · renamedOpen · 31BDetails | 66.7% | 72 | 0 |
No row of this kind yet.
Paired original-tool and renamed-tool conditions use the same task meaning and mock state. Compare their pass rates without treating the paired versions as independent tasks.
Tiers within each size class: for each pair of setups, every task (24 per track, public and private) is one paired item, won by the setup that passed more of its 3 repetitions; ties drop out. Exact two-sided sign test, Holm-corrected over all pairs of the class, p < 0.05. Rows in accuracy order: the top row starts Tier 1, and a row starts the next tier when the test separates it from its tier's top row. Accuracy: both tracks (48 tasks); Renamed tools: each track (24 tasks). – : not tiered (class not verified, incomplete or not qualified). Serving and template failures are unscored, not model zeroes.
Latency Seconds per task — lower is better.
| Model | Latency | Scored attempts | Unscored attempts | |
|---|---|---|---|---|
| – | LFM3-1.7B offOpen · 1.7BDetails | 0.530 s/task | 252 | 0 |
| – | LFM3-1.7B onOpen · 1.7BDetails | 1.271 s/task | 252 | 0 |
| – | Gemma 4 E2B onOpen · 5.1BDetails | 4.309 s/task | 252 | 0 |
| – | MiniCPM5-2B offOpen · 2.5BDetails | 0.746 s/task | 252 | 0 |
| – | MiniCPM5-2B onOpen · 2.5BDetails | 2.655 s/task | 252 | 0 |
| – | Qwen3.5-2B offOpen · 2.0BDetails | 1.343 s/task | 252 | 0 |
| – | Qwen3.5-2B onOpen · 2.0BDetails | 2.917 s/task | 252 | 0 |
| – | Qwen3.5 4B offOpen · 4.7BDetails | 1.878 s/task | 252 | 0 |
| – | Qwen3.5 4B onOpen · 4.7BDetails | 4.507 s/task | 252 | 0 |
| – | Raon-1 nano offOpen · 3.1BDetails | 2.320 s/task | 252 | 0 |
| – | Raon-1 nano onOpen · 3.1BDetails | 4.257 s/task | 252 | 0 |
| – | Gemma 4 E2B (tool-result cue template) offOpen · 5.1BDetails | 1.711 s/task | 252 | 0 |
| – | Granite 4.2 3B offOpen · 3.0BDetails | 1.514 s/task | 252 | 0 |
| – | Granite 4.2 3B onOpen · 3.0BDetails | 8.037 s/task | 252 | 0 |
| – | Granite 4.2 8B offOpen · 8.0BDetails | 3.324 s/task | 252 | 0 |
| – | Granite 4.2 8B onOpen · 8.0BDetails | 19.213 s/task | 252 | 0 |
| – | Gemma 4 12B offOpen · 12BDetails | 2.284 s/task | 252 | 0 |
| – | Gemma 4 12B onOpen · 12BDetails | 7.256 s/task | 252 | 0 |
| – | LFM2.5-24B-A2B onOpen · 24B MoE · 2.5B activeDetails | 8.233 s/task | 252 | 0 |
| – | Qwen3.8-27B offOpen · 27BDetails | 8.800 s/task | 252 | 0 |
| – | Qwen3.8-27B onOpen · 27BDetails | 18.498 s/task | 252 | 0 |
| – | Gemma 4 31B offOpen · 31BDetails | 5.150 s/task | 252 | 0 |
| – | Gemma 4 31B onOpen · 31BDetails | 15.245 s/task | 252 | 0 |
| – | Gemma 4 E2B offOpen · 5.1BDetails | Not qualified | Not scored | Not scored |
No row of this kind yet.
Observed end-to-end seconds per evaluated task with engine, precision, server and concurrency recorded. Compare like-for-like setups; synthetic fixture timings are not model latency.
Not tiered. Serving and template failures are unscored, not model zeroes.
Latency Seconds per task — lower is better.
| Model | Latency | Scored attempts | Unscored attempts | |
|---|---|---|---|---|
| – | LFM3-1.7B offOpen · 1.7BDetails | 2.145 s/task | 144 | 0 |
| – | LFM3-1.7B onOpen · 1.7BDetails | 4.835 s/task | 144 | 0 |
| – | Gemma 4 E2B onOpen · 5.1BDetails | 10.974 s/task | 144 | 0 |
| – | MiniCPM5-2B offOpen · 2.5BDetails | 2.353 s/task | 144 | 0 |
| – | MiniCPM5-2B onOpen · 2.5BDetails | 8.338 s/task | 144 | 0 |
| – | Qwen3.5-2B offOpen · 2.0BDetails | 5.866 s/task | 144 | 0 |
| – | Qwen3.5-2B onOpen · 2.0BDetails | 11.172 s/task | 144 | 0 |
| – | Qwen3.5 4B offOpen · 4.7BDetails | 8.048 s/task | 144 | 0 |
| – | Qwen3.5 4B onOpen · 4.7BDetails | 15.285 s/task | 144 | 0 |
| – | Raon-1 nano offOpen · 3.1BDetails | 8.829 s/task | 144 | 0 |
| – | Raon-1 nano onOpen · 3.1BDetails | 22.222 s/task | 144 | 0 |
| – | Gemma 4 E2B (tool-result cue template) offOpen · 5.1BDetails | 6.976 s/task | 144 | 0 |
| – | Granite 4.2 3B offOpen · 3.0BDetails | 12.157 s/task | 144 | 0 |
| – | Granite 4.2 3B onOpen · 3.0BDetails | 61.376 s/task | 144 | 0 |
| – | Granite 4.2 8B offOpen · 8.0BDetails | 23.233 s/task | 144 | 0 |
| – | Granite 4.2 8B onOpen · 8.0BDetails | 94.753 s/task | 143 | 1 |
| – | Gemma 4 12B offOpen · 12BDetails | 8.252 s/task | 144 | 0 |
| – | Gemma 4 12B onOpen · 12BDetails | 23.041 s/task | 144 | 0 |
| – | LFM2.5-24B-A2B onOpen · 24B MoE · 2.5B activeDetails | 21.228 s/task | 144 | 0 |
| – | Qwen3.8-27B offOpen · 27BDetails | 25.824 s/task | 144 | 0 |
| – | Qwen3.8-27B onOpen · 27BDetails | 53.048 s/task | 144 | 0 |
| – | Gemma 4 31B offOpen · 31BDetails | 29.182 s/task | 144 | 0 |
| – | Gemma 4 31B onOpen · 31BDetails | 63.261 s/task | 144 | 0 |
No row of this kind yet.
Observed end-to-end seconds per evaluated task with engine, precision, server and concurrency recorded. Compare like-for-like setups; synthetic fixture timings are not model latency.
Not tiered. Serving and template failures are unscored, not model zeroes.
Cost USD per evaluated task — lower is better.
| Model | Cost | GPU time · cost per run | Requests in flight | |
|---|---|---|---|---|
| – | LFM3-1.7B offOpen · 1.7BDetails | $0.000026/task | 10 s · $0.0066/run | 16 |
| – | LFM3-1.7B onOpen · 1.7BDetails | $0.000064/task | 23 s · $0.0162/run | 16 |
| – | Gemma 4 E2B onOpen · 5.1BDetails | $0.000195/task | 71 s · $0.0492/run | 16 |
| – | MiniCPM5-2B offOpen · 2.5BDetails | $0.000034/task | 12 s · $0.0085/run | 16 |
| – | MiniCPM5-2B onOpen · 2.5BDetails | $0.000158/task | 57 s · $0.0397/run | 16 |
| – | Qwen3.5-2B offOpen · 2.0BDetails | $0.000071/task | 26 s · $0.0180/run | 16 |
| – | Qwen3.5-2B onOpen · 2.0BDetails | $0.000163/task | 59 s · $0.0411/run | 16 |
| – | Qwen3.5 4B offOpen · 4.7BDetails | $0.000086/task | 31 s · $0.0216/run | 16 |
| – | Qwen3.5 4B onOpen · 4.7BDetails | $0.000214/task | 78 s · $0.0539/run | 16 |
| – | Raon-1 nano offOpen · 3.1BDetails | $0.000146/task | 53 s · $0.0369/run | 16 |
| – | Raon-1 nano onOpen · 3.1BDetails | $0.000238/task | 87 s · $0.0600/run | 16 |
| – | Gemma 4 E2B (tool-result cue template) offOpen · 5.1BDetails | $0.000086/task | 31 s · $0.0216/run | 16 |
| – | Granite 4.2 3B offOpen · 3.0BDetails | $0.000099/task | 36 s · $0.0249/run | 16 |
| – | Granite 4.2 3B onOpen · 3.0BDetails | $0.000432/task | 157 s · $0.1089/run | 16 |
| – | Granite 4.2 8B offOpen · 8.0BDetails | $0.000224/task | 82 s · $0.0564/run | 16 |
| – | Granite 4.2 8B onOpen · 8.0BDetails | $0.000910/task | 332 s · $0.2294/run | 16 |
| – | Gemma 4 12B offOpen · 12BDetails | $0.000105/task | 38 s · $0.0266/run | 16 |
| – | Gemma 4 12B onOpen · 12BDetails | $0.000332/task | 121 s · $0.0836/run | 16 |
| – | LFM2.5-24B-A2B onOpen · 24B MoE · 2.5B activeDetails | $0.000370/task | 135 s · $0.0932/run | 16 |
| – | Qwen3.8-27B offOpen · 27BDetails | $0.000394/task | 144 s · $0.0993/run | 16 |
| – | Qwen3.8-27B onOpen · 27BDetails | $0.000864/task | 315 s · $0.2177/run | 16 |
| – | Gemma 4 31B offOpen · 31BDetails | $0.000236/task | 86 s · $0.0594/run | 16 |
| – | Gemma 4 31B onOpen · 31BDetails | $0.000691/task | 252 s · $0.1740/run | 16 |
| – | Gemma 4 E2B offOpen · 5.1BDetails | Not qualified | Not scored | Not scored |
No row of this kind yet.
Documented charge per evaluated task from actual usage and applicable prices or verified local accounting. Unknown cost stays unreported, never zero. Different-engine cost and speed are not presented as like-for-like.
Not tiered. Serving and template failures are unscored, not model zeroes.
Cost USD per evaluated task — lower is better.
| Model | Cost | GPU time · cost per run | Requests in flight | |
|---|---|---|---|---|
| – | LFM3-1.7B offOpen · 1.7BDetails | $0.000108/task | 22 s · $0.0156/run | 16 |
| – | LFM3-1.7B onOpen · 1.7BDetails | $0.000246/task | 51 s · $0.0355/run | 16 |
| – | Gemma 4 E2B onOpen · 5.1BDetails | $0.000510/task | 106 s · $0.0734/run | 16 |
| – | MiniCPM5-2B offOpen · 2.5BDetails | $0.000114/task | 24 s · $0.0165/run | 16 |
| – | MiniCPM5-2B onOpen · 2.5BDetails | $0.000442/task | 92 s · $0.0637/run | 16 |
| – | Qwen3.5-2B offOpen · 2.0BDetails | $0.000316/task | 66 s · $0.0455/run | 16 |
| – | Qwen3.5-2B onOpen · 2.0BDetails | $0.000537/task | 112 s · $0.0774/run | 16 |
| – | Qwen3.5 4B offOpen · 4.7BDetails | $0.000367/task | 76 s · $0.0528/run | 16 |
| – | Qwen3.5 4B onOpen · 4.7BDetails | $0.000719/task | 150 s · $0.1035/run | 16 |
| – | Raon-1 nano offOpen · 3.1BDetails | $0.000451/task | 94 s · $0.0649/run | 16 |
| – | Raon-1 nano onOpen · 3.1BDetails | $0.001319/task | 275 s · $0.1899/run | 16 |
| – | Gemma 4 E2B (tool-result cue template) offOpen · 5.1BDetails | $0.000326/task | 68 s · $0.0469/run | 16 |
| – | Granite 4.2 3B offOpen · 3.0BDetails | $0.001283/task | 267 s · $0.1847/run | 16 |
| – | Granite 4.2 3B onOpen · 3.0BDetails | $0.003479/task | 724 s · $0.5010/run | 16 |
| – | Granite 4.2 8B offOpen · 8.0BDetails | $0.001391/task | 290 s · $0.2003/run | 16 |
| – | Granite 4.2 8B onOpen · 8.0BDetails | $0.005874/task | 1214 s · $0.8400/run | 16 |
| – | Gemma 4 12B offOpen · 12BDetails | $0.000398/task | 83 s · $0.0573/run | 16 |
| – | Gemma 4 12B onOpen · 12BDetails | $0.001100/task | 229 s · $0.1585/run | 16 |
| – | LFM2.5-24B-A2B onOpen · 24B MoE · 2.5B activeDetails | $0.000965/task | 201 s · $0.1390/run | 16 |
| – | Qwen3.8-27B offOpen · 27BDetails | $0.001161/task | 242 s · $0.1672/run | 16 |
| – | Qwen3.8-27B onOpen · 27BDetails | $0.002477/task | 516 s · $0.3566/run | 16 |
| – | Gemma 4 31B offOpen · 31BDetails | $0.001349/task | 281 s · $0.1943/run | 16 |
| – | Gemma 4 31B onOpen · 31BDetails | $0.002921/task | 608 s · $0.4207/run | 16 |
No row of this kind yet.
Documented charge per evaluated task from actual usage and applicable prices or verified local accounting. Unknown cost stays unreported, never zero. Different-engine cost and speed are not presented as like-for-like.
Not tiered. Serving and template failures are unscored, not model zeroes.
Details of each setup
Provider: local vLLM served by @logic on bariton (#9534) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4
237/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"gave_up_or_loop": 5, "wrong_args": 7, "wrong_tool": 3}
Tokens: 1034078
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 71 s · $0.0492 for the run, $0.000195 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
234/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"no_call": 1, "wrong_args": 6, "wrong_tool": 11}
Tokens: 1560191
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 157 s · $0.1089 for the run, $0.000432 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
231/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 10, "gave_up_or_loop": 5, "wrong_args": 5, "wrong_tool": 1}
Tokens: 888087
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 23 s · $0.0162 for the run, $0.000064 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
230/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 1, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 11, "wrong_tool": 7}
Tokens: 1026831
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
228/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 4, "wrong_args": 18, "wrong_tool": 2}
Tokens: 1133799
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 57 s · $0.0397 for the run, $0.000158 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
226/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 5, "wrong_args": 15, "wrong_tool": 6}
Tokens: 1138207
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 78 s · $0.0539 for the run, $0.000214 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the run, identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4
Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.
225/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 4, "gave_up_or_loop": 1, "wrong_args": 22}
Tokens: 797158
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
224/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 15, "wrong_tool": 4}
Tokens: 1023064
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 87 s · $0.0600 for the run, $0.000238 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
220/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 7, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 13, "wrong_tool": 9}
Tokens: 1267484
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 59 s · $0.0411 for the run, $0.000163 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
213/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"bad_format": 5, "extra_call": 7, "gave_up_or_loop": 2, "wrong_args": 19, "wrong_tool": 6}
Tokens: 773212
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 10 s · $0.0066 for the run, $0.000026 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
205/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"bad_format": 2, "extra_call": 7, "gave_up_or_loop": 14, "wrong_args": 21, "wrong_tool": 3}
Tokens: 894011
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 12 s · $0.0085 for the run, $0.000034 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
201/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"bad_format": 3, "extra_call": 22, "wrong_args": 20, "wrong_tool": 6}
Tokens: 1179082
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 53 s · $0.0369 for the run, $0.000146 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
200/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"gave_up_or_loop": 5, "made_up_result": 1, "wrong_args": 39, "wrong_tool": 7}
Tokens: 1058883
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 36 s · $0.0249 for the run, $0.000099 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
185/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 14, "gave_up_or_loop": 11, "made_up_result": 2, "wrong_args": 33, "wrong_tool": 7}
Tokens: 1173418
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 26 s · $0.0180 for the run, $0.000071 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Not qualified: Qualification failed on public task single-01: after a correct get_price call and its mock result, the model's next token was <|tool_response> (an end-of-generation id), so the reply was empty. With thinking off, neither the maker's chat template nor vLLM's example template adds a continuation cue after a tool result. One observation; cause not proven.
No score: the setup did not complete the qualification tool round trip, so its full run was not admitted.
Qualification evidence sha256: 11b2effa6e9f1f57d2fd1a76001d65b3f2a5f497ee815666bc2a8b6e8d41a18a
Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
238/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"gave_up_or_loop": 2, "wrong_tool": 12}
Tokens: 728093
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 38 s · $0.0266 for the run, $0.000105 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
235/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"gave_up_or_loop": 2, "wrong_args": 8, "wrong_tool": 7}
Tokens: 1580185
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 332 s · $0.2294 for the run, $0.000910 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
234/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 2, "wrong_args": 4, "wrong_tool": 12}
Tokens: 870626
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 121 s · $0.0836 for the run, $0.000332 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
211/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 3, "gave_up_or_loop": 3, "wrong_args": 21, "wrong_tool": 14}
Tokens: 1026531
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 82 s · $0.0564 for the run, $0.000224 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.
252/252 scored passed; 0 unscored; 252 attempted
Failure tags: {}
Tokens: 691572
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 86 s · $0.0594 for the run, $0.000236 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.
249/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"bad_format": 2, "wrong_tool": 1}
Tokens: 783015
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 252 s · $0.1740 for the run, $0.000691 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
244/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 5, "wrong_args": 1, "wrong_tool": 2}
Tokens: 948341
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 144 s · $0.0993 for the run, $0.000394 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183
Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2
234/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 2, "wrong_tool": 7}
Tokens: 998331
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 135 s · $0.0932 for the run, $0.000370 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
232/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"bad_format": 5, "extra_call": 10, "wrong_args": 1, "wrong_tool": 4}
Tokens: 1098479
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 315 s · $0.2177 for the run, $0.000864 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
129/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 1, "wrong_args": 14}
Tokens: 2194487
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 76 s · $0.0528 for the run, $0.000367 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
127/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 4, "no_call": 1, "wrong_args": 12}
Tokens: 4304813
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 724 s · $0.5010 for the run, $0.003479 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
120/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 6, "wrong_args": 18}
Tokens: 2288473
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 92 s · $0.0637 for the run, $0.000442 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
116/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 3, "wrong_args": 25}
Tokens: 2457570
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 150 s · $0.1035 for the run, $0.000719 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the standard run; #10423 GPU 1 for the hard set; identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4
Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.
111/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 1, "made_up_result": 5, "wrong_args": 27}
Tokens: 1836873
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 68 s · $0.0469 for the run, $0.000326 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#9534 standard set; #10411 hard set, identical argv) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4
106/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 11, "wrong_args": 27}
Tokens: 2048615
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 106 s · $0.0734 for the run, $0.000510 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
102/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 12, "wrong_args": 30}
Tokens: 2137047
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 275 s · $0.1899 for the run, $0.001319 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
98/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 5, "wrong_args": 40, "wrong_tool": 1}
Tokens: 2603860
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 112 s · $0.0774 for the run, $0.000537 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
93/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 25, "extra_call": 5, "wrong_args": 21}
Tokens: 1790981
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 51 s · $0.0355 for the run, $0.000246 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
83/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 3, "gave_up_or_loop": 8, "made_up_result": 1, "wrong_args": 47, "wrong_tool": 2}
Tokens: 2473838
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 66 s · $0.0455 for the run, $0.000316 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
58/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 12, "gave_up_or_loop": 5, "wrong_args": 68, "wrong_tool": 1}
Tokens: 2601502
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 267 s · $0.1847 for the run, $0.001283 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
42/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 18, "extra_call": 6, "gave_up_or_loop": 4, "wrong_args": 73, "wrong_tool": 1}
Tokens: 1340110
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 22 s · $0.0156 for the run, $0.000108 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
42/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 18, "extra_call": 7, "gave_up_or_loop": 6, "wrong_args": 71}
Tokens: 1586690
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 24 s · $0.0165 for the run, $0.000114 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
36/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 1, "extra_call": 15, "gave_up_or_loop": 6, "wrong_args": 81, "wrong_tool": 5}
Tokens: 2355910
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 94 s · $0.0649 for the run, $0.000451 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
137/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 1, "wrong_args": 6}
Tokens: 2017522
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 229 s · $0.1585 for the run, $0.001100 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
Run note: Not tiered: one of its 144 task runs is unscored. On a long-output task the server refused the request because the prompt plus the fixed 32,768-token answer budget exceeded its 69,632-token context (HTTP 400). Serving and configuration errors are not counted as model failures; the score is over the scored runs.
131/143 scored passed; 1 unscored; 144 attempted
Failure tags: {"wrong_args": 12}
Tokens: Not reported
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 1214 s · $0.8400 for the run, $0.005874 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
103/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 7, "wrong_args": 34}
Tokens: 1798075
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 83 s · $0.0573 for the run, $0.000398 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
61/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 14, "wrong_args": 68, "wrong_tool": 1}
Tokens: 2353976
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 290 s · $0.2003 for the run, $0.001391 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
131/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 13}
Tokens: 1830591
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 608 s · $0.4207 for the run, $0.002921 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10561 hard set on GPU 1, identical argv) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183
Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2
130/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 8, "extra_call": 1, "wrong_args": 5}
Tokens: 2110323
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 201 s · $0.1390 for the run, $0.000965 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
127/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 9, "extra_call": 3, "wrong_args": 5}
Tokens: 2204967
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 516 s · $0.3566 for the run, $0.002477 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
107/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 1, "extra_call": 6, "wrong_args": 30}
Tokens: 2144354
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 242 s · $0.1672 for the run, $0.001161 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
93/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 20, "extra_call": 6, "wrong_args": 25}
Tokens: 1565256
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 281 s · $0.1943 for the run, $0.001349 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"bad_format": 5, "extra_call": 7, "gave_up_or_loop": 2, "wrong_args": 19, "wrong_tool": 6}
Tokens: 773212
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 10 s · $0.0066 for the run, $0.000026 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
27/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"bad_format": 5, "extra_call": 7, "gave_up_or_loop": 2, "wrong_args": 19, "wrong_tool": 6}
Tokens: 773212
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 10 s · $0.0066 for the run, $0.000026 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"bad_format": 5, "extra_call": 7, "gave_up_or_loop": 2, "wrong_args": 19, "wrong_tool": 6}
Tokens: 773212
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 10 s · $0.0066 for the run, $0.000026 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
25/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"bad_format": 5, "extra_call": 7, "gave_up_or_loop": 2, "wrong_args": 19, "wrong_tool": 6}
Tokens: 773212
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 10 s · $0.0066 for the run, $0.000026 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"bad_format": 5, "extra_call": 7, "gave_up_or_loop": 2, "wrong_args": 19, "wrong_tool": 6}
Tokens: 773212
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 10 s · $0.0066 for the run, $0.000026 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
35/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"bad_format": 5, "extra_call": 7, "gave_up_or_loop": 2, "wrong_args": 19, "wrong_tool": 6}
Tokens: 773212
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 10 s · $0.0066 for the run, $0.000026 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 10, "gave_up_or_loop": 5, "wrong_args": 5, "wrong_tool": 1}
Tokens: 888087
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 23 s · $0.0162 for the run, $0.000064 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
41/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 10, "gave_up_or_loop": 5, "wrong_args": 5, "wrong_tool": 1}
Tokens: 888087
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 23 s · $0.0162 for the run, $0.000064 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 10, "gave_up_or_loop": 5, "wrong_args": 5, "wrong_tool": 1}
Tokens: 888087
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 23 s · $0.0162 for the run, $0.000064 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
32/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 10, "gave_up_or_loop": 5, "wrong_args": 5, "wrong_tool": 1}
Tokens: 888087
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 23 s · $0.0162 for the run, $0.000064 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
38/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 10, "gave_up_or_loop": 5, "wrong_args": 5, "wrong_tool": 1}
Tokens: 888087
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 23 s · $0.0162 for the run, $0.000064 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
36/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 10, "gave_up_or_loop": 5, "wrong_args": 5, "wrong_tool": 1}
Tokens: 888087
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 23 s · $0.0162 for the run, $0.000064 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9534) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"gave_up_or_loop": 5, "wrong_args": 7, "wrong_tool": 3}
Tokens: 1034078
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 71 s · $0.0492 for the run, $0.000195 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#9534) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4
36/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"gave_up_or_loop": 5, "wrong_args": 7, "wrong_tool": 3}
Tokens: 1034078
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 71 s · $0.0492 for the run, $0.000195 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#9534) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"gave_up_or_loop": 5, "wrong_args": 7, "wrong_tool": 3}
Tokens: 1034078
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 71 s · $0.0492 for the run, $0.000195 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#9534) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4
39/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"gave_up_or_loop": 5, "wrong_args": 7, "wrong_tool": 3}
Tokens: 1034078
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 71 s · $0.0492 for the run, $0.000195 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#9534) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"gave_up_or_loop": 5, "wrong_args": 7, "wrong_tool": 3}
Tokens: 1034078
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 71 s · $0.0492 for the run, $0.000195 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#9534) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4
36/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"gave_up_or_loop": 5, "wrong_args": 7, "wrong_tool": 3}
Tokens: 1034078
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 71 s · $0.0492 for the run, $0.000195 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"bad_format": 2, "extra_call": 7, "gave_up_or_loop": 14, "wrong_args": 21, "wrong_tool": 3}
Tokens: 894011
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 12 s · $0.0085 for the run, $0.000034 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
30/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"bad_format": 2, "extra_call": 7, "gave_up_or_loop": 14, "wrong_args": 21, "wrong_tool": 3}
Tokens: 894011
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 12 s · $0.0085 for the run, $0.000034 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"bad_format": 2, "extra_call": 7, "gave_up_or_loop": 14, "wrong_args": 21, "wrong_tool": 3}
Tokens: 894011
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 12 s · $0.0085 for the run, $0.000034 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
16/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"bad_format": 2, "extra_call": 7, "gave_up_or_loop": 14, "wrong_args": 21, "wrong_tool": 3}
Tokens: 894011
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 12 s · $0.0085 for the run, $0.000034 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
41/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"bad_format": 2, "extra_call": 7, "gave_up_or_loop": 14, "wrong_args": 21, "wrong_tool": 3}
Tokens: 894011
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 12 s · $0.0085 for the run, $0.000034 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
34/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"bad_format": 2, "extra_call": 7, "gave_up_or_loop": 14, "wrong_args": 21, "wrong_tool": 3}
Tokens: 894011
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 12 s · $0.0085 for the run, $0.000034 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 4, "wrong_args": 18, "wrong_tool": 2}
Tokens: 1133799
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 57 s · $0.0397 for the run, $0.000158 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
30/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 4, "wrong_args": 18, "wrong_tool": 2}
Tokens: 1133799
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 57 s · $0.0397 for the run, $0.000158 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 4, "wrong_args": 18, "wrong_tool": 2}
Tokens: 1133799
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 57 s · $0.0397 for the run, $0.000158 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
32/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 4, "wrong_args": 18, "wrong_tool": 2}
Tokens: 1133799
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 57 s · $0.0397 for the run, $0.000158 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 4, "wrong_args": 18, "wrong_tool": 2}
Tokens: 1133799
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 57 s · $0.0397 for the run, $0.000158 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
40/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 4, "wrong_args": 18, "wrong_tool": 2}
Tokens: 1133799
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 57 s · $0.0397 for the run, $0.000158 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
40/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 14, "gave_up_or_loop": 11, "made_up_result": 2, "wrong_args": 33, "wrong_tool": 7}
Tokens: 1173418
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 26 s · $0.0180 for the run, $0.000071 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
26/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 14, "gave_up_or_loop": 11, "made_up_result": 2, "wrong_args": 33, "wrong_tool": 7}
Tokens: 1173418
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 26 s · $0.0180 for the run, $0.000071 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
37/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 14, "gave_up_or_loop": 11, "made_up_result": 2, "wrong_args": 33, "wrong_tool": 7}
Tokens: 1173418
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 26 s · $0.0180 for the run, $0.000071 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
19/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 14, "gave_up_or_loop": 11, "made_up_result": 2, "wrong_args": 33, "wrong_tool": 7}
Tokens: 1173418
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 26 s · $0.0180 for the run, $0.000071 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
35/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 14, "gave_up_or_loop": 11, "made_up_result": 2, "wrong_args": 33, "wrong_tool": 7}
Tokens: 1173418
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 26 s · $0.0180 for the run, $0.000071 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
28/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 14, "gave_up_or_loop": 11, "made_up_result": 2, "wrong_args": 33, "wrong_tool": 7}
Tokens: 1173418
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 26 s · $0.0180 for the run, $0.000071 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 7, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 13, "wrong_tool": 9}
Tokens: 1267484
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 59 s · $0.0411 for the run, $0.000163 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
34/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 7, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 13, "wrong_tool": 9}
Tokens: 1267484
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 59 s · $0.0411 for the run, $0.000163 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 7, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 13, "wrong_tool": 9}
Tokens: 1267484
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 59 s · $0.0411 for the run, $0.000163 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
36/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 7, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 13, "wrong_tool": 9}
Tokens: 1267484
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 59 s · $0.0411 for the run, $0.000163 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
37/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 7, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 13, "wrong_tool": 9}
Tokens: 1267484
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 59 s · $0.0411 for the run, $0.000163 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
29/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 7, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 13, "wrong_tool": 9}
Tokens: 1267484
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 59 s · $0.0411 for the run, $0.000163 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 1, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 11, "wrong_tool": 7}
Tokens: 1026831
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
37/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 1, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 11, "wrong_tool": 7}
Tokens: 1026831
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 1, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 11, "wrong_tool": 7}
Tokens: 1026831
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 1, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 11, "wrong_tool": 7}
Tokens: 1026831
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
40/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 1, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 11, "wrong_tool": 7}
Tokens: 1026831
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
27/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 1, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 11, "wrong_tool": 7}
Tokens: 1026831
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 5, "wrong_args": 15, "wrong_tool": 6}
Tokens: 1138207
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 78 s · $0.0539 for the run, $0.000214 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
40/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 5, "wrong_args": 15, "wrong_tool": 6}
Tokens: 1138207
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 78 s · $0.0539 for the run, $0.000214 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
41/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 5, "wrong_args": 15, "wrong_tool": 6}
Tokens: 1138207
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 78 s · $0.0539 for the run, $0.000214 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
30/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 5, "wrong_args": 15, "wrong_tool": 6}
Tokens: 1138207
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 78 s · $0.0539 for the run, $0.000214 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
40/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 5, "wrong_args": 15, "wrong_tool": 6}
Tokens: 1138207
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 78 s · $0.0539 for the run, $0.000214 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
33/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 5, "wrong_args": 15, "wrong_tool": 6}
Tokens: 1138207
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 78 s · $0.0539 for the run, $0.000214 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"bad_format": 3, "extra_call": 22, "wrong_args": 20, "wrong_tool": 6}
Tokens: 1179082
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 53 s · $0.0369 for the run, $0.000146 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
30/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"bad_format": 3, "extra_call": 22, "wrong_args": 20, "wrong_tool": 6}
Tokens: 1179082
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 53 s · $0.0369 for the run, $0.000146 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
40/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"bad_format": 3, "extra_call": 22, "wrong_args": 20, "wrong_tool": 6}
Tokens: 1179082
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 53 s · $0.0369 for the run, $0.000146 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
24/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"bad_format": 3, "extra_call": 22, "wrong_args": 20, "wrong_tool": 6}
Tokens: 1179082
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 53 s · $0.0369 for the run, $0.000146 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
32/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"bad_format": 3, "extra_call": 22, "wrong_args": 20, "wrong_tool": 6}
Tokens: 1179082
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 53 s · $0.0369 for the run, $0.000146 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
33/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"bad_format": 3, "extra_call": 22, "wrong_args": 20, "wrong_tool": 6}
Tokens: 1179082
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 53 s · $0.0369 for the run, $0.000146 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
41/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 15, "wrong_tool": 4}
Tokens: 1023064
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 87 s · $0.0600 for the run, $0.000238 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
36/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 15, "wrong_tool": 4}
Tokens: 1023064
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 87 s · $0.0600 for the run, $0.000238 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 15, "wrong_tool": 4}
Tokens: 1023064
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 87 s · $0.0600 for the run, $0.000238 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
26/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 15, "wrong_tool": 4}
Tokens: 1023064
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 87 s · $0.0600 for the run, $0.000238 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 15, "wrong_tool": 4}
Tokens: 1023064
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 87 s · $0.0600 for the run, $0.000238 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
37/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 15, "wrong_tool": 4}
Tokens: 1023064
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 87 s · $0.0600 for the run, $0.000238 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the run, identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4
Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 4, "gave_up_or_loop": 1, "wrong_args": 22}
Tokens: 797158
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the run, identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4
Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.
30/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 4, "gave_up_or_loop": 1, "wrong_args": 22}
Tokens: 797158
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the run, identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4
Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 4, "gave_up_or_loop": 1, "wrong_args": 22}
Tokens: 797158
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the run, identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4
Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.
31/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 4, "gave_up_or_loop": 1, "wrong_args": 22}
Tokens: 797158
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the run, identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4
Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.
39/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 4, "gave_up_or_loop": 1, "wrong_args": 22}
Tokens: 797158
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the run, identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4
Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.
41/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 4, "gave_up_or_loop": 1, "wrong_args": 22}
Tokens: 797158
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"gave_up_or_loop": 5, "made_up_result": 1, "wrong_args": 39, "wrong_tool": 7}
Tokens: 1058883
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 36 s · $0.0249 for the run, $0.000099 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
25/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"gave_up_or_loop": 5, "made_up_result": 1, "wrong_args": 39, "wrong_tool": 7}
Tokens: 1058883
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 36 s · $0.0249 for the run, $0.000099 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"gave_up_or_loop": 5, "made_up_result": 1, "wrong_args": 39, "wrong_tool": 7}
Tokens: 1058883
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 36 s · $0.0249 for the run, $0.000099 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
26/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"gave_up_or_loop": 5, "made_up_result": 1, "wrong_args": 39, "wrong_tool": 7}
Tokens: 1058883
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 36 s · $0.0249 for the run, $0.000099 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"gave_up_or_loop": 5, "made_up_result": 1, "wrong_args": 39, "wrong_tool": 7}
Tokens: 1058883
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 36 s · $0.0249 for the run, $0.000099 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
23/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"gave_up_or_loop": 5, "made_up_result": 1, "wrong_args": 39, "wrong_tool": 7}
Tokens: 1058883
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 36 s · $0.0249 for the run, $0.000099 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
41/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"no_call": 1, "wrong_args": 6, "wrong_tool": 11}
Tokens: 1560191
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 157 s · $0.1089 for the run, $0.000432 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
36/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"no_call": 1, "wrong_args": 6, "wrong_tool": 11}
Tokens: 1560191
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 157 s · $0.1089 for the run, $0.000432 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"no_call": 1, "wrong_args": 6, "wrong_tool": 11}
Tokens: 1560191
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 157 s · $0.1089 for the run, $0.000432 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"no_call": 1, "wrong_args": 6, "wrong_tool": 11}
Tokens: 1560191
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 157 s · $0.1089 for the run, $0.000432 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"no_call": 1, "wrong_args": 6, "wrong_tool": 11}
Tokens: 1560191
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 157 s · $0.1089 for the run, $0.000432 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
31/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"no_call": 1, "wrong_args": 6, "wrong_tool": 11}
Tokens: 1560191
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 157 s · $0.1089 for the run, $0.000432 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 3, "gave_up_or_loop": 3, "wrong_args": 21, "wrong_tool": 14}
Tokens: 1026531
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 82 s · $0.0564 for the run, $0.000224 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
32/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 3, "gave_up_or_loop": 3, "wrong_args": 21, "wrong_tool": 14}
Tokens: 1026531
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 82 s · $0.0564 for the run, $0.000224 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
41/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 3, "gave_up_or_loop": 3, "wrong_args": 21, "wrong_tool": 14}
Tokens: 1026531
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 82 s · $0.0564 for the run, $0.000224 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
27/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 3, "gave_up_or_loop": 3, "wrong_args": 21, "wrong_tool": 14}
Tokens: 1026531
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 82 s · $0.0564 for the run, $0.000224 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 3, "gave_up_or_loop": 3, "wrong_args": 21, "wrong_tool": 14}
Tokens: 1026531
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 82 s · $0.0564 for the run, $0.000224 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
27/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 3, "gave_up_or_loop": 3, "wrong_args": 21, "wrong_tool": 14}
Tokens: 1026531
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 82 s · $0.0564 for the run, $0.000224 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"gave_up_or_loop": 2, "wrong_args": 8, "wrong_tool": 7}
Tokens: 1580185
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 332 s · $0.2294 for the run, $0.000910 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
35/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"gave_up_or_loop": 2, "wrong_args": 8, "wrong_tool": 7}
Tokens: 1580185
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 332 s · $0.2294 for the run, $0.000910 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"gave_up_or_loop": 2, "wrong_args": 8, "wrong_tool": 7}
Tokens: 1580185
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 332 s · $0.2294 for the run, $0.000910 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
40/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"gave_up_or_loop": 2, "wrong_args": 8, "wrong_tool": 7}
Tokens: 1580185
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 332 s · $0.2294 for the run, $0.000910 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"gave_up_or_loop": 2, "wrong_args": 8, "wrong_tool": 7}
Tokens: 1580185
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 332 s · $0.2294 for the run, $0.000910 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
34/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"gave_up_or_loop": 2, "wrong_args": 8, "wrong_tool": 7}
Tokens: 1580185
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 332 s · $0.2294 for the run, $0.000910 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"gave_up_or_loop": 2, "wrong_tool": 12}
Tokens: 728093
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 38 s · $0.0266 for the run, $0.000105 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"gave_up_or_loop": 2, "wrong_tool": 12}
Tokens: 728093
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 38 s · $0.0266 for the run, $0.000105 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"gave_up_or_loop": 2, "wrong_tool": 12}
Tokens: 728093
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 38 s · $0.0266 for the run, $0.000105 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"gave_up_or_loop": 2, "wrong_tool": 12}
Tokens: 728093
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 38 s · $0.0266 for the run, $0.000105 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"gave_up_or_loop": 2, "wrong_tool": 12}
Tokens: 728093
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 38 s · $0.0266 for the run, $0.000105 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
28/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"gave_up_or_loop": 2, "wrong_tool": 12}
Tokens: 728093
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 38 s · $0.0266 for the run, $0.000105 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 2, "wrong_args": 4, "wrong_tool": 12}
Tokens: 870626
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 121 s · $0.0836 for the run, $0.000332 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
38/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 2, "wrong_args": 4, "wrong_tool": 12}
Tokens: 870626
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 121 s · $0.0836 for the run, $0.000332 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 2, "wrong_args": 4, "wrong_tool": 12}
Tokens: 870626
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 121 s · $0.0836 for the run, $0.000332 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
41/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 2, "wrong_args": 4, "wrong_tool": 12}
Tokens: 870626
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 121 s · $0.0836 for the run, $0.000332 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 2, "wrong_args": 4, "wrong_tool": 12}
Tokens: 870626
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 121 s · $0.0836 for the run, $0.000332 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
29/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 2, "wrong_args": 4, "wrong_tool": 12}
Tokens: 870626
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 121 s · $0.0836 for the run, $0.000332 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183
Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 2, "wrong_tool": 7}
Tokens: 998331
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 135 s · $0.0932 for the run, $0.000370 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183
Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2
37/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 2, "wrong_tool": 7}
Tokens: 998331
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 135 s · $0.0932 for the run, $0.000370 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183
Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 2, "wrong_tool": 7}
Tokens: 998331
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 135 s · $0.0932 for the run, $0.000370 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183
Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 2, "wrong_tool": 7}
Tokens: 998331
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 135 s · $0.0932 for the run, $0.000370 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183
Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2
37/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 2, "wrong_tool": 7}
Tokens: 998331
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 135 s · $0.0932 for the run, $0.000370 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183
Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2
34/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 2, "wrong_tool": 7}
Tokens: 998331
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 135 s · $0.0932 for the run, $0.000370 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 5, "wrong_args": 1, "wrong_tool": 2}
Tokens: 948341
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 144 s · $0.0993 for the run, $0.000394 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 5, "wrong_args": 1, "wrong_tool": 2}
Tokens: 948341
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 144 s · $0.0993 for the run, $0.000394 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 5, "wrong_args": 1, "wrong_tool": 2}
Tokens: 948341
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 144 s · $0.0993 for the run, $0.000394 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
37/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 5, "wrong_args": 1, "wrong_tool": 2}
Tokens: 948341
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 144 s · $0.0993 for the run, $0.000394 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 5, "wrong_args": 1, "wrong_tool": 2}
Tokens: 948341
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 144 s · $0.0993 for the run, $0.000394 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
39/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"extra_call": 5, "wrong_args": 1, "wrong_tool": 2}
Tokens: 948341
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 144 s · $0.0993 for the run, $0.000394 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"bad_format": 5, "extra_call": 10, "wrong_args": 1, "wrong_tool": 4}
Tokens: 1098479
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 315 s · $0.2177 for the run, $0.000864 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
41/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"bad_format": 5, "extra_call": 10, "wrong_args": 1, "wrong_tool": 4}
Tokens: 1098479
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 315 s · $0.2177 for the run, $0.000864 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
37/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"bad_format": 5, "extra_call": 10, "wrong_args": 1, "wrong_tool": 4}
Tokens: 1098479
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 315 s · $0.2177 for the run, $0.000864 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
36/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"bad_format": 5, "extra_call": 10, "wrong_args": 1, "wrong_tool": 4}
Tokens: 1098479
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 315 s · $0.2177 for the run, $0.000864 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"bad_format": 5, "extra_call": 10, "wrong_args": 1, "wrong_tool": 4}
Tokens: 1098479
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 315 s · $0.2177 for the run, $0.000864 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
34/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"bad_format": 5, "extra_call": 10, "wrong_args": 1, "wrong_tool": 4}
Tokens: 1098479
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 315 s · $0.2177 for the run, $0.000864 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {}
Tokens: 691572
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 86 s · $0.0594 for the run, $0.000236 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {}
Tokens: 691572
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 86 s · $0.0594 for the run, $0.000236 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {}
Tokens: 691572
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 86 s · $0.0594 for the run, $0.000236 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {}
Tokens: 691572
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 86 s · $0.0594 for the run, $0.000236 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {}
Tokens: 691572
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 86 s · $0.0594 for the run, $0.000236 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {}
Tokens: 691572
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 86 s · $0.0594 for the run, $0.000236 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"bad_format": 2, "wrong_tool": 1}
Tokens: 783015
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 252 s · $0.1740 for the run, $0.000691 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"bad_format": 2, "wrong_tool": 1}
Tokens: 783015
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 252 s · $0.1740 for the run, $0.000691 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"bad_format": 2, "wrong_tool": 1}
Tokens: 783015
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 252 s · $0.1740 for the run, $0.000691 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.
40/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"bad_format": 2, "wrong_tool": 1}
Tokens: 783015
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 252 s · $0.1740 for the run, $0.000691 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.
42/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"bad_format": 2, "wrong_tool": 1}
Tokens: 783015
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 252 s · $0.1740 for the run, $0.000691 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.
41/42 scored passed; 0 unscored; 42 attempted
Failure tags: {"bad_format": 2, "wrong_tool": 1}
Tokens: 783015
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 252 s · $0.1740 for the run, $0.000691 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Not qualified: Qualification failed on public task single-01: after a correct get_price call and its mock result, the model's next token was <|tool_response> (an end-of-generation id), so the reply was empty. With thinking off, neither the maker's chat template nor vLLM's example template adds a continuation cue after a tool result. One observation; cause not proven.
No score: the setup did not complete the qualification tool round trip, so its full run was not admitted.
Qualification evidence sha256: 11b2effa6e9f1f57d2fd1a76001d65b3f2a5f497ee815666bc2a8b6e8d41a18a
Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
0/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 18, "extra_call": 6, "gave_up_or_loop": 4, "wrong_args": 73, "wrong_tool": 1}
Tokens: 1340110
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 22 s · $0.0156 for the run, $0.000108 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
2/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 18, "extra_call": 6, "gave_up_or_loop": 4, "wrong_args": 73, "wrong_tool": 1}
Tokens: 1340110
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 22 s · $0.0156 for the run, $0.000108 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
10/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 18, "extra_call": 6, "gave_up_or_loop": 4, "wrong_args": 73, "wrong_tool": 1}
Tokens: 1340110
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 22 s · $0.0156 for the run, $0.000108 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
18/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 18, "extra_call": 6, "gave_up_or_loop": 4, "wrong_args": 73, "wrong_tool": 1}
Tokens: 1340110
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 22 s · $0.0156 for the run, $0.000108 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
9/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 18, "extra_call": 6, "gave_up_or_loop": 4, "wrong_args": 73, "wrong_tool": 1}
Tokens: 1340110
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 22 s · $0.0156 for the run, $0.000108 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
3/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 18, "extra_call": 6, "gave_up_or_loop": 4, "wrong_args": 73, "wrong_tool": 1}
Tokens: 1340110
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 22 s · $0.0156 for the run, $0.000108 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
12/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 25, "extra_call": 5, "wrong_args": 21}
Tokens: 1790981
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 51 s · $0.0355 for the run, $0.000246 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
18/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 25, "extra_call": 5, "wrong_args": 21}
Tokens: 1790981
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 51 s · $0.0355 for the run, $0.000246 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
19/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 25, "extra_call": 5, "wrong_args": 21}
Tokens: 1790981
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 51 s · $0.0355 for the run, $0.000246 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
24/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 25, "extra_call": 5, "wrong_args": 21}
Tokens: 1790981
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 51 s · $0.0355 for the run, $0.000246 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
14/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 25, "extra_call": 5, "wrong_args": 21}
Tokens: 1790981
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 51 s · $0.0355 for the run, $0.000246 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
6/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 25, "extra_call": 5, "wrong_args": 21}
Tokens: 1790981
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 51 s · $0.0355 for the run, $0.000246 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9534 standard set; #10411 hard set, identical argv) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4
19/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 11, "wrong_args": 27}
Tokens: 2048615
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 106 s · $0.0734 for the run, $0.000510 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#9534 standard set; #10411 hard set, identical argv) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4
15/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 11, "wrong_args": 27}
Tokens: 2048615
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 106 s · $0.0734 for the run, $0.000510 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#9534 standard set; #10411 hard set, identical argv) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4
20/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 11, "wrong_args": 27}
Tokens: 2048615
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 106 s · $0.0734 for the run, $0.000510 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#9534 standard set; #10411 hard set, identical argv) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4
24/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 11, "wrong_args": 27}
Tokens: 2048615
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 106 s · $0.0734 for the run, $0.000510 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#9534 standard set; #10411 hard set, identical argv) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4
21/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 11, "wrong_args": 27}
Tokens: 2048615
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 106 s · $0.0734 for the run, $0.000510 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#9534 standard set; #10411 hard set, identical argv) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4
7/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 11, "wrong_args": 27}
Tokens: 2048615
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 106 s · $0.0734 for the run, $0.000510 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
2/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 18, "extra_call": 7, "gave_up_or_loop": 6, "wrong_args": 71}
Tokens: 1586690
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 24 s · $0.0165 for the run, $0.000114 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
4/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 18, "extra_call": 7, "gave_up_or_loop": 6, "wrong_args": 71}
Tokens: 1586690
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 24 s · $0.0165 for the run, $0.000114 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
2/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 18, "extra_call": 7, "gave_up_or_loop": 6, "wrong_args": 71}
Tokens: 1586690
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 24 s · $0.0165 for the run, $0.000114 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
16/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 18, "extra_call": 7, "gave_up_or_loop": 6, "wrong_args": 71}
Tokens: 1586690
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 24 s · $0.0165 for the run, $0.000114 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
17/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 18, "extra_call": 7, "gave_up_or_loop": 6, "wrong_args": 71}
Tokens: 1586690
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 24 s · $0.0165 for the run, $0.000114 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
1/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 18, "extra_call": 7, "gave_up_or_loop": 6, "wrong_args": 71}
Tokens: 1586690
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 24 s · $0.0165 for the run, $0.000114 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
21/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 6, "wrong_args": 18}
Tokens: 2288473
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 92 s · $0.0637 for the run, $0.000442 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
17/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 6, "wrong_args": 18}
Tokens: 2288473
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 92 s · $0.0637 for the run, $0.000442 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
22/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 6, "wrong_args": 18}
Tokens: 2288473
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 92 s · $0.0637 for the run, $0.000442 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
24/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 6, "wrong_args": 18}
Tokens: 2288473
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 92 s · $0.0637 for the run, $0.000442 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
23/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 6, "wrong_args": 18}
Tokens: 2288473
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 92 s · $0.0637 for the run, $0.000442 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
13/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 6, "wrong_args": 18}
Tokens: 2288473
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 92 s · $0.0637 for the run, $0.000442 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
18/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 3, "gave_up_or_loop": 8, "made_up_result": 1, "wrong_args": 47, "wrong_tool": 2}
Tokens: 2473838
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 66 s · $0.0455 for the run, $0.000316 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
7/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 3, "gave_up_or_loop": 8, "made_up_result": 1, "wrong_args": 47, "wrong_tool": 2}
Tokens: 2473838
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 66 s · $0.0455 for the run, $0.000316 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
15/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 3, "gave_up_or_loop": 8, "made_up_result": 1, "wrong_args": 47, "wrong_tool": 2}
Tokens: 2473838
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 66 s · $0.0455 for the run, $0.000316 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
20/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 3, "gave_up_or_loop": 8, "made_up_result": 1, "wrong_args": 47, "wrong_tool": 2}
Tokens: 2473838
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 66 s · $0.0455 for the run, $0.000316 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
21/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 3, "gave_up_or_loop": 8, "made_up_result": 1, "wrong_args": 47, "wrong_tool": 2}
Tokens: 2473838
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 66 s · $0.0455 for the run, $0.000316 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
2/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 3, "gave_up_or_loop": 8, "made_up_result": 1, "wrong_args": 47, "wrong_tool": 2}
Tokens: 2473838
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 66 s · $0.0455 for the run, $0.000316 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
18/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 5, "wrong_args": 40, "wrong_tool": 1}
Tokens: 2603860
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 112 s · $0.0774 for the run, $0.000537 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
11/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 5, "wrong_args": 40, "wrong_tool": 1}
Tokens: 2603860
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 112 s · $0.0774 for the run, $0.000537 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
20/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 5, "wrong_args": 40, "wrong_tool": 1}
Tokens: 2603860
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 112 s · $0.0774 for the run, $0.000537 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
24/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 5, "wrong_args": 40, "wrong_tool": 1}
Tokens: 2603860
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 112 s · $0.0774 for the run, $0.000537 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
20/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 5, "wrong_args": 40, "wrong_tool": 1}
Tokens: 2603860
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 112 s · $0.0774 for the run, $0.000537 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
5/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 5, "wrong_args": 40, "wrong_tool": 1}
Tokens: 2603860
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 112 s · $0.0774 for the run, $0.000537 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
24/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 1, "wrong_args": 14}
Tokens: 2194487
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 76 s · $0.0528 for the run, $0.000367 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
21/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 1, "wrong_args": 14}
Tokens: 2194487
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 76 s · $0.0528 for the run, $0.000367 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
23/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 1, "wrong_args": 14}
Tokens: 2194487
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 76 s · $0.0528 for the run, $0.000367 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
24/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 1, "wrong_args": 14}
Tokens: 2194487
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 76 s · $0.0528 for the run, $0.000367 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
22/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 1, "wrong_args": 14}
Tokens: 2194487
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 76 s · $0.0528 for the run, $0.000367 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
15/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 1, "wrong_args": 14}
Tokens: 2194487
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 76 s · $0.0528 for the run, $0.000367 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
21/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 3, "wrong_args": 25}
Tokens: 2457570
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 150 s · $0.1035 for the run, $0.000719 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
22/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 3, "wrong_args": 25}
Tokens: 2457570
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 150 s · $0.1035 for the run, $0.000719 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
21/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 3, "wrong_args": 25}
Tokens: 2457570
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 150 s · $0.1035 for the run, $0.000719 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
23/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 3, "wrong_args": 25}
Tokens: 2457570
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 150 s · $0.1035 for the run, $0.000719 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
20/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 3, "wrong_args": 25}
Tokens: 2457570
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 150 s · $0.1035 for the run, $0.000719 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
9/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 3, "wrong_args": 25}
Tokens: 2457570
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 150 s · $0.1035 for the run, $0.000719 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
1/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 1, "extra_call": 15, "gave_up_or_loop": 6, "wrong_args": 81, "wrong_tool": 5}
Tokens: 2355910
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 94 s · $0.0649 for the run, $0.000451 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
1/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 1, "extra_call": 15, "gave_up_or_loop": 6, "wrong_args": 81, "wrong_tool": 5}
Tokens: 2355910
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 94 s · $0.0649 for the run, $0.000451 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
2/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 1, "extra_call": 15, "gave_up_or_loop": 6, "wrong_args": 81, "wrong_tool": 5}
Tokens: 2355910
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 94 s · $0.0649 for the run, $0.000451 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
11/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 1, "extra_call": 15, "gave_up_or_loop": 6, "wrong_args": 81, "wrong_tool": 5}
Tokens: 2355910
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 94 s · $0.0649 for the run, $0.000451 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
21/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 1, "extra_call": 15, "gave_up_or_loop": 6, "wrong_args": 81, "wrong_tool": 5}
Tokens: 2355910
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 94 s · $0.0649 for the run, $0.000451 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
0/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 1, "extra_call": 15, "gave_up_or_loop": 6, "wrong_args": 81, "wrong_tool": 5}
Tokens: 2355910
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 94 s · $0.0649 for the run, $0.000451 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
16/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 12, "wrong_args": 30}
Tokens: 2137047
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 275 s · $0.1899 for the run, $0.001319 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
11/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 12, "wrong_args": 30}
Tokens: 2137047
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 275 s · $0.1899 for the run, $0.001319 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
23/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 12, "wrong_args": 30}
Tokens: 2137047
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 275 s · $0.1899 for the run, $0.001319 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
24/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 12, "wrong_args": 30}
Tokens: 2137047
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 275 s · $0.1899 for the run, $0.001319 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
21/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 12, "wrong_args": 30}
Tokens: 2137047
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 275 s · $0.1899 for the run, $0.001319 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
7/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 12, "wrong_args": 30}
Tokens: 2137047
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 275 s · $0.1899 for the run, $0.001319 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the standard run; #10423 GPU 1 for the hard set; identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4
Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.
18/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 1, "made_up_result": 5, "wrong_args": 27}
Tokens: 1836873
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 68 s · $0.0469 for the run, $0.000326 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the standard run; #10423 GPU 1 for the hard set; identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4
Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.
17/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 1, "made_up_result": 5, "wrong_args": 27}
Tokens: 1836873
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 68 s · $0.0469 for the run, $0.000326 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the standard run; #10423 GPU 1 for the hard set; identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4
Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.
24/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 1, "made_up_result": 5, "wrong_args": 27}
Tokens: 1836873
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 68 s · $0.0469 for the run, $0.000326 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the standard run; #10423 GPU 1 for the hard set; identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4
Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.
24/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 1, "made_up_result": 5, "wrong_args": 27}
Tokens: 1836873
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 68 s · $0.0469 for the run, $0.000326 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the standard run; #10423 GPU 1 for the hard set; identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4
Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.
23/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 1, "made_up_result": 5, "wrong_args": 27}
Tokens: 1836873
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 68 s · $0.0469 for the run, $0.000326 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the standard run; #10423 GPU 1 for the hard set; identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4
Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.
5/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 1, "made_up_result": 5, "wrong_args": 27}
Tokens: 1836873
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 68 s · $0.0469 for the run, $0.000326 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
5/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 12, "gave_up_or_loop": 5, "wrong_args": 68, "wrong_tool": 1}
Tokens: 2601502
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 267 s · $0.1847 for the run, $0.001283 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
5/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 12, "gave_up_or_loop": 5, "wrong_args": 68, "wrong_tool": 1}
Tokens: 2601502
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 267 s · $0.1847 for the run, $0.001283 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
14/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 12, "gave_up_or_loop": 5, "wrong_args": 68, "wrong_tool": 1}
Tokens: 2601502
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 267 s · $0.1847 for the run, $0.001283 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
16/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 12, "gave_up_or_loop": 5, "wrong_args": 68, "wrong_tool": 1}
Tokens: 2601502
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 267 s · $0.1847 for the run, $0.001283 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
15/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 12, "gave_up_or_loop": 5, "wrong_args": 68, "wrong_tool": 1}
Tokens: 2601502
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 267 s · $0.1847 for the run, $0.001283 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
3/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 12, "gave_up_or_loop": 5, "wrong_args": 68, "wrong_tool": 1}
Tokens: 2601502
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 267 s · $0.1847 for the run, $0.001283 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
24/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 4, "no_call": 1, "wrong_args": 12}
Tokens: 4304813
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 724 s · $0.5010 for the run, $0.003479 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
21/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 4, "no_call": 1, "wrong_args": 12}
Tokens: 4304813
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 724 s · $0.5010 for the run, $0.003479 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
21/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 4, "no_call": 1, "wrong_args": 12}
Tokens: 4304813
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 724 s · $0.5010 for the run, $0.003479 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
24/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 4, "no_call": 1, "wrong_args": 12}
Tokens: 4304813
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 724 s · $0.5010 for the run, $0.003479 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
18/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 4, "no_call": 1, "wrong_args": 12}
Tokens: 4304813
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 724 s · $0.5010 for the run, $0.003479 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
19/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 4, "no_call": 1, "wrong_args": 12}
Tokens: 4304813
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 724 s · $0.5010 for the run, $0.003479 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
8/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 14, "wrong_args": 68, "wrong_tool": 1}
Tokens: 2353976
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 290 s · $0.2003 for the run, $0.001391 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
5/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 14, "wrong_args": 68, "wrong_tool": 1}
Tokens: 2353976
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 290 s · $0.2003 for the run, $0.001391 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
9/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 14, "wrong_args": 68, "wrong_tool": 1}
Tokens: 2353976
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 290 s · $0.2003 for the run, $0.001391 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
18/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 14, "wrong_args": 68, "wrong_tool": 1}
Tokens: 2353976
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 290 s · $0.2003 for the run, $0.001391 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
19/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 14, "wrong_args": 68, "wrong_tool": 1}
Tokens: 2353976
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 290 s · $0.2003 for the run, $0.001391 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
2/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 14, "wrong_args": 68, "wrong_tool": 1}
Tokens: 2353976
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 290 s · $0.2003 for the run, $0.001391 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
Run note: Not tiered: one of its 144 task runs is unscored. On a long-output task the server refused the request because the prompt plus the fixed 32,768-token answer budget exceeded its 69,632-token context (HTTP 400). Serving and configuration errors are not counted as model failures; the score is over the scored runs.
24/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"wrong_args": 12}
Tokens: Not reported
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 1214 s · $0.8400 for the run, $0.005874 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
Run note: Not tiered: one of its 144 task runs is unscored. On a long-output task the server refused the request because the prompt plus the fixed 32,768-token answer budget exceeded its 69,632-token context (HTTP 400). Serving and configuration errors are not counted as model failures; the score is over the scored runs.
19/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"wrong_args": 12}
Tokens: Not reported
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 1214 s · $0.8400 for the run, $0.005874 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
Run note: Not tiered: one of its 144 task runs is unscored. On a long-output task the server refused the request because the prompt plus the fixed 32,768-token answer budget exceeded its 69,632-token context (HTTP 400). Serving and configuration errors are not counted as model failures; the score is over the scored runs.
24/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"wrong_args": 12}
Tokens: Not reported
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 1214 s · $0.8400 for the run, $0.005874 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
Run note: Not tiered: one of its 144 task runs is unscored. On a long-output task the server refused the request because the prompt plus the fixed 32,768-token answer budget exceeded its 69,632-token context (HTTP 400). Serving and configuration errors are not counted as model failures; the score is over the scored runs.
24/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"wrong_args": 12}
Tokens: Not reported
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 1214 s · $0.8400 for the run, $0.005874 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
Run note: Not tiered: one of its 144 task runs is unscored. On a long-output task the server refused the request because the prompt plus the fixed 32,768-token answer budget exceeded its 69,632-token context (HTTP 400). Serving and configuration errors are not counted as model failures; the score is over the scored runs.
23/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"wrong_args": 12}
Tokens: Not reported
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 1214 s · $0.8400 for the run, $0.005874 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
Run note: Not tiered: one of its 144 task runs is unscored. On a long-output task the server refused the request because the prompt plus the fixed 32,768-token answer budget exceeded its 69,632-token context (HTTP 400). Serving and configuration errors are not counted as model failures; the score is over the scored runs.
17/23 scored passed; 1 unscored; 24 attempted
Failure tags: {"wrong_args": 12}
Tokens: Not reported
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 1214 s · $0.8400 for the run, $0.005874 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
24/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 7, "wrong_args": 34}
Tokens: 1798075
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 83 s · $0.0573 for the run, $0.000398 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
20/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 7, "wrong_args": 34}
Tokens: 1798075
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 83 s · $0.0573 for the run, $0.000398 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
17/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 7, "wrong_args": 34}
Tokens: 1798075
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 83 s · $0.0573 for the run, $0.000398 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
18/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 7, "wrong_args": 34}
Tokens: 1798075
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 83 s · $0.0573 for the run, $0.000398 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
24/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 7, "wrong_args": 34}
Tokens: 1798075
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 83 s · $0.0573 for the run, $0.000398 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
0/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 7, "wrong_args": 34}
Tokens: 1798075
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 83 s · $0.0573 for the run, $0.000398 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
24/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 1, "wrong_args": 6}
Tokens: 2017522
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 229 s · $0.1585 for the run, $0.001100 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
24/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 1, "wrong_args": 6}
Tokens: 2017522
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 229 s · $0.1585 for the run, $0.001100 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
24/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 1, "wrong_args": 6}
Tokens: 2017522
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 229 s · $0.1585 for the run, $0.001100 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
24/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 1, "wrong_args": 6}
Tokens: 2017522
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 229 s · $0.1585 for the run, $0.001100 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
24/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 1, "wrong_args": 6}
Tokens: 2017522
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 229 s · $0.1585 for the run, $0.001100 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
17/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"extra_call": 1, "wrong_args": 6}
Tokens: 2017522
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 229 s · $0.1585 for the run, $0.001100 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10561 hard set on GPU 1, identical argv) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183
Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2
23/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 8, "extra_call": 1, "wrong_args": 5}
Tokens: 2110323
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 201 s · $0.1390 for the run, $0.000965 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10561 hard set on GPU 1, identical argv) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183
Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2
24/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 8, "extra_call": 1, "wrong_args": 5}
Tokens: 2110323
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 201 s · $0.1390 for the run, $0.000965 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10561 hard set on GPU 1, identical argv) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183
Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2
22/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 8, "extra_call": 1, "wrong_args": 5}
Tokens: 2110323
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 201 s · $0.1390 for the run, $0.000965 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10561 hard set on GPU 1, identical argv) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183
Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2
24/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 8, "extra_call": 1, "wrong_args": 5}
Tokens: 2110323
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 201 s · $0.1390 for the run, $0.000965 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10561 hard set on GPU 1, identical argv) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183
Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2
21/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 8, "extra_call": 1, "wrong_args": 5}
Tokens: 2110323
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 201 s · $0.1390 for the run, $0.000965 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10561 hard set on GPU 1, identical argv) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183
Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2
16/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 8, "extra_call": 1, "wrong_args": 5}
Tokens: 2110323
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 201 s · $0.1390 for the run, $0.000965 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
18/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 1, "extra_call": 6, "wrong_args": 30}
Tokens: 2144354
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 242 s · $0.1672 for the run, $0.001161 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
17/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 1, "extra_call": 6, "wrong_args": 30}
Tokens: 2144354
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 242 s · $0.1672 for the run, $0.001161 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
24/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 1, "extra_call": 6, "wrong_args": 30}
Tokens: 2144354
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 242 s · $0.1672 for the run, $0.001161 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
18/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 1, "extra_call": 6, "wrong_args": 30}
Tokens: 2144354
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 242 s · $0.1672 for the run, $0.001161 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
24/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 1, "extra_call": 6, "wrong_args": 30}
Tokens: 2144354
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 242 s · $0.1672 for the run, $0.001161 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
6/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 1, "extra_call": 6, "wrong_args": 30}
Tokens: 2144354
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 242 s · $0.1672 for the run, $0.001161 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
21/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 9, "extra_call": 3, "wrong_args": 5}
Tokens: 2204967
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 516 s · $0.3566 for the run, $0.002477 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
21/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 9, "extra_call": 3, "wrong_args": 5}
Tokens: 2204967
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 516 s · $0.3566 for the run, $0.002477 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
24/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 9, "extra_call": 3, "wrong_args": 5}
Tokens: 2204967
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 516 s · $0.3566 for the run, $0.002477 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
23/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 9, "extra_call": 3, "wrong_args": 5}
Tokens: 2204967
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 516 s · $0.3566 for the run, $0.002477 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
23/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 9, "extra_call": 3, "wrong_args": 5}
Tokens: 2204967
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 516 s · $0.3566 for the run, $0.002477 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
15/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 9, "extra_call": 3, "wrong_args": 5}
Tokens: 2204967
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 516 s · $0.3566 for the run, $0.002477 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
18/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 20, "extra_call": 6, "wrong_args": 25}
Tokens: 1565256
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 281 s · $0.1943 for the run, $0.001349 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
15/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 20, "extra_call": 6, "wrong_args": 25}
Tokens: 1565256
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 281 s · $0.1943 for the run, $0.001349 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
19/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 20, "extra_call": 6, "wrong_args": 25}
Tokens: 1565256
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 281 s · $0.1943 for the run, $0.001349 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
18/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 20, "extra_call": 6, "wrong_args": 25}
Tokens: 1565256
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 281 s · $0.1943 for the run, $0.001349 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
20/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 20, "extra_call": 6, "wrong_args": 25}
Tokens: 1565256
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 281 s · $0.1943 for the run, $0.001349 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
3/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 20, "extra_call": 6, "wrong_args": 25}
Tokens: 1565256
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 281 s · $0.1943 for the run, $0.001349 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
24/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 13}
Tokens: 1830591
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 608 s · $0.4207 for the run, $0.002921 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
24/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 13}
Tokens: 1830591
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 608 s · $0.4207 for the run, $0.002921 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
18/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 13}
Tokens: 1830591
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 608 s · $0.4207 for the run, $0.002921 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
24/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 13}
Tokens: 1830591
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 608 s · $0.4207 for the run, $0.002921 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
23/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 13}
Tokens: 1830591
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 608 s · $0.4207 for the run, $0.002921 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
18/24 scored passed; 0 unscored; 24 attempted
Failure tags: {"bad_format": 13}
Tokens: 1830591
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 608 s · $0.4207 for the run, $0.002921 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
213/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"bad_format": 5, "extra_call": 7, "gave_up_or_loop": 2, "wrong_args": 19, "wrong_tool": 6}
Tokens: 773212
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 10 s · $0.0066 for the run, $0.000026 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
231/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 10, "gave_up_or_loop": 5, "wrong_args": 5, "wrong_tool": 1}
Tokens: 888087
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 23 s · $0.0162 for the run, $0.000064 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9534) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4
237/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"gave_up_or_loop": 5, "wrong_args": 7, "wrong_tool": 3}
Tokens: 1034078
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 71 s · $0.0492 for the run, $0.000195 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
205/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"bad_format": 2, "extra_call": 7, "gave_up_or_loop": 14, "wrong_args": 21, "wrong_tool": 3}
Tokens: 894011
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 12 s · $0.0085 for the run, $0.000034 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
228/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 4, "wrong_args": 18, "wrong_tool": 2}
Tokens: 1133799
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 57 s · $0.0397 for the run, $0.000158 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
185/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 14, "gave_up_or_loop": 11, "made_up_result": 2, "wrong_args": 33, "wrong_tool": 7}
Tokens: 1173418
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 26 s · $0.0180 for the run, $0.000071 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
220/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 7, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 13, "wrong_tool": 9}
Tokens: 1267484
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 59 s · $0.0411 for the run, $0.000163 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
230/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 1, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 11, "wrong_tool": 7}
Tokens: 1026831
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
226/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 5, "wrong_args": 15, "wrong_tool": 6}
Tokens: 1138207
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 78 s · $0.0539 for the run, $0.000214 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
201/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"bad_format": 3, "extra_call": 22, "wrong_args": 20, "wrong_tool": 6}
Tokens: 1179082
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 53 s · $0.0369 for the run, $0.000146 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
224/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 15, "wrong_tool": 4}
Tokens: 1023064
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 87 s · $0.0600 for the run, $0.000238 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the run, identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4
Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.
225/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 4, "gave_up_or_loop": 1, "wrong_args": 22}
Tokens: 797158
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
200/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"gave_up_or_loop": 5, "made_up_result": 1, "wrong_args": 39, "wrong_tool": 7}
Tokens: 1058883
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 36 s · $0.0249 for the run, $0.000099 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
234/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"no_call": 1, "wrong_args": 6, "wrong_tool": 11}
Tokens: 1560191
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 157 s · $0.1089 for the run, $0.000432 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
211/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 3, "gave_up_or_loop": 3, "wrong_args": 21, "wrong_tool": 14}
Tokens: 1026531
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 82 s · $0.0564 for the run, $0.000224 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
235/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"gave_up_or_loop": 2, "wrong_args": 8, "wrong_tool": 7}
Tokens: 1580185
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 332 s · $0.2294 for the run, $0.000910 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
238/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"gave_up_or_loop": 2, "wrong_tool": 12}
Tokens: 728093
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 38 s · $0.0266 for the run, $0.000105 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
234/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 2, "wrong_args": 4, "wrong_tool": 12}
Tokens: 870626
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 121 s · $0.0836 for the run, $0.000332 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183
Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2
234/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 2, "wrong_tool": 7}
Tokens: 998331
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 135 s · $0.0932 for the run, $0.000370 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
244/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 5, "wrong_args": 1, "wrong_tool": 2}
Tokens: 948341
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 144 s · $0.0993 for the run, $0.000394 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
232/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"bad_format": 5, "extra_call": 10, "wrong_args": 1, "wrong_tool": 4}
Tokens: 1098479
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 315 s · $0.2177 for the run, $0.000864 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.
252/252 scored passed; 0 unscored; 252 attempted
Failure tags: {}
Tokens: 691572
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 86 s · $0.0594 for the run, $0.000236 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.
249/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"bad_format": 2, "wrong_tool": 1}
Tokens: 783015
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 252 s · $0.1740 for the run, $0.000691 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Not qualified: Qualification failed on public task single-01: after a correct get_price call and its mock result, the model's next token was <|tool_response> (an end-of-generation id), so the reply was empty. With thinking off, neither the maker's chat template nor vLLM's example template adds a continuation cue after a tool result. One observation; cause not proven.
No score: the setup did not complete the qualification tool round trip, so its full run was not admitted.
Qualification evidence sha256: 11b2effa6e9f1f57d2fd1a76001d65b3f2a5f497ee815666bc2a8b6e8d41a18a
Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
42/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 18, "extra_call": 6, "gave_up_or_loop": 4, "wrong_args": 73, "wrong_tool": 1}
Tokens: 1340110
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 22 s · $0.0156 for the run, $0.000108 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
93/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 25, "extra_call": 5, "wrong_args": 21}
Tokens: 1790981
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 51 s · $0.0355 for the run, $0.000246 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9534 standard set; #10411 hard set, identical argv) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4
106/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 11, "wrong_args": 27}
Tokens: 2048615
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 106 s · $0.0734 for the run, $0.000510 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
42/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 18, "extra_call": 7, "gave_up_or_loop": 6, "wrong_args": 71}
Tokens: 1586690
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 24 s · $0.0165 for the run, $0.000114 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
120/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 6, "wrong_args": 18}
Tokens: 2288473
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 92 s · $0.0637 for the run, $0.000442 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
83/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 3, "gave_up_or_loop": 8, "made_up_result": 1, "wrong_args": 47, "wrong_tool": 2}
Tokens: 2473838
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 66 s · $0.0455 for the run, $0.000316 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
98/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 5, "wrong_args": 40, "wrong_tool": 1}
Tokens: 2603860
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 112 s · $0.0774 for the run, $0.000537 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
129/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 1, "wrong_args": 14}
Tokens: 2194487
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 76 s · $0.0528 for the run, $0.000367 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
116/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 3, "wrong_args": 25}
Tokens: 2457570
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 150 s · $0.1035 for the run, $0.000719 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
36/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 1, "extra_call": 15, "gave_up_or_loop": 6, "wrong_args": 81, "wrong_tool": 5}
Tokens: 2355910
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 94 s · $0.0649 for the run, $0.000451 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
102/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 12, "wrong_args": 30}
Tokens: 2137047
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 275 s · $0.1899 for the run, $0.001319 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the standard run; #10423 GPU 1 for the hard set; identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4
Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.
111/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 1, "made_up_result": 5, "wrong_args": 27}
Tokens: 1836873
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 68 s · $0.0469 for the run, $0.000326 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
58/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 12, "gave_up_or_loop": 5, "wrong_args": 68, "wrong_tool": 1}
Tokens: 2601502
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 267 s · $0.1847 for the run, $0.001283 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
127/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 4, "no_call": 1, "wrong_args": 12}
Tokens: 4304813
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 724 s · $0.5010 for the run, $0.003479 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
61/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 14, "wrong_args": 68, "wrong_tool": 1}
Tokens: 2353976
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 290 s · $0.2003 for the run, $0.001391 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
Run note: Not tiered: one of its 144 task runs is unscored. On a long-output task the server refused the request because the prompt plus the fixed 32,768-token answer budget exceeded its 69,632-token context (HTTP 400). Serving and configuration errors are not counted as model failures; the score is over the scored runs.
131/143 scored passed; 1 unscored; 144 attempted
Failure tags: {"wrong_args": 12}
Tokens: Not reported
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 1214 s · $0.8400 for the run, $0.005874 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
103/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 7, "wrong_args": 34}
Tokens: 1798075
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 83 s · $0.0573 for the run, $0.000398 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
137/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 1, "wrong_args": 6}
Tokens: 2017522
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 229 s · $0.1585 for the run, $0.001100 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10561 hard set on GPU 1, identical argv) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183
Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2
130/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 8, "extra_call": 1, "wrong_args": 5}
Tokens: 2110323
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 201 s · $0.1390 for the run, $0.000965 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
107/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 1, "extra_call": 6, "wrong_args": 30}
Tokens: 2144354
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 242 s · $0.1672 for the run, $0.001161 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
127/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 9, "extra_call": 3, "wrong_args": 5}
Tokens: 2204967
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 516 s · $0.3566 for the run, $0.002477 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
93/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 20, "extra_call": 6, "wrong_args": 25}
Tokens: 1565256
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 281 s · $0.1943 for the run, $0.001349 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
131/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 13}
Tokens: 1830591
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 608 s · $0.4207 for the run, $0.002921 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#9534) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4
118/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"gave_up_or_loop": 5, "wrong_args": 7, "wrong_tool": 3}
Tokens: 1034078
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 71 s · $0.0492 for the run, $0.000195 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
116/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"extra_call": 1, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 11, "wrong_tool": 7}
Tokens: 1026831
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
116/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"no_call": 1, "wrong_args": 6, "wrong_tool": 11}
Tokens: 1560191
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 157 s · $0.1089 for the run, $0.000432 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
115/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"extra_call": 10, "gave_up_or_loop": 5, "wrong_args": 5, "wrong_tool": 1}
Tokens: 888087
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 23 s · $0.0162 for the run, $0.000064 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the run, identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4
Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.
114/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"extra_call": 4, "gave_up_or_loop": 1, "wrong_args": 22}
Tokens: 797158
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
113/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"extra_call": 7, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 13, "wrong_tool": 9}
Tokens: 1267484
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 59 s · $0.0411 for the run, $0.000163 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
112/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"extra_call": 5, "wrong_args": 15, "wrong_tool": 6}
Tokens: 1138207
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 78 s · $0.0539 for the run, $0.000214 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
111/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"extra_call": 4, "wrong_args": 18, "wrong_tool": 2}
Tokens: 1133799
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 57 s · $0.0397 for the run, $0.000158 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
111/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 15, "wrong_tool": 4}
Tokens: 1023064
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 87 s · $0.0600 for the run, $0.000238 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
109/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"bad_format": 5, "extra_call": 7, "gave_up_or_loop": 2, "wrong_args": 19, "wrong_tool": 6}
Tokens: 773212
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 10 s · $0.0066 for the run, $0.000026 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
99/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"gave_up_or_loop": 5, "made_up_result": 1, "wrong_args": 39, "wrong_tool": 7}
Tokens: 1058883
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 36 s · $0.0249 for the run, $0.000099 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
98/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"bad_format": 2, "extra_call": 7, "gave_up_or_loop": 14, "wrong_args": 21, "wrong_tool": 3}
Tokens: 894011
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 12 s · $0.0085 for the run, $0.000034 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
97/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"bad_format": 3, "extra_call": 22, "wrong_args": 20, "wrong_tool": 6}
Tokens: 1179082
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 53 s · $0.0369 for the run, $0.000146 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
93/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"extra_call": 14, "gave_up_or_loop": 11, "made_up_result": 2, "wrong_args": 33, "wrong_tool": 7}
Tokens: 1173418
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 26 s · $0.0180 for the run, $0.000071 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Not qualified: Qualification failed on public task single-01: after a correct get_price call and its mock result, the model's next token was <|tool_response> (an end-of-generation id), so the reply was empty. With thinking off, neither the maker's chat template nor vLLM's example template adds a continuation cue after a tool result. One observation; cause not proven.
No score: the setup did not complete the qualification tool round trip, so its full run was not admitted.
Qualification evidence sha256: 11b2effa6e9f1f57d2fd1a76001d65b3f2a5f497ee815666bc2a8b6e8d41a18a
Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
119/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"gave_up_or_loop": 2, "wrong_tool": 12}
Tokens: 728093
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 38 s · $0.0266 for the run, $0.000105 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
117/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"gave_up_or_loop": 2, "wrong_args": 8, "wrong_tool": 7}
Tokens: 1580185
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 332 s · $0.2294 for the run, $0.000910 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
117/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"extra_call": 2, "wrong_args": 4, "wrong_tool": 12}
Tokens: 870626
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 121 s · $0.0836 for the run, $0.000332 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
107/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"extra_call": 3, "gave_up_or_loop": 3, "wrong_args": 21, "wrong_tool": 14}
Tokens: 1026531
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 82 s · $0.0564 for the run, $0.000224 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.
126/126 scored passed; 0 unscored; 126 attempted
Failure tags: {}
Tokens: 691572
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 86 s · $0.0594 for the run, $0.000236 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.
125/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"bad_format": 2, "wrong_tool": 1}
Tokens: 783015
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 252 s · $0.1740 for the run, $0.000691 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
121/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"extra_call": 5, "wrong_args": 1, "wrong_tool": 2}
Tokens: 948341
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 144 s · $0.0993 for the run, $0.000394 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
117/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"bad_format": 5, "extra_call": 10, "wrong_args": 1, "wrong_tool": 4}
Tokens: 1098479
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 315 s · $0.2177 for the run, $0.000864 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183
Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2
116/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 2, "wrong_tool": 7}
Tokens: 998331
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 135 s · $0.0932 for the run, $0.000370 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md
Provider: local vLLM served by @logic on bariton (#9534) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4
119/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"gave_up_or_loop": 5, "wrong_args": 7, "wrong_tool": 3}
Tokens: 1034078
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 71 s · $0.0492 for the run, $0.000195 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
118/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"no_call": 1, "wrong_args": 6, "wrong_tool": 11}
Tokens: 1560191
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 157 s · $0.1089 for the run, $0.000432 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
117/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"extra_call": 4, "wrong_args": 18, "wrong_tool": 2}
Tokens: 1133799
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 57 s · $0.0397 for the run, $0.000158 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
116/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"extra_call": 10, "gave_up_or_loop": 5, "wrong_args": 5, "wrong_tool": 1}
Tokens: 888087
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 23 s · $0.0162 for the run, $0.000064 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
114/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"extra_call": 1, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 11, "wrong_tool": 7}
Tokens: 1026831
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
114/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"extra_call": 5, "wrong_args": 15, "wrong_tool": 6}
Tokens: 1138207
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 78 s · $0.0539 for the run, $0.000214 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
113/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 15, "wrong_tool": 4}
Tokens: 1023064
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 87 s · $0.0600 for the run, $0.000238 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the run, identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4
Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.
111/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"extra_call": 4, "gave_up_or_loop": 1, "wrong_args": 22}
Tokens: 797158
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
107/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"bad_format": 2, "extra_call": 7, "gave_up_or_loop": 14, "wrong_args": 21, "wrong_tool": 3}
Tokens: 894011
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 12 s · $0.0085 for the run, $0.000034 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
107/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"extra_call": 7, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 13, "wrong_tool": 9}
Tokens: 1267484
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 59 s · $0.0411 for the run, $0.000163 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
104/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"bad_format": 5, "extra_call": 7, "gave_up_or_loop": 2, "wrong_args": 19, "wrong_tool": 6}
Tokens: 773212
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 10 s · $0.0066 for the run, $0.000026 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
104/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"bad_format": 3, "extra_call": 22, "wrong_args": 20, "wrong_tool": 6}
Tokens: 1179082
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 53 s · $0.0369 for the run, $0.000146 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
101/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"gave_up_or_loop": 5, "made_up_result": 1, "wrong_args": 39, "wrong_tool": 7}
Tokens: 1058883
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 36 s · $0.0249 for the run, $0.000099 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
92/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"extra_call": 14, "gave_up_or_loop": 11, "made_up_result": 2, "wrong_args": 33, "wrong_tool": 7}
Tokens: 1173418
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 26 s · $0.0180 for the run, $0.000071 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
119/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"gave_up_or_loop": 2, "wrong_tool": 12}
Tokens: 728093
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 38 s · $0.0266 for the run, $0.000105 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
118/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"gave_up_or_loop": 2, "wrong_args": 8, "wrong_tool": 7}
Tokens: 1580185
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 332 s · $0.2294 for the run, $0.000910 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
117/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"extra_call": 2, "wrong_args": 4, "wrong_tool": 12}
Tokens: 870626
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 121 s · $0.0836 for the run, $0.000332 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
104/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"extra_call": 3, "gave_up_or_loop": 3, "wrong_args": 21, "wrong_tool": 14}
Tokens: 1026531
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 82 s · $0.0564 for the run, $0.000224 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.
126/126 scored passed; 0 unscored; 126 attempted
Failure tags: {}
Tokens: 691572
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 86 s · $0.0594 for the run, $0.000236 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.
124/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"bad_format": 2, "wrong_tool": 1}
Tokens: 783015
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 252 s · $0.1740 for the run, $0.000691 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
123/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"extra_call": 5, "wrong_args": 1, "wrong_tool": 2}
Tokens: 948341
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 144 s · $0.0993 for the run, $0.000394 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183
Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2
118/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 2, "wrong_tool": 7}
Tokens: 998331
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 135 s · $0.0932 for the run, $0.000370 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
115/126 scored passed; 0 unscored; 126 attempted
Failure tags: {"bad_format": 5, "extra_call": 10, "wrong_args": 1, "wrong_tool": 4}
Tokens: 1098479
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 315 s · $0.2177 for the run, $0.000864 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
68/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"bad_format": 1, "wrong_args": 14}
Tokens: 2194487
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 76 s · $0.0528 for the run, $0.000367 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
61/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"extra_call": 4, "no_call": 1, "wrong_args": 12}
Tokens: 4304813
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 724 s · $0.5010 for the run, $0.003479 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
60/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"bad_format": 6, "wrong_args": 18}
Tokens: 2288473
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 92 s · $0.0637 for the run, $0.000442 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
56/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"bad_format": 3, "wrong_args": 25}
Tokens: 2457570
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 150 s · $0.1035 for the run, $0.000719 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the standard run; #10423 GPU 1 for the hard set; identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4
Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.
55/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"extra_call": 1, "made_up_result": 5, "wrong_args": 27}
Tokens: 1836873
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 68 s · $0.0469 for the run, $0.000326 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#9534 standard set; #10411 hard set, identical argv) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4
51/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"bad_format": 11, "wrong_args": 27}
Tokens: 2048615
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 106 s · $0.0734 for the run, $0.000510 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
51/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"extra_call": 12, "wrong_args": 30}
Tokens: 2137047
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 275 s · $0.1899 for the run, $0.001319 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
48/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"extra_call": 5, "wrong_args": 40, "wrong_tool": 1}
Tokens: 2603860
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 112 s · $0.0774 for the run, $0.000537 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
47/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"bad_format": 25, "extra_call": 5, "wrong_args": 21}
Tokens: 1790981
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 51 s · $0.0355 for the run, $0.000246 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
42/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"extra_call": 3, "gave_up_or_loop": 8, "made_up_result": 1, "wrong_args": 47, "wrong_tool": 2}
Tokens: 2473838
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 66 s · $0.0455 for the run, $0.000316 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
28/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"extra_call": 12, "gave_up_or_loop": 5, "wrong_args": 68, "wrong_tool": 1}
Tokens: 2601502
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 267 s · $0.1847 for the run, $0.001283 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
26/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"bad_format": 18, "extra_call": 7, "gave_up_or_loop": 6, "wrong_args": 71}
Tokens: 1586690
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 24 s · $0.0165 for the run, $0.000114 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
24/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"bad_format": 18, "extra_call": 6, "gave_up_or_loop": 4, "wrong_args": 73, "wrong_tool": 1}
Tokens: 1340110
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 22 s · $0.0156 for the run, $0.000108 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
19/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"bad_format": 1, "extra_call": 15, "gave_up_or_loop": 6, "wrong_args": 81, "wrong_tool": 5}
Tokens: 2355910
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 94 s · $0.0649 for the run, $0.000451 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
Run note: Not tiered: one of its 144 task runs is unscored. On a long-output task the server refused the request because the prompt plus the fixed 32,768-token answer budget exceeded its 69,632-token context (HTTP 400). Serving and configuration errors are not counted as model failures; the score is over the scored runs.
68/71 scored passed; 1 unscored; 72 attempted
Failure tags: {"wrong_args": 12}
Tokens: Not reported
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 1214 s · $0.8400 for the run, $0.005874 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
68/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"extra_call": 1, "wrong_args": 6}
Tokens: 2017522
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 229 s · $0.1585 for the run, $0.001100 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
51/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"extra_call": 7, "wrong_args": 34}
Tokens: 1798075
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 83 s · $0.0573 for the run, $0.000398 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
30/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"extra_call": 14, "wrong_args": 68, "wrong_tool": 1}
Tokens: 2353976
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 290 s · $0.2003 for the run, $0.001391 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10561 hard set on GPU 1, identical argv) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183
Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2
66/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"bad_format": 8, "extra_call": 1, "wrong_args": 5}
Tokens: 2110323
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 201 s · $0.1390 for the run, $0.000965 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
65/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"bad_format": 13}
Tokens: 1830591
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 608 s · $0.4207 for the run, $0.002921 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
64/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"bad_format": 9, "extra_call": 3, "wrong_args": 5}
Tokens: 2204967
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 516 s · $0.3566 for the run, $0.002477 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
54/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"bad_format": 1, "extra_call": 6, "wrong_args": 30}
Tokens: 2144354
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 242 s · $0.1672 for the run, $0.001161 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
45/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"bad_format": 20, "extra_call": 6, "wrong_args": 25}
Tokens: 1565256
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 281 s · $0.1943 for the run, $0.001349 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
66/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"extra_call": 4, "no_call": 1, "wrong_args": 12}
Tokens: 4304813
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 724 s · $0.5010 for the run, $0.003479 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
61/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"bad_format": 1, "wrong_args": 14}
Tokens: 2194487
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 76 s · $0.0528 for the run, $0.000367 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
60/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"bad_format": 6, "wrong_args": 18}
Tokens: 2288473
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 92 s · $0.0637 for the run, $0.000442 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
60/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"bad_format": 3, "wrong_args": 25}
Tokens: 2457570
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 150 s · $0.1035 for the run, $0.000719 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the standard run; #10423 GPU 1 for the hard set; identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4
Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.
56/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"extra_call": 1, "made_up_result": 5, "wrong_args": 27}
Tokens: 1836873
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 68 s · $0.0469 for the run, $0.000326 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#9534 standard set; #10411 hard set, identical argv) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4
55/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"bad_format": 11, "wrong_args": 27}
Tokens: 2048615
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 106 s · $0.0734 for the run, $0.000510 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
51/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"extra_call": 12, "wrong_args": 30}
Tokens: 2137047
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 275 s · $0.1899 for the run, $0.001319 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
50/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"extra_call": 5, "wrong_args": 40, "wrong_tool": 1}
Tokens: 2603860
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 112 s · $0.0774 for the run, $0.000537 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
46/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"bad_format": 25, "extra_call": 5, "wrong_args": 21}
Tokens: 1790981
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 51 s · $0.0355 for the run, $0.000246 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
41/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"extra_call": 3, "gave_up_or_loop": 8, "made_up_result": 1, "wrong_args": 47, "wrong_tool": 2}
Tokens: 2473838
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 66 s · $0.0455 for the run, $0.000316 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
30/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"extra_call": 12, "gave_up_or_loop": 5, "wrong_args": 68, "wrong_tool": 1}
Tokens: 2601502
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 267 s · $0.1847 for the run, $0.001283 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
18/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"bad_format": 18, "extra_call": 6, "gave_up_or_loop": 4, "wrong_args": 73, "wrong_tool": 1}
Tokens: 1340110
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 22 s · $0.0156 for the run, $0.000108 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
17/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"bad_format": 1, "extra_call": 15, "gave_up_or_loop": 6, "wrong_args": 81, "wrong_tool": 5}
Tokens: 2355910
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 94 s · $0.0649 for the run, $0.000451 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
16/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"bad_format": 18, "extra_call": 7, "gave_up_or_loop": 6, "wrong_args": 71}
Tokens: 1586690
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 24 s · $0.0165 for the run, $0.000114 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
69/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"extra_call": 1, "wrong_args": 6}
Tokens: 2017522
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 229 s · $0.1585 for the run, $0.001100 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
Run note: Not tiered: one of its 144 task runs is unscored. On a long-output task the server refused the request because the prompt plus the fixed 32,768-token answer budget exceeded its 69,632-token context (HTTP 400). Serving and configuration errors are not counted as model failures; the score is over the scored runs.
63/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"wrong_args": 12}
Tokens: Not reported
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 1214 s · $0.8400 for the run, $0.005874 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
52/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"extra_call": 7, "wrong_args": 34}
Tokens: 1798075
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 83 s · $0.0573 for the run, $0.000398 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
31/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"extra_call": 14, "wrong_args": 68, "wrong_tool": 1}
Tokens: 2353976
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 290 s · $0.2003 for the run, $0.001391 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
66/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"bad_format": 13}
Tokens: 1830591
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 608 s · $0.4207 for the run, $0.002921 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10561 hard set on GPU 1, identical argv) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183
Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2
64/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"bad_format": 8, "extra_call": 1, "wrong_args": 5}
Tokens: 2110323
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 201 s · $0.1390 for the run, $0.000965 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
63/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"bad_format": 9, "extra_call": 3, "wrong_args": 5}
Tokens: 2204967
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 516 s · $0.3566 for the run, $0.002477 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
53/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"bad_format": 1, "extra_call": 6, "wrong_args": 30}
Tokens: 2144354
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 242 s · $0.1672 for the run, $0.001161 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
48/72 scored passed; 0 unscored; 72 attempted
Failure tags: {"bad_format": 20, "extra_call": 6, "wrong_args": 25}
Tokens: 1565256
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 281 s · $0.1943 for the run, $0.001349 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
213/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"bad_format": 5, "extra_call": 7, "gave_up_or_loop": 2, "wrong_args": 19, "wrong_tool": 6}
Tokens: 773212
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 10 s · $0.0066 for the run, $0.000026 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
231/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 10, "gave_up_or_loop": 5, "wrong_args": 5, "wrong_tool": 1}
Tokens: 888087
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 23 s · $0.0162 for the run, $0.000064 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9534) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4
237/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"gave_up_or_loop": 5, "wrong_args": 7, "wrong_tool": 3}
Tokens: 1034078
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 71 s · $0.0492 for the run, $0.000195 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
205/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"bad_format": 2, "extra_call": 7, "gave_up_or_loop": 14, "wrong_args": 21, "wrong_tool": 3}
Tokens: 894011
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 12 s · $0.0085 for the run, $0.000034 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
228/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 4, "wrong_args": 18, "wrong_tool": 2}
Tokens: 1133799
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 57 s · $0.0397 for the run, $0.000158 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
185/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 14, "gave_up_or_loop": 11, "made_up_result": 2, "wrong_args": 33, "wrong_tool": 7}
Tokens: 1173418
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 26 s · $0.0180 for the run, $0.000071 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
220/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 7, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 13, "wrong_tool": 9}
Tokens: 1267484
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 59 s · $0.0411 for the run, $0.000163 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
230/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 1, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 11, "wrong_tool": 7}
Tokens: 1026831
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
226/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 5, "wrong_args": 15, "wrong_tool": 6}
Tokens: 1138207
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 78 s · $0.0539 for the run, $0.000214 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
201/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"bad_format": 3, "extra_call": 22, "wrong_args": 20, "wrong_tool": 6}
Tokens: 1179082
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 53 s · $0.0369 for the run, $0.000146 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
224/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 15, "wrong_tool": 4}
Tokens: 1023064
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 87 s · $0.0600 for the run, $0.000238 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the run, identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4
Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.
225/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 4, "gave_up_or_loop": 1, "wrong_args": 22}
Tokens: 797158
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
200/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"gave_up_or_loop": 5, "made_up_result": 1, "wrong_args": 39, "wrong_tool": 7}
Tokens: 1058883
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 36 s · $0.0249 for the run, $0.000099 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
234/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"no_call": 1, "wrong_args": 6, "wrong_tool": 11}
Tokens: 1560191
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 157 s · $0.1089 for the run, $0.000432 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
211/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 3, "gave_up_or_loop": 3, "wrong_args": 21, "wrong_tool": 14}
Tokens: 1026531
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 82 s · $0.0564 for the run, $0.000224 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
235/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"gave_up_or_loop": 2, "wrong_args": 8, "wrong_tool": 7}
Tokens: 1580185
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 332 s · $0.2294 for the run, $0.000910 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
238/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"gave_up_or_loop": 2, "wrong_tool": 12}
Tokens: 728093
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 38 s · $0.0266 for the run, $0.000105 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
234/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 2, "wrong_args": 4, "wrong_tool": 12}
Tokens: 870626
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 121 s · $0.0836 for the run, $0.000332 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183
Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2
234/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 2, "wrong_tool": 7}
Tokens: 998331
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 135 s · $0.0932 for the run, $0.000370 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
244/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 5, "wrong_args": 1, "wrong_tool": 2}
Tokens: 948341
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 144 s · $0.0993 for the run, $0.000394 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
232/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"bad_format": 5, "extra_call": 10, "wrong_args": 1, "wrong_tool": 4}
Tokens: 1098479
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 315 s · $0.2177 for the run, $0.000864 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.
252/252 scored passed; 0 unscored; 252 attempted
Failure tags: {}
Tokens: 691572
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 86 s · $0.0594 for the run, $0.000236 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.
249/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"bad_format": 2, "wrong_tool": 1}
Tokens: 783015
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 252 s · $0.1740 for the run, $0.000691 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Not qualified: Qualification failed on public task single-01: after a correct get_price call and its mock result, the model's next token was <|tool_response> (an end-of-generation id), so the reply was empty. With thinking off, neither the maker's chat template nor vLLM's example template adds a continuation cue after a tool result. One observation; cause not proven.
No score: the setup did not complete the qualification tool round trip, so its full run was not admitted.
Qualification evidence sha256: 11b2effa6e9f1f57d2fd1a76001d65b3f2a5f497ee815666bc2a8b6e8d41a18a
Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
42/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 18, "extra_call": 6, "gave_up_or_loop": 4, "wrong_args": 73, "wrong_tool": 1}
Tokens: 1340110
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 22 s · $0.0156 for the run, $0.000108 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
93/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 25, "extra_call": 5, "wrong_args": 21}
Tokens: 1790981
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 51 s · $0.0355 for the run, $0.000246 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9534 standard set; #10411 hard set, identical argv) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4
106/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 11, "wrong_args": 27}
Tokens: 2048615
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 106 s · $0.0734 for the run, $0.000510 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
42/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 18, "extra_call": 7, "gave_up_or_loop": 6, "wrong_args": 71}
Tokens: 1586690
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 24 s · $0.0165 for the run, $0.000114 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
120/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 6, "wrong_args": 18}
Tokens: 2288473
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 92 s · $0.0637 for the run, $0.000442 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
83/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 3, "gave_up_or_loop": 8, "made_up_result": 1, "wrong_args": 47, "wrong_tool": 2}
Tokens: 2473838
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 66 s · $0.0455 for the run, $0.000316 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
98/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 5, "wrong_args": 40, "wrong_tool": 1}
Tokens: 2603860
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 112 s · $0.0774 for the run, $0.000537 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
129/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 1, "wrong_args": 14}
Tokens: 2194487
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 76 s · $0.0528 for the run, $0.000367 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
116/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 3, "wrong_args": 25}
Tokens: 2457570
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 150 s · $0.1035 for the run, $0.000719 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
36/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 1, "extra_call": 15, "gave_up_or_loop": 6, "wrong_args": 81, "wrong_tool": 5}
Tokens: 2355910
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 94 s · $0.0649 for the run, $0.000451 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
102/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 12, "wrong_args": 30}
Tokens: 2137047
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 275 s · $0.1899 for the run, $0.001319 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the standard run; #10423 GPU 1 for the hard set; identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4
Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.
111/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 1, "made_up_result": 5, "wrong_args": 27}
Tokens: 1836873
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 68 s · $0.0469 for the run, $0.000326 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
58/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 12, "gave_up_or_loop": 5, "wrong_args": 68, "wrong_tool": 1}
Tokens: 2601502
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 267 s · $0.1847 for the run, $0.001283 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
127/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 4, "no_call": 1, "wrong_args": 12}
Tokens: 4304813
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 724 s · $0.5010 for the run, $0.003479 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
61/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 14, "wrong_args": 68, "wrong_tool": 1}
Tokens: 2353976
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 290 s · $0.2003 for the run, $0.001391 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
Run note: Not tiered: one of its 144 task runs is unscored. On a long-output task the server refused the request because the prompt plus the fixed 32,768-token answer budget exceeded its 69,632-token context (HTTP 400). Serving and configuration errors are not counted as model failures; the score is over the scored runs.
131/143 scored passed; 1 unscored; 144 attempted
Failure tags: {"wrong_args": 12}
Tokens: Not reported
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 1214 s · $0.8400 for the run, $0.005874 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
103/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 7, "wrong_args": 34}
Tokens: 1798075
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 83 s · $0.0573 for the run, $0.000398 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
137/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 1, "wrong_args": 6}
Tokens: 2017522
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 229 s · $0.1585 for the run, $0.001100 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10561 hard set on GPU 1, identical argv) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183
Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2
130/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 8, "extra_call": 1, "wrong_args": 5}
Tokens: 2110323
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 201 s · $0.1390 for the run, $0.000965 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
107/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 1, "extra_call": 6, "wrong_args": 30}
Tokens: 2144354
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 242 s · $0.1672 for the run, $0.001161 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
127/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 9, "extra_call": 3, "wrong_args": 5}
Tokens: 2204967
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 516 s · $0.3566 for the run, $0.002477 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
93/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 20, "extra_call": 6, "wrong_args": 25}
Tokens: 1565256
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 281 s · $0.1943 for the run, $0.001349 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
131/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 13}
Tokens: 1830591
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 608 s · $0.4207 for the run, $0.002921 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
213/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"bad_format": 5, "extra_call": 7, "gave_up_or_loop": 2, "wrong_args": 19, "wrong_tool": 6}
Tokens: 773212
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 10 s · $0.0066 for the run, $0.000026 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9528) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
231/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 10, "gave_up_or_loop": 5, "wrong_args": 5, "wrong_tool": 1}
Tokens: 888087
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 23 s · $0.0162 for the run, $0.000064 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9534) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4
237/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"gave_up_or_loop": 5, "wrong_args": 7, "wrong_tool": 3}
Tokens: 1034078
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 71 s · $0.0492 for the run, $0.000195 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
205/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"bad_format": 2, "extra_call": 7, "gave_up_or_loop": 14, "wrong_args": 21, "wrong_tool": 3}
Tokens: 894011
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 12 s · $0.0085 for the run, $0.000034 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9641) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
228/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 4, "wrong_args": 18, "wrong_tool": 2}
Tokens: 1133799
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 57 s · $0.0397 for the run, $0.000158 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
185/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 14, "gave_up_or_loop": 11, "made_up_result": 2, "wrong_args": 33, "wrong_tool": 7}
Tokens: 1173418
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 26 s · $0.0180 for the run, $0.000071 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9668) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
220/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 7, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 13, "wrong_tool": 9}
Tokens: 1267484
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 59 s · $0.0411 for the run, $0.000163 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
230/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 1, "gave_up_or_loop": 2, "made_up_result": 1, "wrong_args": 11, "wrong_tool": 7}
Tokens: 1026831
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#9689) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
226/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 5, "wrong_args": 15, "wrong_tool": 6}
Tokens: 1138207
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 78 s · $0.0539 for the run, $0.000214 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
201/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"bad_format": 3, "extra_call": 22, "wrong_args": 20, "wrong_tool": 6}
Tokens: 1179082
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 53 s · $0.0369 for the run, $0.000146 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
224/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 15, "wrong_tool": 4}
Tokens: 1023064
GPU: 1 × bariton GPU 0 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 87 s · $0.0600 for the run, $0.000238 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the run, identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4
Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.
225/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 4, "gave_up_or_loop": 1, "wrong_args": 22}
Tokens: 797158
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 31 s · $0.0216 for the run, $0.000086 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
200/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"gave_up_or_loop": 5, "made_up_result": 1, "wrong_args": 39, "wrong_tool": 7}
Tokens: 1058883
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 36 s · $0.0249 for the run, $0.000099 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10256) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
234/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"no_call": 1, "wrong_args": 6, "wrong_tool": 11}
Tokens: 1560191
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 157 s · $0.1089 for the run, $0.000432 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
211/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 3, "gave_up_or_loop": 3, "wrong_args": 21, "wrong_tool": 14}
Tokens: 1026531
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 82 s · $0.0564 for the run, $0.000224 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10281) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
235/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"gave_up_or_loop": 2, "wrong_args": 8, "wrong_tool": 7}
Tokens: 1580185
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 332 s · $0.2294 for the run, $0.000910 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
238/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"gave_up_or_loop": 2, "wrong_tool": 12}
Tokens: 728093
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 38 s · $0.0266 for the run, $0.000105 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10304) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
234/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 2, "wrong_args": 4, "wrong_tool": 12}
Tokens: 870626
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 121 s · $0.0836 for the run, $0.000332 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183
Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2
234/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 8, "gave_up_or_loop": 1, "wrong_args": 2, "wrong_tool": 7}
Tokens: 998331
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 135 s · $0.0932 for the run, $0.000370 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
244/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"extra_call": 5, "wrong_args": 1, "wrong_tool": 2}
Tokens: 948341
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 144 s · $0.0993 for the run, $0.000394 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
232/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"bad_format": 5, "extra_call": 10, "wrong_args": 1, "wrong_tool": 4}
Tokens: 1098479
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 315 s · $0.2177 for the run, $0.000864 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.
252/252 scored passed; 0 unscored; 252 attempted
Failure tags: {}
Tokens: 691572
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 86 s · $0.0594 for the run, $0.000236 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the runs, argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
Run note: At the ceiling: 98% or more on both tracks (Toolbench's near-ceiling rule), so this task set cannot tell it apart from a perfect score; harder tasks would be needed to rank it further.
249/252 scored passed; 0 unscored; 252 attempted
Failure tags: {"bad_format": 2, "wrong_tool": 1}
Tokens: 783015
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 252 s · $0.1740 for the run, $0.000691 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Not qualified: Qualification failed on public task single-01: after a correct get_price call and its mock result, the model's next token was <|tool_response> (an end-of-generation id), so the reply was empty. With thinking off, neither the maker's chat template nor vLLM's example template adds a continuation cue after a tool result. One observation; cause not proven.
No score: the setup did not complete the qualification tool round trip, so its full run was not admitted.
Qualification evidence sha256: 11b2effa6e9f1f57d2fd1a76001d65b3f2a5f497ee815666bc2a8b6e8d41a18a
Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
42/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 18, "extra_call": 6, "gave_up_or_loop": 4, "wrong_args": 73, "wrong_tool": 1}
Tokens: 1340110
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 22 s · $0.0156 for the run, $0.000108 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9528 standard set; #10380 hard set, identical argv) · revision: 0dc4268b561499c648c7ba7c9466bf3d8d01517a
Template: 0dc4268b561499c648c7ba7c9466bf3d8d01517a · parser: qwen3_xml
93/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 25, "extra_call": 5, "wrong_args": 21}
Tokens: 1790981
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 51 s · $0.0355 for the run, $0.000246 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 1.73B total · class source: https://huggingface.co/LiquidAI/LFM3-1.7B/blob/0dc4268b561499c648c7ba7c9466bf3d8d01517a/README.md
Provider: local vLLM served by @logic on bariton (#9534 standard set; #10411 hard set, identical argv) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 · parser: gemma4
106/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 11, "wrong_args": 27}
Tokens: 2048615
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 106 s · $0.0734 for the run, $0.000510 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
42/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 18, "extra_call": 7, "gave_up_or_loop": 6, "wrong_args": 71}
Tokens: 1586690
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 24 s · $0.0165 for the run, $0.000114 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9641 standard set; #10441 hard set, identical argv) · revision: 42433e90dad95123ab87729fd7011e7110629556
Template: 42433e90dad95123ab87729fd7011e7110629556 · parser: minicpm5
120/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 6, "wrong_args": 18}
Tokens: 2288473
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 92 s · $0.0637 for the run, $0.000442 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2.52B total · class source: https://huggingface.co/openbmb/MiniCPM5-2B/blob/42433e90dad95123ab87729fd7011e7110629556/README.md
Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
83/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 3, "gave_up_or_loop": 8, "made_up_result": 1, "wrong_args": 47, "wrong_tool": 2}
Tokens: 2473838
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 66 s · $0.0455 for the run, $0.000316 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9668 standard set; #10459 hard set, identical argv) · revision: 15852e8c16360a2fea060d615a32b45270f8a8fc
Template: 15852e8c16360a2fea060d615a32b45270f8a8fc · parser: qwen3_coder
98/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 5, "wrong_args": 40, "wrong_tool": 1}
Tokens: 2603860
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 112 s · $0.0774 for the run, $0.000537 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 2B total · class source: https://huggingface.co/Qwen/Qwen3.5-2B/blob/15852e8c16360a2fea060d615a32b45270f8a8fc/README.md
Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
129/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 1, "wrong_args": 14}
Tokens: 2194487
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 76 s · $0.0528 for the run, $0.000367 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local vLLM served by @logic on bariton (#9689 standard set; #10468 hard set, identical argv) · revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Template: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a · parser: qwen3_coder
116/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 3, "wrong_args": 25}
Tokens: 2457570
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 150 s · $0.1035 for the run, $0.000719 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 4.66B total · class source: https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
36/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 1, "extra_call": 15, "gave_up_or_loop": 6, "wrong_args": 81, "wrong_tool": 5}
Tokens: 2355910
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 94 s · $0.0649 for the run, $0.000451 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local KRAFTON vLLM fork served by @logic on bariton (#9725 standard set; #10474 hard set, identical command line) · revision: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee
Template: fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee · parser: raon_1
102/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 12, "wrong_args": 30}
Tokens: 2137047
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 275 s · $0.1899 for the run, $0.001319 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3.1B total · class source: https://huggingface.co/KRAFTON/Raon-1-nano/blob/fcf2d1d55e84e2f9fbe7b5d724dcfc4d861cfcee/README.md
Provider: local vLLM served by @logic on bariton (#10207 GPU 3 for qualification; #10260 GPU 2 for the standard run; #10423 GPU 1 for the hard set; identical argv/image/template) · revision: 4adf15748de9e1a0097b6274cee9c15836c41c37
Template: 4adf15748de9e1a0097b6274cee9c15836c41c37 + evidence/gemma-off-cue-10181/chat_template-cue.jinja (non-maker: upstream gemma-4-31B-it ad62905 thinking-off cue after a tool response) · parser: gemma4
Run note: Non-maker template (Ümit #10181): the maker's chat template plus the two-line fix proposed in https://github.com/google-deepmind/gemma/issues/829 (also added by Google to gemma-4-31B-it at commit ad62905): an empty thought '<|channel>thought\n<channel|>' after a tool result with thinking off (template sha256 84d5a64c…). Thinking is off by request, but the parser classed 58% of output tokens as reasoning (275 of 580 replies). The maker-template thinking-off setup stays 'Not qualified'.
111/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 1, "made_up_result": 5, "wrong_args": 27}
Tokens: 1836873
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 68 s · $0.0469 for the run, $0.000326 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 5.1B total · class source: https://huggingface.co/google/gemma-4-E2B-it/blob/4adf15748de9e1a0097b6274cee9c15836c41c37/README.md
Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
58/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 12, "gave_up_or_loop": 5, "wrong_args": 68, "wrong_tool": 1}
Tokens: 2601502
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 267 s · $0.1847 for the run, $0.001283 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10256 standard set; #10380 hard set, identical argv) · revision: 4e9c275e232f10211ec88644d106a1f6a1b60e06
Template: 4e9c275e232f10211ec88644d106a1f6a1b60e06 · parser: qwen3_coder
127/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 4, "no_call": 1, "wrong_args": 12}
Tokens: 4304813
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 32) · GPU time 724 s · $0.5010 for the run, $0.003479 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 3B total · class source: https://huggingface.co/ibm-granite/granite-4.2-3b/blob/4e9c275e232f10211ec88644d106a1f6a1b60e06/README.md
Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
61/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 14, "wrong_args": 68, "wrong_tool": 1}
Tokens: 2353976
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 290 s · $0.2003 for the run, $0.001391 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10281 standard set; #10458 hard set, identical argv) · revision: a7042144125ce5ae64f89610e8289647fc000b9a
Template: a7042144125ce5ae64f89610e8289647fc000b9a · parser: qwen3_coder
Run note: Not tiered: one of its 144 task runs is unscored. On a long-output task the server refused the request because the prompt plus the fixed 32,768-token answer budget exceeded its 69,632-token context (HTTP 400). Serving and configuration errors are not counted as model failures; the score is over the scored runs.
131/143 scored passed; 1 unscored; 144 attempted
Failure tags: {"wrong_args": 12}
Tokens: Not reported
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 1214 s · $0.8400 for the run, $0.005874 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 8B total · class source: https://huggingface.co/ibm-granite/granite-4.2-8b/blob/a7042144125ce5ae64f89610e8289647fc000b9a/README.md
Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
103/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 7, "wrong_args": 34}
Tokens: 1798075
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 83 s · $0.0573 for the run, $0.000398 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10304 standard set; #10528 hard set, identical argv) · revision: 43c9d014b231474fa7212dbb48a762566c2b255d
Template: 43c9d014b231474fa7212dbb48a762566c2b255d · parser: gemma4
137/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"extra_call": 1, "wrong_args": 6}
Tokens: 2017522
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 229 s · $0.1585 for the run, $0.001100 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 11.95B total · class source: https://huggingface.co/google/gemma-4-12B-it/blob/43c9d014b231474fa7212dbb48a762566c2b255d/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10561 hard set on GPU 1, identical argv) · revision: 1769516ac588c4c558089c94f27d96b0dc67b183
Template: 1769516ac588c4c558089c94f27d96b0dc67b183 · parser: lfm2
130/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 8, "extra_call": 1, "wrong_args": 5}
Tokens: 2110323
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 201 s · $0.1390 for the run, $0.000965 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 24B total · class source: https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/1769516ac588c4c558089c94f27d96b0dc67b183/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
107/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 1, "extra_call": 6, "wrong_args": 30}
Tokens: 2144354
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 242 s · $0.1672 for the run, $0.001161 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154 standard set; #10522 hard set on GPU 1, identical argv) · revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Template: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 · parser: qwen3_coder
127/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 9, "extra_call": 3, "wrong_args": 5}
Tokens: 2204967
GPU: 1 × bariton GPU 1 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 516 s · $0.3566 for the run, $0.002477 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 27B total · class source: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
93/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 20, "extra_call": 6, "wrong_args": 25}
Tokens: 1565256
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 281 s · $0.1943 for the run, $0.001349 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Provider: local vLLM served by @logic on bariton (#10154 GPU 0 for qualification; #10267 GPU 2 for the standard runs; #10577 GPU 2 for the hard set; argv/image/template identical) · revision: 842da3794eaa0b77d5f08bae87a17459d91ff475
Template: 842da3794eaa0b77d5f08bae87a17459d91ff475 · parser: gemma4
131/144 scored passed; 0 unscored; 144 attempted
Failure tags: {"bad_format": 13}
Tokens: 1830591
GPU: 1 × bariton GPU 2 (RTX PRO 6000, 96 GB), 16 requests in flight (server cap 16) · GPU time 608 s · $0.4207 for the run, $0.002921 per scored task run (modeled GPU-rental cost, not a paid invoice: GPU-seconds × GPUs × $2.49 per GPU-hour ÷ scored task runs, as on the Logic page. GPU-seconds run from the first request reaching the server to the end of the last answer (server timestamps in whole seconds, so ±1 s); no pauses; the card ran nothing else meanwhile)
Parameters: 30.7B total · class source: https://huggingface.co/google/gemma-4-31B-it/blob/842da3794eaa0b77d5f08bae87a17459d91ff475/README.md
Methodology
All tool observations use fixed synthetic finance fixtures, never real trades. Model answers are measured independently; tool-calling scores in model cards are maker-reported.
Original and renamed tracks are paired. Three repetitions per task; serving/configuration and verified template failures remain unscored. Public and private scores are separate; only public per-task details leave local storage.
Size class by total parameters, the same figures as the suite's other benches (from the maker's card, checked against the exact weights run): Tiny under 6B, Small 6B–15B, Medium over 15B up to 40B, Large over 40B.
No chance correction is applied. Tiers within each size class: for each pair of setups, every task (42 per track, public and private) is one paired item, won by the setup that passed more of its 3 repetitions; ties drop out. Exact two-sided sign test, Holm-corrected over all pairs of the class, p < 0.05. Rows in accuracy order: the top row starts Tier 1, and a row starts the next tier when the test separates it from its tier's top row. Accuracy: both tracks (84 tasks); Renamed tools: each track (42 tasks). – : not tiered (class not verified, incomplete or not qualified). Every pair's wins, losses, ties and p-values. Hard set: the same test over its 24 tasks per track; its pairs. Unknown cost and tokens stay unreported.