Can devstral-cuda run a multi-turn, tool-calling agent loop on this hardware, and how does it compare to the rest of the shelf?
Zero tool-call parse failures in about 4,000 model calls over 375 completed tasks, across four architectures (mistral3, gpt-oss, qwen35, g9v3). llama.cpp's --jinja parsers handled every format. Every failure was reasoning or policy behaviour.
A model that asks a question never gets an answer and scores zero. Half of gpt-oss's stalls never found the order they were meant to act on. With those removed, the stall problem can at most tie gpt-oss with devstral (57.4% each), not put it ahead.
The retail policy requires an explicit “yes” before any write. The scripted customer never says it: 0 of 560 writes were confirmed. Devstral repeated identical writes 24 times across 14 tasks.
| Model | n | Pass | Stall: lookup | Stall: asked | Ceiling | Unterm. | Err | Parse fail | Hours |
|---|---|---|---|---|---|---|---|---|---|
| qwen3.6-27b-abliteratedq5_k_m | 115 | 76 · 66.1% | 2 | 6 | 71.3% | 9 | 8 | 0 | 7.80 |
| g9v3-39a5bq4_k_m · partial | 30 | 17 · 56.7% | 0 | 9 | 86.7% | 0 | 0 | 0 | 1.09 |
| devstral-patchedq8_0 | 115 | 61 · 53.0% | 0 | 5 | 57.4% | 3 | 0 | 0 | 3.45 |
| gpt-oss-20bq8_0 | 115 | 52 · 45.2% | 16 | 14 | 57.4% | 0 | 0 | 0 | 1.52 |
gpt-oss and qwen3.6 each had one more stall that found the order and stopped silently.
Each bar is one model's tasks, as a share of the tasks it ran. The blue and amber segments together make up its ceiling.
Real τ-bench drives the customer with a frontier LLM. There is no metered API key on this box, and 24–28 GB of agent weights leave no VRAM for a second local model, so the customer is scripted. The full instruction arrives on turn 1, and every later turn is a fixed nudge that adds nothing. These numbers are not comparable to published τ-bench results. The harness is easier for some models than for others.
A scripted customer cannot answer a question. A model that asks “Option A or B? Confirm the payment method?” never gets an answer and scores zero. But not every stall is a model waiting on the customer, so each one is checked against the ground truth: did the agent call get_order_details on any order the correct writes touch?
get_user_details, which lists every order, and asked the customer for an order ID it could have looked up. Tasks 6, 58 and 112 stop after a single lookup call. Only 14 stalls found the right order and then asked, and some of those were legitimate by policy: a cancellation reason the task withholds until asked (114), or a second payment method when a gift card falls short (49).| devstral | gpt-oss | qwen3.6 | g9v3 (30) | |
|---|---|---|---|---|
| Stall: found order, asked (harness) | 5 | 14 | 6 | 9 |
| Stall: never opened target order | 0 | 16 | 2 | 0 |
| Stall: found order, stopped silently | 0 | 1 | 1 | 0 |
| Acted, wrong arguments | 36 | 22 | 19 | 3 |
| Acted, required output missed | 10 | 10 | 2 | 1 |
| Unterminated | 3 | 0 | 1 | 0 |
| Error (TimeoutError) | 0 | 0 | 8 | 0 |
| Total failures | 54 | 63 | 39 | 13 |
| Harness-attributable share | 9.3% | 22.2% | 15.4% | 69.2% |
The “asked” test is run.py's ASKING regex, which matches any final message ending in a question. It is generous, so the ceilings are upper bounds twice over. Stall totals match the earlier draft; only the split changed.
get_user_details would have returned. A control plane can supply a “yes”; it cannot supply an order ID the model didn't think to look up. That weakens the case for gpt-oss as a gated planner without ruling it out.g9v3's 69.2% harness share is the largest here. On 30 tasks, its 86.7% ceiling is the least reliable number in the report.
The retail system prompt says:
Before taking consequential actions that update the database (cancel, modify, return, exchange), you have to list the action detail and obtain explicit user confirmation (yes) to proceed.
Neither customer policy ever says yes. The turn-1 restatement ends “Please go ahead and complete it”, a blanket go-ahead given before the agent has listed any action. That doesn't satisfy the policy as written.
| devstral | gpt-oss | qwen3.6 | g9v3 (30) | |
|---|---|---|---|---|
| Writes issued | 250 | 125 | 156 | 29 |
| …without a prior “yes” | 250 | 125 | 156 | 29 |
| Passes containing an unconfirmed write | 56 / 61 | 44 / 52 | 72 / 76 | 13 / 17 |
| Writes beyond the ground truth | 87 | 10 | 16 | 1 |
| Identical writes repeated | 24 in 14 tasks | 0 | 2 in 1 task | 0 |
The remaining passes are tasks that needed no write. Under this harness, no model can pass a write task while following its instructions. A model that lists the action and asks for a yes is complying, and scores zero for it.
This doesn't separate the models, because every write is unconfirmed by design. What it changes is how devstral's low stall rate reads: it takes the blanket go-ahead as enough and doesn't follow the confirmation rule in its own system prompt.
confirm customer policyrun.py --user-policy confirm answers a confirmation request and restates the instruction, instead of the default nudge, which adds nothing. A/B test on devstral, tasks 40–63:
| nudge | confirm | |
|---|---|---|
| Pass | 14 / 24 | 15 / 24 |
| Stalled | 5 | 0 |
| Mean steps | 10.3 | 15.7 |
| Time per task | 98 s | 147 s (1.49×) |
Gained 40, 45, 55 and 58; lost 56, 60 and 62. Net +1 on 24 tasks, and a net loss on tasks 51–63 (7/13 → 6/13): within noise, at 1.5× the time. Stalls went to zero, but an over-action problem replaced them. Task 62's correct behaviour is to change nothing, yet under confirm devstral issued a spurious cancel_pending_order. Rewording the reply to refuse any wider mandate made it worse: two spurious writes instead of one. What drives this is the extra turn itself, not its wording.
nudge stays the default. The real fix is an LLM customer: a canned reply can't tell “yes, do what you asked about” from “no, that's outside my request”. gemma4-e4b (6.0 GB) fits beside devstral (23.9 GB) in 32 GB, with about 2.8 GB left for both KV caches. It's feasible but not built.
available: true. It called get_product_details correctly and then counted wrong: an aggregation error, not a tool error.modify_user_address rewrites. Five of the 14 tasks still passed. A backend without those checks would have executed every repeat. This is the most concerning failure for a control plane, and an argument for idempotency keys on every write tool.new_item_ids.17 tasks were solved by no model. Their required outputs are mostly long numbers (746342064230, 1288.65, 46.66), so reporting exact values is harder here than sequencing tool calls.
| Model | Wall clock | vs devstral |
|---|---|---|
| gpt-oss-20b | 1.52 h | 0.44× |
| devstral | 3.45 h | 1.00× |
| qwen3.6-27b | 7.80 h | 2.26× |
qwen3.6 gains 13 points over devstral for 2.3× the wall clock, 3× the unterminated tasks and 8 timeouts. gpt-oss is 2.3× faster than devstral, ties it at the most generous reading of the stall problem, and never repeated a write.
--jinja, ctx 32768, f16 KV cache, --models-max 1, so one model is loaded at a time.GGML_ASSERT(obj_new) in build_arch_graph), so using f16 everywhere put all four models on one config. g9v3 also needs the fork build at ~/llama-cuda12-g9v3.enable_thinking: false, confirmed by empty reasoning_content and completion tokens halved (98→52, 95→52). gpt-oss reasons through its built-in harmony format and can't be cleanly switched off; devstral doesn't reason.calculate_reward(), unmodified: the ground-truth actions are replayed on a fresh database, the database states are compared, and required output strings are checked. No partial credit.TimeoutError, all scored as failures, so its true rate is somewhat above 66.1%..PARTIAL-superseded-by-unsloth files come from the abandoned uncensored-Q6_K_P run and contain no results.qwen3.8-27b-unsloth_q8_0 was pulled from this run to be scheduled separately; drive_unsloth.sh is staged but hasn't been run.cd ~/Projects/Tests/2026-09-22/devstral-tau-scripted
./drive_all.sh # devstral, gpt-oss, qwen3.6, g9v3 -- resumable
./drive_unsloth.sh # qwen3.8 unsloth q8_0, standalone
../../.venv/bin/python summarise.pyA model is skipped if its JSON already holds 115 results, so an interrupted run resumes at the next model. summarise.py runs on the CPU only and prints every number here except the A/B test.