[self-reply because comment edit window ended]: I now have a head-to-head benchmark. On my wheezy laptop (specs in sibling comment), this new dense model, Qwen 3.8 27B, gets ~4 tokens/second on generation. The older mixture-of-experts model, Qwen 3.6-35B-A3B, gets ~20 tokens/s.

MoE is literally 5x times faster (on CPU) than comparable dense Qwen!

Tested:

Qwen3.6-35B-A3B-UD-Q4_K_XL

Qwen3.8-27B-UD-Q4_K_XL

35A3 might be more comparable to 10 dense.

27 dense is far more capable than 35A3.

So you happen to know how to demonstrate this higher capacity, what do we mean by that? Can understand bigger codebases? And how do we explain it, what part of the model does it?

If you are not already using MTP, you should be able to get ~2x decode tokens/s with Qwen 3.8 27B.

yeah, that's the A3B part; going up to A5B would probably also feel comfortable.

on the 395+ AI MAX w/128GB, the A10B qwen 3.5 can do a lot of long running work if you don't need to baby sit it. deer-flow works well like that.

Got Qwen3.8 to run on my Framework 12 Intel Core 13 Gen Raptor Lake i5-1334U small laptop with 48G RAM stick:

llama serve -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL

but it failed my basic prompt to compose a vim regex to match CamelCaseWords

downgraded a bit with Q4_K_M from ollama run qwen3.8:27b

and /set nothink and at least 1 regex matched FooBar

prompt eval at 2.8 t/s eval at 0.94t/s

I particularly enjoyed this usage of the regex: /%\1\%/ ... onward for 700+ characters of \%\/ :-)

Mind sharing your laptops specs? Just interested to see what is needed to locally run Qwen3.6-35B-A3B

Don't mind! It's 64 GiB dual-channel DDR5-6400, i.e. roughly 100 GiB/s of bandwidth. (AMD 7840U (Zen 4))

I'm using a Q4 quantization from unsloth (Qwen3.6-35B-A3B-UD-Q4_K_XL). It gets up to ~20 tokens/second in generation. I don't know precisely how much KV cache I can safely use, but it's in between 140k–256k. (I.e., 140k reliably works, 256k kernel-crashes from OOM. Don't feel like bisecting).

Inference is llama.cpp with the Vulkan GPU backend on Linux. (I.e., -DGGML_VULKAN=1 on the llama.cpp build, and --gpu-layers all on llama-cli or llama-server. (And for my specific setup, two kernel parameters specific to amdgpu: ttm.pages_limit and ttm.page_pool_size. A driver VRAM limiter. Look it up if you're on amdgpu!)).

Thanks! I don't have any knowledge of running models locally.

I assume it would not be able to handle an unquantized Qwen3.6-35B or is it irrelevant as you almost always would want to run a quantized version of the model on consumer hardware?

not parent, but 4-bit quantization is generally consider a good trade off for speed/performance, so you might use it even when you aren't on consumer hardware, but definitely when you are on consumer hardware.