A 35 billion parameter open-weight model is now running usefully on 12GB graphics cards, and that changes the economics of local AI for small teams.
The interesting part of the latest Qwen discussion is not that another model has landed. It is that a model with 35 billion total parameters is being pushed into the kind of hardware many developers already have under a desk, with enough speed and context to make local coding work feel less like a compromise.
According to a r/LocalLLaMA post that drew roughly 158 points and 38 comments within 11 hours, Qwen3.6-35B-A3B was tested on an RTX 3060 with 12GB of VRAM, 32GB of DDR4 memory, Windows, CUDA 13.x and a Qwen3.6-35B-A3B-MTP-IQ4_XS.gguf quant. The reported best plain llama-bench result was about 914 tokens per second for prompt processing on pp512 and 46.8 tokens per second for generation on tg128. For daily coding, the poster settled on a 32k context profile that generated around 43.4 tokens per second while leaving about 273 MiB of VRAM free.
That is not a laboratory number with an unlimited server budget. It is a practical developer setup. A 16k context run was slightly faster at about 44.5 tokens per second, but it left only 37 MiB of VRAM free, which is the kind of margin that makes a workstation feel fragile. The 32k profile matters more because coding assistants need room for files, tool output, plans, failed attempts and follow-up instructions. A model that only works in a tiny chat window is less useful for startups trying to build real products with it.
Qwen3.6-35B-A3B is a sparse mixture-of-experts model from Alibaba's Qwen team. The official model card lists 35 billion total parameters, but only about 3 billion activated per token. It also lists 256 experts, with 8 routed experts and 1 shared expert active, plus a native context length of 262,144 tokens that can be extended further with the right serving setup.
That architecture is the point. Dense models force the hardware to move through the whole parameter set during inference. MoE models keep a larger body of knowledge available, but route each token through only a smaller active slice. Quantization then squeezes the storage footprint further. In this Reddit test, the IQ4_XS GGUF quant and careful MoE offloading were doing much of the work, especially through the ncmoe setting that controls how many MoE blocks remain off the GPU.
The sweep in the post shows how sensitive this can be. Plain decoding stayed strong around ncmoe 18 and ncmoe 17, then fell sharply at ncmoe 16. KV cache choices also mattered. On this setup, q8_0 for both key and value cache was basically free compared with q4_0, while mixed q8 and q4 or q5 choices were much slower. That is a useful reminder for founders: local inference is not just model selection. It is model, quant, runtime, context length, cache format, driver version and patience.
There was also a test of MTP speculative decoding through a llama.cpp branch. The best result was around 47.7 tokens per second, only about 2% faster than a well-tuned plain decoding setup. That is still interesting, but it is not the headline. The headline is that normal decoding already looked good enough for a coding workflow on a 12GB card.
Why founders should care
For startups, the immediate appeal is control. Local inference means sensitive code, customer records, contracts, support tickets or internal documents do not have to leave the machine or the office network. That does not eliminate security work, but it changes the risk profile for teams in health, finance, legal, developer tools and enterprise SaaS.
It also changes cost planning. API pricing is usually fine at prototype scale, then starts showing up in uncomfortable places once agents run long context, retry tool calls, summarize repositories or serve many users. If a solo founder can run a capable coding model locally at 30 to 45 tokens per second, the early product cycle becomes less dependent on cloud inference limits. That can mean more experiments, more private test data and fewer decisions shaped by token anxiety.
The quality question is still open. Some commenters said the model felt close to stronger hosted assistants in certain coding flows, while others argued that lower quants can feel weakened for serious work and that Q6 or Q8 still performs better. Those impressions are valuable, but they are not benchmarks. Community tests tend to mix different prompts, builds, GPUs, drivers, samplers and expectations. A founder should treat them as a scouting report, not a procurement policy.
The better approach is simple. If local inference matters to the business, build a small internal eval before building around the model. Use actual tasks from the company: refactor a service, summarize a customer thread, generate a migration, inspect a policy document, write a test suite. Compare Qwen3.6-35B-A3B against a smaller dense model and a hosted frontier model. Measure not only tokens per second, but how often the output compiles, follows instructions, preserves context and avoids confident mistakes.
That is where this development becomes more than a hobbyist milestone. Consumer GPUs are becoming strategically relevant again because open-weight MoE models are giving them a job they can actually do. The next thing to watch is not whether every 12GB card can run a 35B model. It is whether enough founders can turn these community setups into reliable private workflows that reduce cost, protect data and move product work faster.
Also read: ICE wants smart glasses to make facial recognition harder to ignore • HiDream-O1-Image puts pixel space back in the image model race • ChatGPT on Android may put Codex sessions in your pocket
