Local LLM deployment
Running a capable large language model on your own machine is no longer a research project. With open-weight models like Qwen 3.6 27B, you get mid-2025 frontier-level intelligence on a laptop or desktop GPU — offline, private, and under your control.Why run models locally
Cloud APIs are convenient, but they come with trade-offs:- No take-backs: A cloud model can be deprecated, rate-limited, or taken down overnight — as Claude Fable 5 users learned in June 2026.
- Privacy: Sensitive code, medical data, or personal notes never leave your machine.
- Cost at scale: API credits are cheap now due to heavy subsidies. When the subsidy era ends, local models become the economical default for frequent use.
- Customization: You can fine-tune a local model for your domain, coding style, or project conventions.
The current sweet spot: Qwen 3.6 27B
As of mid-2026, Qwen 3.6 27B is widely considered the first local model that delivers genuine general intelligence suitable for daily development work. It comes in two variants:
The dense 27B is recommended for quality-sensitive work. The MoE variant is 3× faster and still very capable, suitable for quick iterations.
Intelligence level
On the Artificial Analysis index, Qwen 3.6 27B scores around mid-2025 frontier level:
It significantly outperforms Gemma 4 31B (the previous local coding default) and approaches DeepSeek V4 Flash quality.
Setup with llama.cpp
llama.cpp is recommended over Ollama — it is direct, open, and gives you full control over quantization and inference parameters.1. Download a quantized model
8-bit quantization halves model size with negligible quality loss. Popular GGUF sources:- unsloth/Qwen3.6-27B-MTP-GGUF (with multi-token prediction support)
- bartowski/Qwen_Qwen3.6-27B-GGUF
2. Start the server
-hf— pull from Hugging Face (cached on subsequent runs)--spec-type draft-mtp— multi-token prediction for ~1.8× speedup-ngl 999— offload all layers to GPU-fa on— flash attention-c 65536— 64K context window (native is 256K)
http://127.0.0.1:8080 to chat directly.
3. Connect your coding agent
For OpenCode, add to~/.config/opencode/opencode.jsonc:
Performance reference
Measured on MacBook Max M5 128 GB with 8-bit quantization:
On consumer GPUs:
- RTX 5090 32 GB: Q6_K quantization, ~50 tok/s at 123K context, ~28 GB VRAM
- 32 GB shared memory devices: 4-bit quantization fits comfortably
When to use local vs. cloud
Model evaluation: benchmarks vs. real experience
A recurring pattern in open-weight model releases is the gap between benchmark scores and practical experience. Two data points from July 2026 make this impossible to ignore: Tencent Hy3 (295B total / 21B active MoE) explicitly noted in its model card:模型的实用体验不完全与榜单成绩挂钩。基于广泛的用户反馈和分析，我们定位并优化了一系列体验向能力，获得了产品侧一致且积极的评价。(“The model’s practical experience does not fully correlate with benchmark rankings.”) Anthropic Opus 5 vs Fable 5: Opus 5 beats Fable 5 on nearly every benchmark, yet anyone who has used both meaningfully can tell within minutes that Opus 5 is nowhere near Fable in practical coding and agent tasks. As one observer put it: “this can only mean our benchmarks are now almost completely useless.” The benchmarks are failing because they measure narrow task completion under clean conditions, while real-world usefulness involves dimensions they don’t capture: instruction following consistency across long contexts, refusal calibration, output formatting reliability, tool use judgment, and the ability to recover from mid-task errors. When evaluating any model for daily use, test it on your actual workflows rather than relying solely on leaderboard scores.
Model architecture frontiers: diffusion language models
The dominant LLM architecture is autoregressive — generate tokens left to right, commit to each one, never go back. This creates a fundamental problem for agent workflows: when a tool call produces an error 50 tokens ago, the model can’t edit that part of the output. It must either start over or push forward with incorrect context. Diffusion language models take a different approach. Instead of generating sequentially, they refine the entire output in parallel passes. inclusionAI’s LLaDA2.2-flash (July 2026) introduces the first diffusion model with explicit editing capability:- DELETE and INSERT control tokens let the model actively remove and add content during generation
- The model can dynamically adjust sequence structure, enabling self-correction
- This is particularly valuable for multi-turn tool calling and long-horizon agent interactions
Industry: Anthropic enters the chip race
Anthropic has started early development of its own AI chips, seeking memory semiconductor supply from SK Hynix and reportedly in talks with Samsung for a custom project potentially using Samsung’s 2nm process and advanced packaging. This follows OpenAI’s earlier chip efforts and signals that frontier AI labs now see vertical integration into silicon as strategically necessary.What’s next
The gap is closing fast. GLM 5.2 already delivers frontier-level open-weight intelligence, and the trend toward separating reasoning from factual knowledge will make future models even more efficient on consumer hardware.References
- Qwen 3.6 27B is the sweet spot for local development — setup guide, benchmarks, and real-world impressions
- Will it Mythos? — Qwen 3.6 benchmark comparison
- Simon Willison on Qwen 3.6 27B
- Tencent Hy3 on HuggingFace
- Opus 5 vs Fable 5: benchmarks are broken
- LLaDA2.2-flash: diffusion model with editing
- Anthropic developing own AI chips