Local Models Are Not for Coding

TL;DR

• “Open weight” is not “open source.” You can download the weights. That doesn’t mean you can run them — or that the license is actually open.
• The good open-weight coding models are huge. DeepSeek V4-Pro is a 1.6-trillion-parameter model; Kimi K2.6 is a full trillion. Even squished down with quantization they need hundreds of GB — a data-center node, not your desktop.
• VRAM is the wall, and it keeps getting pricier. Apple pulled its 128 GB+ Mac configs and raised Mac prices; a 96 GB workstation GPU jumped from ~$8,565 to ~$13,250 in a year.
• Local can’t parallelize. Real coding fans out to many agents at once. On a big model that nearly fills the card, one GPU serves one or two streams.
• Owning the hardware loses the math. Run a rig flat-out all year and its entire output would cost a few hundred dollars to buy from an API — while the electricity alone, at California rates, is ~$2,800/year, on top of $13k–63k of GPUs.
• Small local models are great — for offline translation, summaries, writing help, and privacy. Just not for agentic coding.

First, “open weight” is not “open source”

First, some groundwork — because people use these two interchangeably, and they’re not the same. This whole post is really an arithmetic problem: can a normal person actually run these models on their own computer? (Short answer: the good ones, no — and the rest of the post is the math.) But that question only makes sense once you’re clear on what “open” is even promising you.

The Open Source Initiative published an Open Source AI Definition  in October 2024. To call a model open source under it, you need to grant the four freedoms (use, study, modify, share for any purpose) and release three things under open terms: the weights, the full training and inference code, and enough information about the training data that someone skilled could rebuild an equivalent model.

Open weight is a much smaller promise: here are the trained weights — a giant grid of numbers — go run them. You get the finished artifact, not the recipe. No training code, no dataset, usually no real account of how it was built. You can run it and fine-tune it, but you can’t audit it, reproduce it, or see what actually went in. It’s a black box you’re allowed to download. That’s the core split from open source, where the whole point is that nothing is hidden. The OSI is blunt that open weights alone fall short  of open source.

That gap matters in practice:

• Meta’s Llama ships under a community license that isn’t OSI-open. It has a 700-million-monthly-active-users commercial gate , a “Built with Llama” naming rule, and an acceptable-use policy. Useful, generous even — but not open source.
• DeepSeek releases its weights under the plain MIT license . Qwen3 uses Apache-2.0. Both are genuinely permissive.

Here’s the part people miss: even the permissive ones usually still aren’t open source by the OSI’s bar, because the training code and data information aren’t published. A friendly license on the weights is necessary, not sufficient.

So when you read “open source model,” mentally translate it to “open weight” until proven otherwise. And keep the two questions separate: can I legally use this? and can I actually run it? The rest of this post is about the second one.

You can download it. You can’t run it.

The best open-weight coding models right now are enormous. They’re sparse mixture-of-experts models with hundreds of billions to over a trillion parameters. As of July 2026:

• DeepSeek V4-Pro — a 1.6-trillion-parameter  mixture-of-experts model (49B active). Even its smaller sibling, DeepSeek V4-Flash  (284B), ships at roughly 158 GB.
• Qwen3-Coder-480B — about 960 GB  at full precision.
• GLM-5.2 — Zhipu’s ~753B-parameter  flagship (40B active), about 1.5 TB in BF16.
• Kimi K2.6 — a full trillion parameters  (32B active), around 1.1 TB on disk.

“Just quantize it,” someone says. Sure. Quantization trades a little quality for a lot less memory. But it only gets you so far: Qwen3-Coder-480B at 4-bit is still ~290 GB , and Kimi K2.6 crushed to ~2-bit is still ~381 GB. For reference, a server with eight H100 GPUs holds 640 GB of VRAM — and even that can’t hold the biggest of these at full precision.

Footprints from the HuggingFace and Unsloth  model cards. Even at their smallest, the big open coders need 150–380 GB.

The models you can run on a single consumer GPU are one to two orders of magnitude smaller — Qwen3-Coder-30B-A3B  at about 19 GB when quantized (a 30B mixture-of-experts with just 3.3B active), or Devstral-Small-24B  at ~14 GB. They’re the strongest coders that fit a 24–32 GB card, and the leaderboards will happily put them next to frontier models. Fine for a small refactor, a one-file bugfix, a quick script. Point them at a medium or hard programming task — a multi-file change, a real debugging session, anything that needs sustained reasoning across a repo — and the gap shows up fast. Wiring them into a CLI agent like Claude Code doesn’t close it either. You get the same model with a nicer harness. The harness isn’t the bottleneck.

Downloadable is not the same as runnable. And “runs locally” is not the same as “good enough to code with.”

VRAM is the wall (system RAM won’t save you)

The number that decides whether a model runs well is VRAM — the memory attached directly to the GPU. Generating tokens is bound by memory bandwidth: the chip has to read the active weights for every single token. An RTX 5090 reads its VRAM at roughly 1,792 GB/s . The PCIe link to your system RAM moves about 64 GB/s. So the moment part of the model spills out of VRAM into ordinary RAM, generation falls off a cliff.

“I’ll put 256 GB of RAM in my desktop” doesn’t fix it. That RAM is on the wrong side of the bottleneck.

This is why the “big unified memory” machines exist — Apple Silicon, NVIDIA’s DGX Spark, AMD’s Strix Halo boxes. They give the GPU access to a large pool of fast-ish memory. But there’s a catch: capacity and bandwidth pull in opposite directions at this price point.

Read that table as a trap. The 5090 has the bandwidth to run a model fast, but only 32 GB to fit one in. The unified-memory boxes have the capacity but a quarter to a seventh of the bandwidth, so anything large generates slowly. And a real data-center GPU has both — at a price no one’s putting under their desk. You get compute or capacity. A frontier model needs both.

The hardware only gets more expensive

Here’s the cruel twist: the exact hardware you’d need is the hardware that AI demand is making scarce.

Memory makers have been pouring capacity into high-bandwidth memory for AI data centers, and it’s starved the rest of the market. TrendForce  reported conventional DRAM contract prices rose 93–98% quarter-over-quarter in Q1 2026. That flows straight through to everything you’d buy.

Prices as of July 2026. Sources per item below.

• The RTX 5090 launched at a $1,999  list price and now sells on the street for roughly $2,900–$4,300 .
• The 96 GB RTX PRO 6000 went from about $8,565 to ~$13,250 .
• NVIDIA’s DGX Spark got an 18% bump to $4,699 , citing memory costs.
• Apple pulled its 128 GB and larger Mac Studio configs  — the M3 Ultra now caps at 96 GB — roughly doubled its RAM upgrade tiers  (48→128 GB went from $1,000 to $2,000 before it vanished), and raised prices across the Mac line  in June.

Tim Cook, in a June Wall Street Journal interview (reported by CNBC ): “price increases are unavoidable… the memory guys are passing along huge price increases.” When even Apple says the quiet part out loud, betting your local-AI plan on cheap memory is a rough bet.

One machine, one lane

Say you ignore all of that and build a rig anyway. There’s still a problem that money doesn’t fix cleanly: parallelism.

Real coding work isn’t one request at a time. I’ll have a couple of agents running in parallel, one of them spinning up sub-agents to explore different parts of a repo, a tool call or two firing off on the side. The number of models I’m hitting at any given second swings between one and a lot.

On a big open model that nearly fills the card, a single GPU serves one stream well, maybe two — the KV cache eats the leftover VRAM, so there’s nowhere to park more concurrent work. (Smaller models can pack more; but we’re talking about serious coding) It does not fan out to ten. So you’re stuck between two bad options:

• Buy for your peak. Now most of the day that hardware sits idle, which is money doing nothing.
• Buy for your average. Now the moment you need to fan out, you’re blocked, waiting for a lane to free up.

An API doesn’t have this problem. Fire off forty requests and forty run. You’re paying for exactly what you use, and the “buy for peak” cost is somebody else’s to amortize across thousands of customers.

Yes — that bill can look worse than owning a GPU. In most cases that’s not because the API is overpriced; it’s because modern coding harnesses — Cursor, Claude Code, Codex, and the rest — burn tokens at a level of parallelism a single box can’t match. Ten agents, sub-agents, tool calls stacked on top of each other. If you tried to buy enough local hardware to keep up with that fan-out, you’d spend more than the API ever charged you. The expensive part is the concurrency, not the rental.

Doing the math: local 24/7 vs an API

Here’s the comparison that actually matters — not “how much is the hardware,” but “what does the hardware produce, and what would that output cost to simply buy?”

What a rig makes. A single high-end GPU generates maybe 30 tokens a second on a large open model — benchmarks vary a lot , but that’s a fair, even generous, ballpark. Run it every second of the year — all 31.5 million of them — and you get about 946 million tokens. That’s the ceiling: the absolute most a 24/7 rig could ever put out on an open frontier model.

What that output costs to rent. Buy those same 946 million tokens from an API:

(30 tokens/s × 31,536,000 s) ÷ 1,000,000 × price per 1M tokens

At DeepSeek V4-Flash ’s $0.28 per million output tokens, that’s about $265 a year. At the pricier V4-Pro  flagship ($0.87), about $825. Those are output-only numbers — the tokens the rig emits. A real coding session also pays for input (the prompt, the repo context, the tool results), so the API bill for the same work lands higher. Still nowhere near the hardware.

What the rig costs to run. Just the electricity is ~$2,800 a year at California rates. An RTX 5090 pulls about 0.8 kW at the wall under load (measured ~775 W for a full system ) — roughly 19 kWh a day, ~7,000 kWh a year — and PG&E runs near $0.40/kWh  (California already averages 35¢ , second-highest in the US). Cheaper grids cut that number; they don’t flip the comparison.

Rig assumed at 30 tokens/sec, 24/7. API priced at DeepSeek V4 output rates; owning counts the GPU plus ~$2,800/GPU/year of California electricity. As of July 2026.

That’s one stream. Add concurrency — more streams, more VRAM, more GPUs — and the ceiling rises, but not by much. Here’s the absolute most a rig could physically emit in a year:

Each stream runs at a steady 30 tokens/sec. Even four of them flat-out — 120 tok/s — tops out under 4 billion tokens a year, roughly $1,100 of V4-Flash output or $3,300 at V4-Pro rates.

Sit with that gap. A full year of everything the rig can generate — even maxed out on concurrency — is a few hundred to a few thousand dollars of output to just buy. The electricity alone to produce it yourself is ~$2,800 on a California bill, and that’s before the $13,250 for one 96 GB GPU, or the $60,000+ multi-GPU box you’d need to run a model like V4-Pro at all. The hardware never catches up, because you’re not competing with a data center’s prices — you’re competing with its scale.

That scale is concrete. A home rig usually runs one request at a time. A provider packs dozens of requests onto the same GPU so the expensive part — reading the weights from memory — gets shared across many users. Same chip, far more tokens per hour, so the cost per token drops hard. You pay for on-demand access. They earn on keeping the fleet busy. That’s why an API can charge a few dimes per million tokens and still make the math work, while your solo box can’t.

Two honest caveats:

1. The $265 / $825 figures are output-only, and open models often burn more tokens per task. Real jobs also pay for input, and reasoning traces get billed as output, so a real coding session costs more than the raw ceiling math. Artificial Analysis  measures cost per task for exactly this reason. It nudges the rent numbers up — nowhere near enough to close the gap.
2. The frontier closed models still cost more per token. Claude Opus is $5 in / $25 out per million . If open-weight quality is good enough for your task, the API route is genuinely cheap. If it isn’t, a local rig wasn’t going to run the closed model anyway.

And here’s the real superpower of open weights — not that you host them, but that anyone can. On OpenRouter a single open model is served by more than a dozen providers; DeepSeek V4-Pro alone ranges from about $0.44 to $1.74 per million input tokens depending on who you pick for speed. That competition is the win, and you get all of it without owning a single GPU.

Where small local models actually shine

None of this means local models are pointless. It means I keep seeing them pointed at the wrong job.

Small models are already excellent, running right on your phone or laptop:

• Apple Intelligence runs an ~3-billion-parameter on-device model  (2-bit quantized, 15 languages) for writing tools, summaries, and notification triage.
• Google’s Gemma 4  ships tiny E2B and E4B variants (~2–4B effective parameters) built to run offline on phones and edge devices — it’s the base for Gemini Nano 4 on Android , which Google says is up to 4× faster and uses 60% less battery than the last generation.

The tasks they’re great at share a shape — light, bursty, and better off never leaving the device:

• Offline translation on a plane or across a border.
• Summarizing an email thread or a page of notes.
• Writing and autocomplete help as you type.
• Transcription without uploading audio anywhere.
• Anything privacy-sensitive, where the whole point is that the data stays local.

There are coding jobs where local still wins on purpose: an air-gapped machine, code that can’t leave the building, or a tiny autocomplete model sitting in the editor for the next five tokens. Those are real. They’re also not “replace Claude Code with a 1.6T model under my desk.”

Agentic coding is the opposite shape. It’s sustained, not bursty. It wants huge context windows, not a paragraph. It fans out into many parallel calls. And it rewards the very best model you can reach, because the difference between a good answer and a great one compounds across a long task. That’s exactly what a small on-device model can’t give you — and exactly what an API can.

My Take

I love open-weight models. Releases like DeepSeek R1 dragged the entire field forward and forced prices down across the board. I want more of them, not fewer. If a task runs fine on an open model, moving it off the expensive frontier is often the smart call.

I just don’t buy the fantasy that you’re going to run a frontier open-weight coder on your own hardware and get anything close to what the labs give you. The good models don’t fit, the hardware that could hold them is expensive and getting worse, and a single box can’t parallelize the way real work does.

So here’s the split I’d actually make:

Open weights won by making a competitive market, not by moving a data center into your closet. Use the market. Keep the small stuff local. And stop trying to run a trillion-parameter model on a gaming GPU.