"Running out of VRAM and the model will not load" is the most common complaint in local LLM work in 2026. A 24G card barely fits a quantized 70B after squeezing. A 16G card starts warning at load time. An 8G card has about one option. So when VRAM runs short, is quantization the only move?
No, and the alternatives are real. Sorting through the paths the community has shipped, five distinct routes emerge: layer-wise streaming, GGUF quantization, CPU offload hybrids, distilled and small models, and remote APIs. None is strongest, because none solves the same problem. This review crowns no winner; it matches what you are short of to the route that solves it.
What We Compare, and What We Don't
First, this compares memory strategies, not deployment tools. A review of five deployment tools -- Ollama, LM Studio, vLLM, llama.cpp, and GPT4All -- already exists here: our local deployment tools comparison. That piece answers "which tool should I use," comparing interface shape, onboarding difficulty, concurrency, and measured throughput on one machine. This piece answers "which route to take when the card is too small." The two divide the work rather than repeat it: tools execute routes, and the streaming route may not use the tool stack you know.
Second, no cross-route speed benchmarks. These routes run on different hardware, models, and precision combinations, so tokens-per-second figures are not comparable on one scale. What follows is qualitative only: which route is slow by construction and which is fast. For same-tool numbers, the tools comparison above has tables.
Third, every VRAM figure carries its source. Wherever a gigabyte number appears, the source and hardware precondition are stated. All airllm numbers here follow the README wording -- self-reported figures from the project's own benchmark table, which we did not reproduce and do not treat as a common denominator. The thesis: the real choice is never "which method is strongest" but "what are you short of -- VRAM, RAM, or speed."
The Five-Axis Table
<table> <thead> <tr><th>Axis</th><th>Layer-wise streaming</th><th>GGUF quantization</th><th>CPU offload hybrid</th><th>Distilled / small models</th><th>Remote API</th></tr> </thead> <tbody> <tr><td>VRAM footprint</td><td>Lowest: single-digit GB; README wording says 2.8T Kimi K3 runs under 4GB</td><td>Low to medium: falls with bit width, Q4 is the sweet spot</td><td>Low: only layers that fit stay on the card</td><td>Low: swap in a smaller model</td><td>Zero locally: provider-side compute</td></tr> <tr><td>Speed</td><td>Slowest by construction: layers stream from disk or RAM, slowness is design</td><td>Usable: model resident in VRAM or mapped in RAM</td><td>Hinges on PCIe and memory bandwidth</td><td>Fastest: lowest latency</td><td>Set by network and provider load</td></tr> <tr><td>Quality loss</td><td>None: README is explicit, no quantization, distillation, or pruning</td><td>Lossy: low-bit tiers degrade measurably</td><td>Weights untouched, theoretically lossless</td><td>Largest: swapping models swaps the ceiling</td><td>Choice-dependent: can reach flagship tiers</td></tr> <tr><td>Deployment complexity</td><td>Highest: triple-locked version matrix, terabyte-class disk gate</td><td>Lowest to medium: most mature ecosystem</td><td>Medium: you set the layer split</td><td>Lowest: usually a different model name</td><td>Lowest: sign up, key, pay per use</td></tr> <tr><td>Who it suits</td><td>Occasional oversized models, local 2.8T curiosity</td><td>The everyday majority; the default</td><td>Small card, ample RAM, latency tolerated</td><td>Simplicity above the ceiling</td><td>Non-sensitive data, quality and cost first</td></tr> </tbody> </table>One methodological note beyond the table: read it across and you see five routes; read it down and you see five prices. Every route pays somewhere -- streaming in speed, quantization in quality, offload in latency, distillation in ceiling, API in data boundary and bill. No column wins outright, which is why no ranking exists.
Repository addresses, link and plain text: github.com/lyogavin/airllm (plain: github.com/lyogavin/airllm), github.com/ggerganov/llama.cpp (plain: github.com/ggerganov/llama.cpp), github.com/ollama/ollama (plain: github.com/ollama/ollama).
Route One: Layer-Wise Streaming, the VRAM Ceiling Bought with Speed
This is airllm's route, and the most counterintuitive of the five. Its README states the position plainly: no quantization, no distillation, no pruning -- instead it fits a 70B onto a single 4GB card through layer-wise streaming.
The mechanism is not hard to grasp. Inference proceeds layer by layer, and once a layer has been computed it no longer needs to stay resident in VRAM. airllm loads each layer in order, computes, then swaps in the next, so VRAM use depends on "one layer plus the resident set" rather than total parameters. That is why it claims Kimi K3 at 2.8T fits under 4GB.
Measured figures per README wording, all self-reported with hardware preconditions, none reproduced here:
- 70B on a single 4GB card, the headline claim.
- Qwen3.8-27B (dense VL, Gated DeltaNet plus Gated Attention) at 3.33GB VRAM, end-to-end on an RTX 3090.
- Qwen3.8-Flash-Next (125B, including roughly 51B of n-gram embeddings) at 5.95GB VRAM, end-to-end on an RTX 4090.
- Kimi K3 (2.8T parameters, 104 billion activated, 896 experts with 16 active per token) at 3.72GB VRAM, end-to-end on an RTX 6000 Ada, streaming per expert.
- DeepSeek-V3 (671B) at roughly 12GB.
The Kimi K3 number deserves a pause. Running 2.8T parameters inside 3.72GB of VRAM looks like bending physics; it is not. The cost moved to disk and time: full weights of a 2.8T model are a terabyte-class disk burden, and its license is a modified MIT license, not standard MIT.
Three boundaries must ship with that number. First, the slowness is structural, not broken: every token generated walks the streaming path again, and since the project publishes no tokens-per-second figure, neither do we. Second, the version matrix is triple-locked -- a CUDA 12 torch build (flash-attn ships no CUDA 13 wheels), flash-attn itself, and the transformers branch each model demands: Qwen3.8-27B needs transformers 5.8+, Qwen3.8-Flash-Next needs the in-tree qwen4_exp, Kimi K3 needs 4.56.x plus compressed-tensors. Third, the ecosystem has potholes: GitHub API verified 2026-10-12 shows 36,747 stars and 160 open issues.
Who should take it: anyone who occasionally needs a model their card cannot hold, wants to see what a 2.8T feels like locally, and does not mind waiting. Turn-by-turn token conversation will hurt.
Route Two: GGUF Quantization, the Mature Default
Shrinking weights is the least revolutionary idea available. GGUF is the format the llama.cpp ecosystem pushed forward, spanning Q8 down through Q6, Q5, Q4 and below, and Ollama, LM Studio, and llama.cpp all consume it. Onboarding cost is close to zero, and the pitfalls and tier-selection details are in our Ollama local deployment SOP. The route's value is that it is the only route where everyday usability and ecosystem maturity hold simultaneously. No layer-loading concepts, no bus tuning, no version triple lock. The cost is equally plain: quantization is lossy, and low-bit tiers degrade measurably on long-chain reasoning, precise computation, and code generation. The memory you saved is not free -- compressing weights buys VRAM with output quality. The tools comparison puts it bluntly: a 7B at Q4 with 8G to spare feels comfortable; a 70B forced into 16G by quantizing ever harder feels like a slideshow.
Who should take it: the overwhelming majority of everyday local deployers. If the requirement is "install and it works, quality acceptable," quantization wins.
Route Three: CPU Offload Hybrids, for a Small Card and Plenty of RAM
This route neither compresses weights nor streams at crawl speed. It shares the load: layers that fit in VRAM stay there, layers that do not fall back to system RAM, and the runtime shuttles between the two.
Its behavior depends on two hardware parameters: PCIe link bandwidth and system memory bandwidth. A smaller card means more layers crossing the bus, and the speed penalty follows. System RAM capacity has to be genuinely sufficient, the most commonly missed precondition -- plenty of people discover the shortfall only after plugging things in, then get squeezed between OOM and swapping. Its distinct value is that it preserves original weight precision, theoretically lossless, a genuine difference from quantization. So when a task is precision-sensitive and the tier cannot go lower, offload is an overlooked middle option. The drawback is that you set the split.
Who should take it: people whose GPU spec is fixed, who have RAM to spare, and whose tasks are precision-sensitive.
Route Four: Distilled and Small Models, the Most Honest Surrender
This route stops talking about VRAM entirely and switches to a far smaller model, or a distilled version. It is the least engineering effort of the five: swap the model name and you are done, latency is the lowest, laptops run it happily. The cost is the largest and least negotiable: swapping models swaps the capability ceiling. This is not "the same model using less power" but "a different model answering." On complex reasoning, long-chain tasks, and domain questions, the gap to a flagship model does not close because deployment was convenient. The route is also most often misread into one of two extremes -- small means weak, or big is always better. In practice small models are steadier on narrow tasks such as extraction and classification, at a fraction of the cost. Our local coding model comparison breaks model choice down by task type, and one lightweight rollout is in the Mellum 2.1 SOP.
Who should take it: people who rank engineering simplicity above the ceiling, and whose tasks are narrow and stable.
Route Five: Remote APIs, Removing the VRAM Problem from the Table
This route is the simplest of the five: let the compute happen on someone else's machine. The VRAM problem does not exist here -- not solved, nonexistent. A tiny local GPU changes nothing, because the model was never loaded into your VRAM at all.
The cost transfers to two places. The data boundary: requests leave your machine over the public internet, cancelling the "data stays in-house" motive that is the core reason for local deployment. The cost structure: a one-time hardware investment becomes metered billing -- cheaper if usage is light, runaway if it is heavy, and you do not control pricing. The hosted tier of our Qwen3.8-Flash-Next SOP expands on this.
One more reason is easy to miss: when what you need is flagship capability, no local route is in the running at all. No amount of wrangling an 8G card reaches flagship quality, so the reframing runs from "how do I squeeze it into VRAM" to "who pays for the VRAM."
Who should take it: people whose data is not sensitive, who need no offline operation, and who care about quality and total cost.
Each in Its Rightful Place: A Five-Line Decision Guide
Every route here only gets the model running, and whether to trust it remains your call.
- VRAM short by a wide margin, and you can wait: layer-wise streaming. Repository github.com/lyogavin/airllm, Apache-2.0, 36,747 stars (GitHub API, verified 2026-10-12); setup and pitfalls are in our airllm 4GB resource piece.
- Short by a little, and you want everyday conversation: GGUF quantization. The default, most mature ecosystem.
- Small card, plenty of RAM, precision-sensitive tasks: CPU offload. Keeps original weight precision, pays in latency.
- Hopelessly short on VRAM and unwilling to tune: distilled or small models. Trade the ceiling for speed.
- You want flagship capability, or the data is not sensitive: remote API. Cost plus data boundary move to the provider.
FAQ
Q1: Which of the five routes is strongest? There is no strongest, because they do not optimize the same thing. Streaming pushes VRAM to the floor and pays in speed. Quantization solves VRAM and speed together and pays in quality. Offload preserves quality and pays in latency. Small models max out speed and pay in ceiling. API eliminates the VRAM problem and pays in data boundary and billing. The question worth asking is whether you are short VRAM, RAM, or speed.
Q2: Can airllm really run a 70B on a 4GB card? Per the official README wording, yes, via layer-wise streaming with no quantization, no distillation, and no pruning. Two preconditions matter. The official figure is a VRAM footprint number, not a speed number. And every airllm VRAM figure cited here is self-reported, tagged with the hardware it was measured on. We did not reproduce them and make no cross-route comparison; on speed the project publishes no figure, so neither do we.
Q3: How much quality does quantization cost? It depends on the tier and the task. Low-bit tiers show measurable loss on long-chain reasoning, precise computation, and code generation, while narrow work such as extraction or classification barely notices. There is no single answer, only a judgment about how precision-sensitive your task is. One point to restate: the memory quantization saves is paid for with output quality, not found in a drawer. For tiers see our Ollama SOP.
Q4: CPU offload or quantization -- which one? Depends on what you are short of. Quantization compresses VRAM and disk, is usually faster, and is the everyday default. Offload compresses VRAM alone while preserving original weight precision, at the cost of speed that hinges on PCIe and memory bandwidth. When a task is precision-sensitive and the tier cannot drop, offload wins.
Q5: Is a remote API cheating? It is not cheating, it is moving the problem. The VRAM problem genuinely does not exist there, but only because the compute never happens on your machine, and the cost lands on data leaving your host plus metered billing. If staying in-house is one of your core requirements, this route was never a candidate.
Closing
Rather than picking the strongest method, first work out what you are short of. Short by a wide margin and able to wait, take layer-wise streaming. Short by a little and want everyday usability, take quantization. Small card and big RAM, take offload. Unwilling to tune, take a small model. Want flagship quality with non-sensitive data, take the API.
How much VRAM do you have, and which route are you on? Tell us in the comments.