Look at almost any local LLM deployment project and you will find the same underlying move: make the model smaller. Quantization, pruning, distillation, sparsity — all of them exist to squeeze weights into VRAM. One project runs the opposite strategy. lyogavin/airllm does not quantize, does not distill, does not prune. It carries the model into VRAM one layer at a time, and swaps the next layer in as soon as the current one finishes. Repository address: github.com/lyogavin/airllm.
Per the README, this layer-wise streaming approach fits a 70B model onto a single 4GB card, and even runs Kimi K3 — a 2.8-trillion-parameter model — below 4GB. The repository currently sits at 36,747 stars and 3,829 forks under the Apache-2.0 license, was created on June 12, 2023, and was last pushed on October 12, 2026 (all figures verified directly through the GitHub API on 2026-10-12).
What makes this approach interesting is not the size of the numbers, but that it changes which axis you sacrifice. When "it does not fit" is the only hard constraint, you can trade time instead of trading away the model itself.
What It Actually Does
The default assumption in conventional inference is that all weights live in VRAM at once: the bigger the model, the bigger the card you must buy. Quantization routes shrink the weights until they fit, and the bill comes due as accuracy loss. AirLLM starts from the opposite premise. VRAM can be tiny; time can be long. It splits the checkpoint by layer, reads each layer into VRAM during inference, releases it once it has contributed to the computation, and keeps only the current layer plus the necessary intermediate state. Peak memory therefore shifts from "the whole model" to "one layer's weights plus activations." Finer layer splits mean a lower peak and more transfer operations.
The direct corollary is that VRAM usage becomes largely decoupled from total parameter count. The gap between 2.8 trillion parameters and 27 billion parameters on the memory curve gets absorbed by transfer volume.
One point is easy to get wrong here. AirLLM does not save the space the weights themselves would occupy; it saves the space of having all the weights resident simultaneously. The practical result is that model capacity is no longer capped by VRAM at all — it is capped by disk and host memory instead. A cheap small card buys you freedom over model size, and the bill moves elsewhere. The same logic pays off harder for mixture-of-experts models: sparse experts that never fire need never enter VRAM, and that per-expert streaming is why Kimi K3 became the most eye-catching line in this project's README.
The README Numbers, With Their Prerequisites
The README lists several checkable figures, all following the project's own measurements, and every one is tied to specific hardware and version prerequisites. Quoted without those prerequisites, they mean nothing.
- A 70B model in under 4GB on a single card is the headline marketing claim, with no specific hardware named.
- Qwen3.8-Flash-Next (125B, Qwen4ExpForConditionalGeneration architecture, including roughly 51B of n-gram embeddings) at 5.95GB VRAM, end-to-end on an RTX 4090.
- Qwen3.8-27B (a dense multimodal model mixing Gated DeltaNet with Gated Attention, natively vision-capable) at 3.33GB VRAM, end-to-end on an RTX 3090, requiring transformers 5.8 or later.
- Kimi K3 (2.8T) at 3.72GB VRAM, end-to-end on an RTX 6000 Ada.
- DeepSeek-V3 (671B) at roughly 12GB VRAM.
The training side has figures too, from a September 2026 update: training Qwen3.8-Flash-Next (125B) stays under 6GB VRAM on an RTX 3060 Ti, and training Qwen3.8-27B takes roughly 2GB VRAM at sequence length 512.
Note that every one of those numbers is VRAM, not speed. The project publishes no tokens-per-second figure, and this article will not invent one.
Why 3.33GB Is Plausible
The Qwen3.8-27B row is the one worth thinking hardest about. A dense 27-billion-parameter model occupying 3.33GB means VRAM can barely hold the current layer. Three conditions make that work. First, weights stream in layer by layer, so the peak is roughly one layer. Second, the checkpoint must be split and written to disk in advance, which is an installation cost rather than an inference cost. Third, 3.33GB is a measured end-to-end peak, not an average, so the memory curve is jagged.
There is a companion requirement that is easy to miss. Qwen3.8-Flash-Next maps its n-gram table files into host memory, which calls for a machine with 64GB-class RAM; its checkpoint sits at roughly 360GB on disk, and after splitting you can reclaim the original with delete_original=True. VRAM was saved, and the bill for disk and host memory was simply reissued somewhere else.
Versions are a second lock. Qwen3.8-27B requires transformers 5.8 or newer; Qwen3.8-Flash-Next depends on the in-tree qwen4_exp support and needs transformers installed from source; Kimi K3 requires transformers 4.56.x. Three models, three version requirements, and a mismatch means the thing simply will not start.
The Kimi K3 Row: 2.8 Trillion Into 3.72GB
Kimi K3 is the most extreme case in the lineup. It totals 2.8 trillion parameters, activates roughly 104 billion per token (16 of 896 experts selected), and carries a 1M-token context window. AirLLM handles it not layer by layer but per-expert streaming: a mixture-of-experts structure routes each token through only a few experts, so the remaining thousand-plus stay parked on disk. Fitting 2.8T into 3.72GB costs one thing, and that is an enormous amount of disk reads per token.
Kimi K3's own license deserves a plain statement: it uses a modified MIT license, not the standard MIT text, so check the original LICENSE yourself before redistributing or using it commercially. The environment requirements are hard constraints as well. K3's model code forces flash attention, requiring compressed-tensors and flash-attn, plus a CUDA 12 build of torch, since flash-attn ships no prebuilt wheels for CUDA 13. And the realistic prerequisites should be spelled out: K3 launched in July 2026 and opened its full weights later that month, and holding 2.8 trillion parameters on a single node means a terabyte-scale disk bill. AirLLM pushes the VRAM requirement to 3.72GB, but not the storage requirement to zero. The phrase "2.8T runs in under 4GB" describes VRAM, not the whole machine.
The Honest Cost: Speed Is How This Bill Gets Paid
Every number above points at memory. Not one of them points at speed. That gap has to be closed explicitly.
First, this trades speed for VRAM. Layer-by-layer transfer means every forward pass reads weights from disk or host memory into VRAM, so PCIe bandwidth and memory bandwidth set token generation speed directly. On consumer hardware the resulting experience typically lands in the minutes-to-hours range per interaction, nowhere near what interactive dialogue feels like.
Second, the official numbers are memory, not speed. The README contains no tokens-per-second value, and no third party publishes an apples-to-apples benchmark, so any specific throughput figure you encounter is an extrapolation rather than a measurement.
Third, the ecosystem has real rough edges. The GitHub API shows 160 open issues on the repository right now. For a project sitting at 36,747 stars and still being pushed, that is a high number, and it says the matrix of versions, model adaptations, and hardware combinations forks far more widely than a single page of documentation suggests. Worth flagging separately: the repository's primary language on GitHub is Jupyter Notebook, not Python or C++. For an inference framework doing heavy computation, that usually means the core implementation lives in notebook-shaped examples and docs rather than a polished installable package, so treating it as a research tool rather than a production component is the more realistic frame.
Fourth, disk is the hidden threshold. Qwen3.8-Flash-Next needs roughly 360GB on disk, and Kimi K3 is a 2.8T-scale weight set, so terabyte-class storage is the real prerequisite. The "4GB of VRAM" figure sounds light; the storage cost behind it is not.## Training Support: Frozen Weights Stream Too
A September 2026 update carried the same idea to the training side: frozen weights still stream in layer by layer, while only the trainable adapter stays resident in VRAM. The README reports training a 125B model in under 6GB VRAM on an RTX 3060 Ti, and training a 27B model in roughly 2GB VRAM at sequence length 512.
That drops the hardware floor for fine-tuning considerably, but the boundary needs stating: this path applies to adapters and similar small trainable parameter sets, not to full-parameter training of a 2.8T model inside 6GB of VRAM. The frozen weights still have to stream off disk, so I/O pressure is unchanged.
Where It Sits in the Ecosystem
Boundaries against our existing coverage should be drawn clearly. We previously published a comparison of local LLM deployment tools (Local LLM Deployment Tools Compared), which pits Ollama, LM Studio, vLLM, llama.cpp, and GPT4All against each other. AirLLM changes the axis entirely: it asks which memory strategy to use when the model will not fit, not which tool to pick. The two read well back to back.
Placed in the full strategy map, AirLLM occupies the "smallest VRAM, slowest speed" end. The other end compresses the model until it fits, and the GGUF quantization route has the most mature ecosystem — installation and pitfall-avoidance live in our Ollama Local LLM Deployment SOP. Between the two sit CPU offload hybrids that keep some layers on the card and push the rest to host memory, with speed governed by PCIe and memory bandwidth; distillation into small models, trading away flagship capability; or simply calling a remote API, where the VRAM problem does not exist — a route covered in the hosted-API layer of our Qwen3.8-Flash-Next Three-Tier Deployment SOP.
If your goal is not "see if it runs" but a coding assistant you use daily, then model quality is the deciding factor, and our Local Coding Model Comparison covers selection. If you want to start at a specific scale, the Mellum 2.1 Local Deployment SOP and the LingBot World Small Local Deployment SOP walk through two different sizes end to end.
Who Should Use It
Three kinds of people fit. First, anyone who wants to touch a model far too large for their hardware, running it once just to see the output — evaluating an ultra-large model's reasoning quality, or processing one long document. Second, anyone doing runnability validation, needing to confirm on constrained hardware that a particular checkpoint loads and emits tokens. Third, anyone studying layer-wise streaming itself, for whom this implementation is excellent reference code.
Two kinds of people do not fit. Anyone needing per-token interaction, such as real-time conversation or interactive code completion. And anyone whose disk and host memory are also tight: AirLLM moves pressure from VRAM onto storage without conjuring resources from nothing. There is also a time threshold worth computing before you start. The first deployment downloads and splits a checkpoint; Qwen3.8-Flash-Next at roughly 360GB and K3 at terabyte scale means downloads measured in hours, and interrupted transfers are not rare.
FAQ
Q1: Is AirLLM a quantization scheme?
No. The README is explicit that it does not quantize, does not distill, and does not prune; weights load in their original precision, streamed from disk into VRAM layer by layer. Its memory numbers look better than quantized alternatives for one reason: the model itself was never compressed, so the entire cost moved to time.
Q2: What speed does the project actually report?
It does not. Every number in the README is VRAM usage — 3.33GB, 5.95GB, 3.72GB, roughly 12GB — and the training figures are memory rather than throughput as well. No third party publishes an apples-to-apples tokens-per-second benchmark either, so any specific throughput number is extrapolation, not measurement.
Q3: Can Kimi K3 run under AirLLM?
Yes. The README records a 3.72GB VRAM end-to-end measurement on an RTX 6000 Ada, using per-expert streaming with per-token routing. Keep in mind that Kimi K3 itself uses a modified MIT license rather than standard MIT, that it mandates flash attention, and that the environment needs a CUDA 12 build of torch plus transformers 4.56.x.
Q4: Why does it still fail after I follow the README exactly?
The most frequent cause is a version matrix mismatch: Qwen3.8-27B needs transformers 5.8 or newer, Qwen3.8-Flash-Next needs the in-tree qwen4_exp support from transformers (installed from source), and Kimi K3 needs 4.56.x plus compressed-tensors and flash-attn. Next most common is disk and host memory: checkpoints run to hundreds of gigabytes or terabytes, and the n-gram table mapping assumes 64GB-class host memory.
Q5: With AirLLM available, do I still need tools like Ollama?
The two routes solve different problems. AirLLM keeps the model intact without quality loss and pays in speed; the GGUF quantization route (Ollama, llama.cpp) compresses weights for memory and delivers usable speed with some quality loss, and its ecosystem is by far the most mature. Choose by scenario: use AirLLM to run a huge model once in a while, and a quantized solution for daily interaction.
Closing
AirLLM demonstrates one thing: when VRAM runs short, the third option is "slow." It converts large models from "must be resident in VRAM" to "take turns being resident, layer by layer," decoupling parameter count from memory requirement almost entirely. The project is still actively maintained, but 160 open issues and per-model version requirements show this path's engineering cost is real. If you try it, starting with the 3.33GB Qwen3.8-27B is more realistic than starting with 2.8T.