What This Route Actually Solves
Let us be clear about one thing first: AirLLM is not a quantization tool, not a distillation tool, and not a pruning tool. It does exactly one thing: it treats every layer of the model as a separately movable file, loading the layer it needs onto the GPU from disk or host memory, computing that layer, and swapping it back out. In the README's own words this is layer-wise streaming, and it is what lets a 70B-class model fit on a single 4GB card - and even puts a Kimi K3 (2.8T) class model under the 4GB mark.
Why is this worth a full SOP? Because it is the only option on the "not enough VRAM" list that does not degrade model quality. GGUF quantization costs quality, distillation trades away capability, CPU offload hybrids remain hostage to memory bandwidth. AirLLM concentrates the entire cost into a single axis: speed. Slow is acceptable. Slow and wrong answers is not.
The repository is github.com/lyogavin/airllm, under the Apache-2.0 license. Verified against the GitHub API on 2026-10-12: 36,747 stars, 3,829 forks, 160 open issues, last pushed 2026-10-12, with training support added in a 2026-09 update.
Worth separating from the other pieces on this site. Our local deployment tool comparison benchmarks five deployment tools; the Ollama local deployment SOP covers quantization tiers and VRAM pitfalls; the Qwen3.8-Flash-Next deployment SOP maps that model from hosted API to on-device. This article is a completely different branch: no quantization, no distillation, brute-force VRAM savings through layered streaming.
Boundary Statement
Every official AirLLM number is a VRAM footprint, not a speed figure. No hands-on benchmarking was done for this article, so no tokens-per-second number and no time-to-first-token number appears here. The speed discussion is limited to the structural reason for the slowness, because that is design, not a bug.
Step 1: Install the Environment and Lock the Versions
AirLLM's setup is more involved than most people expect, because three interlocking components have to agree. Get this wrong and every later step fails, usually with error messages that do not name the real cause.
The first lock is that torch must be a CUDA 12 build. This is a hard constraint: flash-attn has no prebuilt wheel for CUDA 13, and torch's build tracks CUDA, so a CUDA 13 torch means flash-attn will not compile.
The second lock is flash-attn. Kimi K3's model code mandates flash attention, and the install command is verbatim from the README:
pip install compressed-tensors flash-attnThe third lock is that the transformers branch depends on the model, and this is the one people miss. Qwen3.8-27B needs transformers 5.8+. Kimi K3 needs transformers 4.56.x. Qwen3.8-Flash-Next, a Qwen4ExpForConditionalGeneration model, needs the in-tree qwen4_exp from transformers main, installed straight from source: pip install git+https://github.com/huggingface/transformers.git.
These are not three versions of one requirement. They are a matrix, one cell per model, so running 27B and then K3 in one virtual environment means switching transformers versions in between.
Verify twice, because one check misses half the failures. First, the driver and hardware layer:
nvidia-smiThat confirms the card is present, the driver version, and current VRAM occupancy. Second, whether torch can actually see the card:
python -c "import torch; print(torch.version.cuda, torch.cuda.is_available())"The torch.version.cuda value should read 12.x. If it prints 13.x, the first lock already slipped - reinstall now while it is still cheap. A False from torch.cuda.is_available() points at the driver or the environment, not at AirLLM.
Expected result: nvidia-smi lists the GPU, and the second command prints something like 12.4 True.
Failure signal: torch.cuda.is_available() returns False while nvidia-smi looks healthy - a PyTorch-driver mismatch, not an AirLLM bug.
Step 2: Pick a Target From the Measured Table
Count your card's VRAM, then read the README's measured table.
| Target | VRAM footprint (README) | End-to-end test hardware | Extra hard dependencies |
|---|---|---|---|
| Qwen3.8-27B (dense VL, Gated DeltaNet + Gated Attention) | 3.33GB | RTX 3090 | transformers 5.8+ |
| Qwen3.8-Flash-Next (125B, with roughly 51B n-gram embeddings) | 5.95GB | RTX 4090 | in-tree qwen4_exp; checkpoint about 360GB |
| Kimi K3 (2.8T total, 104B active) | 3.72GB | RTX 6000 Ada | compressed-tensors, flash-attn, transformers 4.56.x |
| DeepSeek-V3 (671B) | about 12GB | README figure | - |
| 70B-class models (headline claim) | 4GB or less | README figure | - |
Qwen3.8-27B at 3.33GB is the most realistic starting tier. Three reasons: the footprint fits what a 4GB card can absorb, the checkpoint is tens of gigabytes rather than hundreds, and it needs one dependency lock instead of three.
Kimi K3's 3.72GB is seductive, but the bill comes due elsewhere: 2.8T parameters, 896 experts with 16 activated per token, a 1M token context, and a TB-scale disk footprint for a single node. Its weights also carry a modified MIT license, not the standard MIT, open-sourced in full on 2026-07-27 after a 2026-07-17 release. Read the LICENSE text yourself rather than assuming the terms you remember.
The Qwen3.8-Flash-Next route has its own quirk: the n-gram embedding tables are mapped to host memory, with the README stating a 64GB host is sufficient, and the decoder layers streamed. Its demand on system RAM is therefore stricter than its demand on VRAM.
Step 3: The First Inference, Where Download Time Dominates
Layered inference presupposes that every layer already exists locally as a file. The first run downloads the full checkpoint first: roughly 360GB for the Qwen3.8-Flash-Next checkpoint, TB-scale for Kimi K3. Interruption partway through is close to inevitable, so two disciplines:
- Do not touch the cache directory once the download completes. Losing an HF cache to a cleanup means re-fetching hundreds of gigabytes. Treat the cache path as a fixed asset, not a scratch directory.
- Reclaim the original with
delete_original=Trueduring the split phase. The parameter deletes the source checkpoint after AirLLM slices it into per-layer files, which for a 360GB-scale model is the single biggest disk reclaim available. It is also irreversible - confirm the split completed and loading works before letting it delete, or you end up with no original and no complete split.
Expected result: the first inference is preceded by a download measured in hours, or days at TB scale. Failure signal: the download keeps breaking at the same point. Before blaming the network, check whether your disk has room for the full checkpoint plus temporary files.
Step 4: Watch VRAM and Manage Speed Expectations
Watch VRAM with nvidia-smi in a second terminal. The peak is the only reliable evidence of whether your card is sufficient; the README figures are the maintainers' baseline on their own hardware, and your peak will not necessarily match.
On speed, this article gives you no number, because the official source gives none either. Inventing a tokens-per-second figure is the easiest way to fool yourself. What is real is the structural slowness: every token walks a cycle of load layer, compute, evict layer, and the bytes moved dwarf the arithmetic performed. Time to first token is therefore noticeably worse than a conventional model, and it is wrong for token-by-token interactive use. Batch tasks that produce a complete answer at once are the shape it is built for.
The test is simple: if you can accept hitting send, making coffee, and coming back to read the result, AirLLM fits. If you need a live conversation, it is the wrong tool.
Step 5: Wiring Up Agents and APIs, and the Training Fork
On the inference side there are two typical usages. The first is one-shot task workflows: feed in a batch of material, get back a report or an analysis, no dialogue in the middle. This is where AirLLM's value is clearest, because it puts a model tier on a 4GB card that was previously out of reach.
The second is wrapping it as an endpoint for an agent, with one caveat: the agent's interaction model must tolerate long latency. If an agent assumes every reply lands in a few hundred milliseconds, attaching AirLLM degrades its behavior - not from wrong answers, but from not being able to wait.
Training is a different path entirely. The README's 2026-09 update describes training as streaming frozen weights layer by layer while keeping the adapter resident in VRAM: only the small adapter parameters (LoRA and friends) stay on the card while the large weights keep flowing from disk. Two README figures: Qwen3.8-Flash-Next (125B) training under 6GB on an RTX 3060 Ti, and Qwen3.8-27B training around 2GB at sequence length 512. This logic differs entirely from inference and needs its own evaluation.
If you end up torn between quantizing and layered streaming, the AirLLM resource page maps the VRAM and deployment routes; the local coding model comparison picks a model by task type, and the Mellum 2.1 local deployment SOP is a far lighter parameter count.
Five Pitfalls
Pitfall one: treating the version matrix as a single lock. CUDA 12 build, flash-attn, and the matching transformers branch are three independent constraints with three different failure points. A CUDA 13 torch build breaks the flash-attn compile. transformers 5.8+ cannot find Kimi K3's model code. A stable release cannot supply qwen4_exp at all, because that module only exists in-tree on main. The three errors look nothing alike but share one root cause, so check the matrix first.
Pitfall two: treating the first download like a normal model download. 360GB is the Qwen3.8-Flash-Next scale; Kimi K3 is TB-scale. Bring the patience of a 7B download and you will interrupt it, clear the cache, and lose hundreds of gigabytes for nothing. There is exactly one correct posture: ample disk, ample time, an untouched cache.
Pitfall three: assuming slow means a broken install. This is the most common misdiagnosis. When a single token takes seconds, the first reaction is that something is misconfigured - yet layered streaming simply is that speed, and the maintainers never promised interactive latency. If you start doubting your environment, reread Step 4.
Pitfall four: treating training as an accessory of inference. Training is frozen-weight streaming with the adapter resident; inference is per-layer streaming of the full weights. Conflating them leads to false assumptions about VRAM budget, load timing, and optimizer state.
Pitfall five: reading K3 as a free lunch for "2.8T on 4GB." The VRAM figure is real, but the cost lives elsewhere: TB-scale disk, hard flash-attn dependencies, and a modified MIT license, which can differ from the standard text on redistribution terms.
Setup Checklist
nvidia-smilists the target GPUtorch.version.cudaprints 12.x andtorch.cuda.is_available()is True- flash-attn installs cleanly (K3 also needs compressed-tensors)
- transformers version checked against the matrix: 5.8+ for 27B, 4.56.x for K3, in-tree qwen4_exp for Flash-Next
- Target model selected, with disk space for the full checkpoint plus temporary files
- Cache directory protected;
delete_original=Trueenabled only after the split is verified nvidia-smirunning alongside to watch peak VRAM- Speed expectations calibrated to "structurally slow," validated with a one-shot task
- If running K3, the modified MIT license text has been read
- Training needs evaluated separately, without reusing inference assumptions
One-line close: AirLLM turns "not enough VRAM" from a dead end into a trade - no quantization, no distillation, layer by layer from disk, with no quality loss, and the entire bill charged to speed. So the first question is not whether your card can hold 3.33GB. It is whether your disk can hold 360GB first, and whether your workflow can tolerate hitting send and walking away.
FAQ
Q1: How fast is AirLLM? Is there an official tokens-per-second figure? A1: No. The official README publishes VRAM footprint numbers only - no tokens per second, no time to first token. The structural reason for the slowness is certain though: every token runs a load, compute, evict cycle whose bytes moved vastly exceed the arithmetic performed. It suits one-shot workflows, not token-by-token interaction.
Q2: Why must torch be a CUDA 12 build? Can another version work?
A2: Because flash-attn has no prebuilt wheel for CUDA 13, and torch's CUDA build version is bound to it. Build torch against CUDA 13 and flash-attn will not compile. Verify by printing torch.version.cuda in the environment - it should read 12.x. Do this before installing flash-attn rather than debugging a failed compile afterwards.
Q3: What license does K3 carry? Can it be used commercially? A3: Kimi K3's weights carry a modified MIT license, not the standard MIT; the binding terms are the LICENSE text in the repository and on the model page, so do not assume the standard MIT wording applies. Read and archive the original before any commercial use. Also price in the operational cost: 2.8T total parameters, 104B active, 896 experts with 16 activated per token, a 1M context, and a TB-scale single-node disk footprint.
Q4: Can one machine run both 27B and K3? A4: Not from a single virtual environment. The transformers branches differ - 27B needs 5.8+, K3 needs 4.56.x. Switch versions between runs, or build two environments. Qwen3.8-Flash-Next is the most demanding case: it requires the in-tree qwen4_exp from transformers main.
Q5: Can a 4GB card really run a 70B or a 2.8T model? A5: The README's measured table gives specific figures: Qwen3.8-27B at 3.33GB end-to-end on an RTX 3090, Qwen3.8-Flash-Next 125B at 5.95GB on an RTX 4090, Kimi K3 2.8T at 3.72GB on an RTX 6000 Ada, and DeepSeek-V3 671B at roughly 12GB. The 70B-under-4GB claim is the README headline. All of these are VRAM footprints, and each holds only if your disk, your time budget, and your patience for speed are large enough.
References
- Official repository (GitHub API, verified 2026-10-12): https://github.com/lyogavin/airllm - Apache-2.0, 36,747 stars, 3,829 forks, 160 open issues
- Official README: https://github.com/lyogavin/airllm/blob/main/README.md - measured VRAM table, transformers branches,
delete_original=True, training figures - Related reading: the sibling piece from this batch, AirLLM resource page; on this site, local deployment tool comparison, Ollama local deployment SOP, Qwen3.8-Flash-Next deployment SOP, local coding model comparison, Mellum 2.1 local deployment SOP
All commands and figures here are a 2026-10-12 snapshot of the official README, not benchmarked on real hardware; no speed data is published officially, so no estimate is offered. Not investment advice.