Weights Downloaded, Now What? A VRAM Planning Guide

Every few weeks another wave of open weights lands: a few hundred billion parameters, sometimes trillions, released for anyone to download. Many people pull the weights to a local machine and get stuck at the same place: it will not run. The problem is not enthusiasm or bandwidth, it is an account that was never tallied, the VRAM account. The weight files are only the first page of that ledger; real memory usage also includes the KV cache, activations, and the runtime itself. Until this account is worked out, every decision about buying GPUs, choosing a quantization level, or setting context length is a guess.

This SOP lays the account open. First, what actually lives in VRAM and how much each piece takes. Then a formula you can apply directly, anchored to a Qwen3-32B comparison table. Then the three most common misjudgments, how to pick precision for your GPU, and how to pick a toolchain for your scenario. It closes with a five-step checklist that runs from reading config.json to load-testing before production. After reading, you should be able to answer, before downloading a single gigabyte: can this card run this model, at what size, with how much concurrency?

One standard applies throughout: there is no "best" local deployment, only deployments whose math closes. The same card supports different models depending on precision, context length, and concurrency target. Every selection decision should be preceded by this VRAM calculation and followed by a load test; decisions made on feel get paid back in out-of-memory errors.

Three ledgers: weights, KV cache, and runtime overhead

Everything in VRAM can be split into three ledgers. Miss one and the deployment fails.

Ledger one is weights. Weight size equals parameter count times bytes per parameter: BF16 and FP16 use 2 bytes per parameter, FP8 uses 1 byte, and INT4 is nominally about 0.5 bytes, though grouped quantization carries scale factors that push the real figure slightly higher, usually estimated at 0.5 to 0.6 bytes. A 32B model at BF16 is therefore roughly 66GB, which is why a single consumer card and a BF16 32B model are never going to happen.

Ledger two is the KV cache, the most frequently missed line. Deployment engineers share an empirical formula in technical communities: KV cache per token equals 2 times the number of layers times the number of KV heads times head_dim times bytes per value, where the leading 2 accounts for keys and values. Two properties matter. The cache grows linearly with context length and concurrency, and it is completely independent of weight quantization. Compress the weights to 4-bit and the cache still expands at FP16 speed; in long-context, multi-user scenarios it can outweigh the weights. Qwen3-32B's cache at FP16 is roughly 256KiB per token, so a single request with 32K context needs about 8.6GB, and ten concurrent requests need about 86GB, more than the BF16 weights themselves. Switching the cache to FP8 halves it: 4.3GB per request, about 43GB at ten concurrent.

Ledger three is runtime overhead. CUDA context, activations, and memory fragmentation do not care which model you run; reserve 4 to 12GB conservatively. After summing all three, apply a 1.1 to 1.3 headroom factor. The working rule: keep weights plus cache plus activations within 70 to 80 percent of usable VRAM and leave the rest for spikes and fragmentation.

Stack the three ledgers together and "how big is the model" stops being a meaningful question. The correct question is "how big, what context length, how many concurrent requests, what precision." The same 32B model at 32K context for one user is one machine; at 128K with eight concurrent users it is an entirely different one. The difference lives in the KV cache and concurrency headroom.

A formula you can use directly, plus one comparison table

The following Python sketch puts all three ledgers in one place. Pull the numbers from the model's config.json and apply:

# From config.json: num_hidden_layers, num_key_value_heads, head_dim
# plus total parameter count params_b (in billions)

BYTES_PER_PARAM = {"bf16": 2.0, "fp8": 1.0, "int4": 0.55}

def plan_vram(params_b, layers, kv_heads, head_dim,
              precision="bf16", kv_dtype=2.0,
              ctx_len=32768, concurrency=1):
    # Ledger 1: weights (GB)
    weights = params_b * BYTES_PER_PARAM[precision] / 1024**3
    # Ledger 2: KV cache (GB), per token = 2 x layers x kv_heads x head_dim x bytes
    kv_per_token = 2 * layers * kv_heads * head_dim * kv_dtype
    kv_total = kv_per_token * ctx_len * concurrency / 1024**3
    # Ledger 3: runtime overhead, conservative midpoint 8GB
    overhead = 8.0
    # Headroom: multiply everything by 1.2
    return (weights + kv_total + overhead) * 1.2

# Qwen3-32B, BF16 weights, FP16 cache, 32K single request
print(plan_vram(32.8, 64, 8, 128))   # ~66GB weights + ~8.6GB cache + overhead
print(plan_vram(32.8, 64, 8, 128, precision="fp8", kv_dtype=1.0))  # much lower

Qwen3-32B (about 32.8B parameters, Apache 2.0) anchors the comparison table, with figures aggregated from measured deployments and community estimates:

One counterintuitive row: GGUF Q8_0 at about 35GB is slightly larger than FP8 at 33GB, because GGUF's per-layer scaling structure carries a little extra storage. Do not pick purely by the smallest number; check which format your inference stack actually consumes.

This same math makes "open weights does not mean runnable" concrete. Kimi K3.1 Pro has about 2.8T parameters: roughly 5.6TB at BF16, 2.8TB at FP8, and even INT4 still lands around 1.4TB. No single consumer machine has a deployment path to that, no matter how aggressively you quantize or offload. Open weights solve acquisition; the compute threshold is still yours to reckon. Enterprise planning guides add one more commonly cited figure: a 70B-class model with 32K context needs about 10GB of KV cache for a single request, before weights and overhead.

Open config.json before you download anything. Layers, KV heads, head_dim, and parameter count are all in there; five numbers and two formulas tell you in ten minutes whether a card qualifies, far cheaper than a failed 60GB download. KV structure also varies across models: architectures using grouped-query attention have far fewer KV heads than attention heads, which is why models with similar parameter counts can have very different memory appetites.

Three common misjudgments (loading is not the same as serving)

Misjudgment one: counting only weights, not the KV cache. This is the most common beginner error and the most expensive. As computed above, 32B at 32K context with ten concurrent users needs about 86GB of cache against 66GB of BF16 weights; the cache matching or exceeding the weights is normal in long-context scenarios. Buy hardware on weights alone and the service hits out-of-memory the moment real concurrency arrives, usually only under load testing.

Misjudgment two: treating "it loaded" as "it works." A successful vLLM startup only proves the weights fit. The truth is in the startup log: GPU KV cache size is printed there directly, and the maximum concurrency derived from it is the machine's real capacity. If the log shows a few hundred MiB of cache and single-digit concurrency, that "successful startup" has no production value. The judgment standard is always the measured value under load, not a process that did not crash.

Misjudgment three: ignoring storage and network bottlenecks. Pulling 140GB of weights over 1GbE takes about 22 minutes of pure transfer, which makes cold starts miserable. In multi-GPU deployments, tensor-parallel all-reduce is the real bulk of east-west traffic; plan PCIe bandwidth and NVLink topology or more cards will not mean more speed. Local deployment also has a data-plane meaning: physical network isolation, weights pulled once and run offline, authenticated APIs, and logged calls all belong in planning.

Choosing precision: the ladder from BF16 to 4-bit

The precision ladder is not "lower is worse"; it is a map you navigate by GPU.

24GB cards (the 4090 class): 8B-scale models run best at BF16 or FP8 with an FP8 cache. For 14B to 32B you are limited to 4-bit quantization (NVFP4, AWQ, GPTQ), and context length is constrained. A 32B model quantized this way weighs about 17 to 20GB, leaving almost no room for cache; fine for personal experimentation, not for concurrent serving.

96GB-class setups: a 70B dense model at FP8 delivers quality close to BF16 with cache to spare. Large MoE models are better served by NVFP4 or MXFP4.

Architecture is a hard constraint too. Blackwell (compute capability 12.0) natively supports FP8 and FP4, so the full ladder is available. Ampere (8.0/8.6) can only do weight quantization; there is no FP4 option. Forcing new precisions onto older silicon fails loudly or quietly collapses performance, so check your card's compute capability before choosing.

One-line principle: if the card cannot fit BF16, do not force it. Prefer official FP8; beyond that, AWQ/GPTQ/GGUF Q4. Validate each step down against your own business data before committing.

Precision and cache are separate decisions. Weights at 4-bit with an FP8 cache is a legal combination and a common production pattern; the reverse works too. The two ledgers are billed independently, so never assume the cache was compressed just because the weights were quantized; both vLLM and Ollama require explicit cache arguments.

Toolchain: serving stack versus personal machine

For serving, the choices are vLLM or SGLang, both exposing OpenAI-compatible APIs. vLLM accepts --kv-cache-dtype fp8 to shrink cache entries to 1 byte per value, halving cache memory; note that without calibration the scales default to 1.0, so production deployments should calibrate with llm-compressor first. SGLang is an equally mainstream option with different throughput characteristics; let your own load tests decide. Both frameworks slice the cache once at startup based on --max-model-len and free VRAM, and never grow it at runtime, so the planning math has to be right beforehand.

For personal machines and workstations, Ollama or llama.cpp with GGUF is the default: the widest format ecosystem and GPU-plus-RAM layer offloading. Setting OLLAMA_KV_CACHE_TYPE=q8_0 in Ollama halves the cache, and q4_0 cuts it to roughly a quarter, but Flash Attention must be enabled or quality degrades noticeably.

Two more paths exist for constrained hardware. KTransformers mixes CPU RAM with GPU VRAM to fit larger models into limited hardware. Strata 0.2.10 targets small VRAM with long contexts: per its release notes, it dynamically shrinks the KV cache window against real-time VRAM headroom and spills to RAM or disk when necessary; an 8GB card can launch Qwen3.8 IQ3_S at 128K context, and its default DFlash2 speculative decoding cuts draft memory from about 5.2GB under MTP to about 625MiB, generating 63 to 73 tokens per second on an 8GB card. These extreme schemes trade throughput and latency stability for feasibility; they suit single-user exploration, not production.

A five-step checklist (with commands)

Step one, read config.json before downloading. Pull layers, KV heads, head_dim, and parameter count, run the three ledgers through the formulas, and only then decide whether to download. Never start from instinct.

Step two, set context and concurrency. Size the window to the longest real input and output in your workload; nine out of ten production workloads never need 128K, and 32K is a sensible starting default. Estimate concurrency from real peaks, not aspirations.

Step three, choose precision. Use the ladder and the table against your GPU and model size: official FP8 first, then AWQ/GPTQ/GGUF Q4.

Step four, start the service and load-test against the logs:

# vLLM: serving deployment, FP8 halves cache, cap concurrency memory
vllm serve Qwen/Qwen3-32B-FP8 \
  --kv-cache-dtype fp8 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.9

# Ollama: personal GGUF deployment, q8_0 halves cache (needs Flash Attention)
OLLAMA_KV_CACHE_TYPE=q8_0 \
OLLAMA_FLASH_ATTENTION=1 \
ollama serve

Watch the measured GPU KV cache size in the startup log, derive maximum concurrency from it, and stress-test with real traffic instead of accepting "it started."

Step five, pin versions and lock down operations. Fix model and framework versions, verify weight hashes, monitor the VRAM curve with nvidia-smi or Prometheus, and run a one-week canary with a fixed regression set before every upgrade.

Each checklist item maps to a real failure mode: downloading without calculating is the first crash, launching without load testing is the second, and getting broken by an untested upgrade is the third. Finish all five steps and you will hold three numbers, actual weight footprint, measured cache size, and real per-card concurrency, recorded so every future sizing discussion has evidence.

FAQ

Q1: What is the biggest model my 24GB card can actually run? A: 8B-scale models run comfortably at BF16 or FP8 with an FP8 cache, with no pressure from a 32K single request. For 14B to 32B you must drop to 4-bit (NVFP4/AWQ/GPTQ); a quantized 32B weighs about 17 to 20GB, which leaves almost no cache headroom on 24GB, so plan for single-request short contexts rather than concurrent serving.

Q2: How much quality does quantization cost? A: FP8 is essentially lossless; a 70B dense model at FP8 on a 96GB-class setup scores close to BF16. INT4 methods (AWQ/GPTQ/Q4_K_M) are usually acceptable on general tasks but blur long-tail knowledge and code details, and grouped-quantization scale factors make real file sizes slightly larger than 0.5 bytes per parameter. The only trustworthy answer comes from regression testing on your own data; ignore every "lossless" claim.

Q3: Why does the KV cache outweigh the weights? A: Because the cache is independent of weight quantization. Compress weights to 4-bit and the cache is still billed at 2 bytes per value (FP16) or more, growing linearly with context length and concurrency. Qwen3-32B's cache is about 256KiB per token at FP16, so 32K context with 10 concurrent requests needs about 86GB, more than the BF16 weights. In long-context multi-user scenarios the cache is the dominant consumer.

Q4: Ollama or vLLM? A: Depends on the scenario. For a personal machine, single user, quick experimentation, pick Ollama or llama.cpp with GGUF, plus OLLAMA_KV_CACHE_TYPE=q8_0 to halve the cache. For production with concurrency and an OpenAI-compatible API, pick vLLM or SGLang with --kv-cache-dtype fp8 and calibrate via llm-compressor. A common path is validating the model with Ollama first, then migrating to vLLM.

Q5: Is 128K context worth enabling? A: For most production workloads, no. Enabling 128K means reserving four times the cache of 32K; for Qwen3-32B that is about 34GB per request at FP16, which likely wipes out your concurrency. Nine out of ten production workloads never approach 128K. Analyze the longest real inputs and outputs in your traffic first, and if long documents matter, prefer retrieval and chunking over maxing out the window.

Related reading

• LLM API Pricing Comparison: How the Token Math Works
• DeepSeek-OCR Local Deployment SOP
• AI Agent Guardrails Deployment SOP