中/EN
Field SOP
Field SOP

Nemotron 3.5 Bolt Local Deployment: 74B on a 24GB Card

NVIDIA released Nemotron 3.5 Bolt 74B-A9B 24GB on October 15, 2026: a compression of the 120B Super VL via Star Elastic budget-conditioned architecture search (expert top-K down from 22 to 9, distilled recovery training over 120B tokens), on a hybrid Mamba-2 and Transformer architecture. The quantization plan is the story - the expert layers that dominate the weights drop to IQ2_XXS at 2.0625 bits per element (about 19GB), Mamba linear layers use NVFP4, and lm_head, embeddings, and attention stay at Q8_0, landing the full model at 22-24GB with accuracy recovered through quantization-aware distillation from the compression stage onward. Four deployment routes, with commands: Ollama (supported since 0.40.5, ollama run nemotron-3.5-bolt, a 25GB download, verify 100 percent GPU via ollama ps); Unsloth (UD-IQ2_XXS about 22GB, UD-IQ1_M about 20.3GB, ctx from a conservative 4096); llama.cpp official recipes (builds after October 15 to recognize nemotron_h_moe; the 28GB recipe runs 65K context with 8-bit KV and 6 draft tokens, the 32GB recipe reaches 262K; llama-server defaults to port 9931 since b11521); and vLLM (Blackwell SM120 with the LiLiCorr drafter for multi-user serving). Two context starting points coexist: 16K-32K for bare weights, 4K for Unsloth whose default drafter eats another 1-2GB. Eight troubleshooting entries cover OOM context cuts, layers spilling to system RAM, the missing --mmproj flag, and the shared 16384 max_tokens budget. OpenMDW-1.1 grants free unrestricted commercial use, leaves outputs unencumbered, requires notices only on redistribution, and carries a patent-retaliation clause. Vendor-reported: GPQA Diamond 86.4, Terminal-Bench 2.1 at 51.0, PinchBench first in class (83.99 TPS on RTX Spark; no official consumer-card figure). The 4GB Bolt E4B followed on October 23.

Published October 23, 20268 min read
<!-- nemotron-3-5-bolt-local-deploy-sop | sop | Nemotron 3.5 Bolt Local Deployment: 74B on a 24GB Card -->

Running a genuinely capable agent model on your own desk usually means a compromise: cloud APIs are convenient but your code and data leave the building, while anything that fits in local VRAM is often too small for real work. NVIDIA's Nemotron 3.5 Bolt 74B-A9B 24GB, released October 15, 2026, attacks that gap: a Mixture-of-Experts model with 74B total parameters and only 9B active per token, squeezed onto a single 24GB card through a mixed-precision scheme that goes down to 2.0625 bits, and framed by NVIDIA as capable of agent work — tool calling, browser operation, multi-step tasks.

This SOP walks the math first (how 74B fits into 24GB at all), then gives real commands for four deployment routes — Ollama, Unsloth, llama.cpp, vLLM — plus a VRAM ledger, context strategy, troubleshooting, and the licensing picture. Numbers and commands come from NVIDIA's official technical page and model card; vendor-reported scores are labeled as such.

1. The math: how 74B fits into 24GB

Bolt is not trained from scratch; it is compressed. The parent is Nemotron 3.5 Super VL, a 120B-total / 12B-active multimodal model. NVIDIA's Star Elastic framework — budget-conditioned architecture search plus end-to-end compression — cut expert routing top-K from 22 to 9, then distillation recovery training ran on 120B tokens (72B at 8K context, 48B at 128K). Architecturally it mixes Mamba-2 layers with Transformer layers and carries a C-RADIO v4-H vision encoder: it reads text, images, and video, but outputs text only.

The VRAM ledger hinges on the quantization scheme. Expert layers dominate the parameter count, so NVIDIA compressed all routed and shared experts to IQ2_XXS at 2.0625 bits per element — 74B times roughly 2.06 bits is about 19GB. Mamba linear layers use NVFP4, while lm_head, embeddings, attention, and Latent MoE weights stay at Q8_0; the weight files land in the 22 to 24GB range. Accuracy lost to quantization is recovered with QAD (quantization-aware distillation), and only weights are quantized, because inference on RTX and DGX Spark hardware is weight-bandwidth bound.

The official pitch: a 2-bit 74B compressed model beats a smaller model trained from scratch and then 4-bit quantized in the same memory budget. Vendor-run scores: GPQA Diamond 86.4 (versus 83.4 for Qwen3.6-35B-A3B at 4-bit), Terminal-Bench 2.1 at 51.0, BrowseComp 66.8, WebArena 50.9; the stated target is decoding at 80 TPS or better inside 24GB while coexisting with everyday apps. PinchBench #1, WebArena #2, and BrowseComp #2 are also self-reported leaderboard placements with no third-party replications yet.

More key facts: native context is 262,144 tokens (a model-card ceiling; usable context depends on your VRAM); ten languages including English, Chinese, Japanese, Korean, French, German, and Spanish; post-training data cutoff of May 2026; and a 1.13B LiLiCorr drafter for speculative decoding (about 0.92GB on disk as GGUF). Under vLLM with 7 draft tokens at concurrency 64, greedy decoding accepts an average of 4.68 tokens per step — valid only under those conditions.

The full training pipeline has five stages: Star Elastic distillation-compression, vision adapter fine-tuning, heavy-vision SFT, RLHF plus end-to-end GRPO in MCP and SWE agent environments, then ModelOpt 2-bit quantization with QAD — roughly 26 trillion pre-training tokens and 3.8 million post-training samples. That pipeline answers the obvious question — why doesn't 2-bit quantization wreck the model? Because accuracy recovery started at compression time, carried by distillation throughout.

On speed, an official comparison is worth reading carefully. On an RTX Spark, AIPerf SPEED-Bench-Coding measured Bolt at 83.99 TPS, below Qwen3.6-35B-A3B at 96.63 and Gemma 4 26B-A4B at 100.56. But in agent replay (AA AgentPerf Local), Bolt finished in 600.80 seconds — better than Qwen3.8-27B at 1043.28, slower than Qwen3.6 at 453.97. Raw token throughput is not its selling point; stability over long agent chains is. All official speed figures come from RTX Spark hardware; no consumer-card TPS numbers have been published.

2. Pre-flight checklist

Match yourself to a hardware tier first:

  1. One 24GB card (RTX 4090 / 3090 class): workable, but weights fill the card. Prefer the smaller Unsloth UD-IQ2_XXS (~22GB) or UD-IQ1_M (~20.3GB) quants, start context at 16K-32K, and keep 64GB of system RAM as a backstop.
  2. 32GB VRAM: the llama.cpp official "28GB recipe" fits 65K context with 8-bit KV cache.
  3. 48GB+ VRAM, or an RTX Spark / DGX Spark with 128GB unified memory: the full 262K-context recipe is on the table.
  4. Mac: the official path is CUDA-first and Ollama does not support Apple Silicon; unofficial options are Unsloth Desktop (macOS) and community MLX conversions.
  5. 16GB VRAM or RAM or below: officially deemed unsuitable — pick a smaller model.

Two version gates: Ollama needs 0.40.5 or newer (support landed October 14, 2026); llama.cpp needs a build from October 15, 2026 or later, because older builds do not recognize the nemotron_h_moe architecture. Reserve about 25GB of disk, plus roughly 1GB for the drafter.

3. Four deployment routes

Route A: Ollama, the easy path

bash
# Requires Ollama >= 0.40.5 (supported since 2026-10-14)
ollama run nemotron-3.5-bolt
ollama ps

The official library carries three tags — latest, 74b-a9b, and 74b-a9b-iq2 — all a 25GB download with 256K context, vision, tools, and thinking mode. After it loads, ollama ps should report 100% GPU; a 15%/85% CPU/GPU split means layers spilled into system RAM and speed will suffer.

Session-time knobs for context and thinking:

text
/set parameter num_ctx 32768
/set nothink

/set nothink disables thinking for quick answers; /set think restores it. Images go in directly by path, such as a screenshot of an error message. To save VRAM, pull the Unsloth quant instead:

bash
ollama run hf.co/unsloth/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-GGUF:UD-IQ2_XXS

To wire it up as the brain of a coding agent:

bash
ollama launch claude --model nemotron-3.5-bolt
ollama launch opencode --model nemotron-3.5-bolt

Route B: Unsloth, the 24GB-card pick

bash
unsloth run --model unsloth/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-GGUF:IQ2_XXS --ctx-size 4096

You can also search "Nemotron 3.5 Bolt" in Unsloth Desktop and download IQ2_XXS directly. Two caveats: a small draft model ships by default for speed and costs an extra 1-2GB of memory, and the official guidance is 32GB RAM or unified memory to start, 48-64GB being comfortable. Context starts at 4096 per the conservative Unsloth/techpilot guidance — raise it only when memory allows.

Route C: llama.cpp, the NVIDIA official recipe

Build first. On Windows, set the CUDA architecture manually (RTX 50 series: 120, RTX 40: 89, Spark: 121):

bash
cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release -j

Download three files: the model, the vision projection (mmproj), and the drafter:

bash
pip install -U huggingface_hub
hf download nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF \
  NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf \
  mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf --local-dir ./bolt
hf download nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr-GGUF \
  NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr.gguf --local-dir ./bolt

The "28GB recipe" for 32GB cards (65K context, 8-bit KV, 6 draft tokens):

bash
./build/bin/llama-server \
  --model ./bolt/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf \
  --mmproj ./bolt/mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf \
  --spec-type draft-dflash --spec-draft-n-max 6 \
  --spec-draft-model ./bolt/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr.gguf --spec-draft-ngl 999 \
  --ctx-size 65536 --parallel 1 --batch-size 2048 --ubatch-size 512 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  -ngl 999 --flash-attn on --jinja --temp 1.0 --top-p 0.95 --port 8080

For 48GB+ cards or a 128GB Spark, swap in --ctx-size 262144, --spec-draft-n-max 7, --batch-size 4096, and --ubatch-size 2048. Three notes: recommended sampling is temperature 1.0 with top_p 0.95; the thinking toggle lives in the chat template's enable_thinking flag, with a medium_effort middle setting; and reasoning plus the answer share one max_tokens budget — official examples use 16,384, and a smaller value truncates answers. Also, since build b11521 (October 9, 2026), llama-server's default port changed from 8080 to 9931 — trust the port printed at startup and add --port if your toolchain hardcodes one. The OpenAI-compatible API lives at /v1.

Route D: vLLM, for servers and multi-user

vLLM requires a pinned nightly image. The target model supports Ampere (A100), Hopper (H100), and Blackwell, but adding the drafter pins you to Blackwell SM120 (including DGX Spark GB10). This route fits multi-user serving and video input — most chat frontends do not support video yet. Shortest smoke test:

bash
vllm serve "nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB"

The full launch command is long (fp8 KV cache, b12x kernels, dflash speculative config, and more); the model card carries it verbatim, so copy from there rather than reconstructing.

4. The VRAM ledger and context strategy

Runtime breakdown for the 28GB recipe on a 32GB card: weights roughly 22-24GB; mmproj a small extra file; the drafter about 1GB — the first thing to cut when memory is tight; KV cache growing with context, where 8-bit KV is what makes 65,536 context fit; and 1-2GB for the desktop. In practice, 64K context on a 32GB card consumed about 28GB — the recipe's name is the headroom.

On a 24GB card there are two starting points for context, and the reasons differ:

  • 16K-32K: computed for the bare model. Weights already fill 24GB, leaving little room for KV — the guidance for Ollama and raw-GGUF routes.
  • 4096: the conservative Unsloth/techpilot guidance, because Unsloth's default draft model adds 1-2GB, and long context plus image input is what pushes a 24GB card over the edge. Techpilot's phrasing: if it crashes or crawls, lower context before anything else.

A one-line mantra from the official community discussion: for long documents, drop the drafter and keep context; for quick chats, keep the drafter and cut context.

One more distinction: a 256K model-card ceiling does not mean you can open 256K. Context is shared across prompt, images, conversation, tool calls, and generation, and all of it costs VRAM. The hybrid Mamba-2 architecture helps — only a minority of attention layers carry a linearly growing cache, while Mamba layers need none — but helping is not free. Rule of thumb: before doubling context, keep at least 20% VRAM in reserve.

5. Troubleshooting list

  1. CUDA OOM: cut --ctx-size first, keep --cache-type-k/v q8_0, then drop the drafter (remove the --spec-* flags to save ~1GB).
  2. Generation far slower than expected: layers have spilled into system RAM. Check ollama ps for 100% GPU; on llama.cpp, lower context or close other VRAM-hungry apps.
  3. Images silently ignored: the --mmproj flag is missing from your llama.cpp command — the most common omission.
  4. Error about nemotron_h_moe: your llama.cpp build predates October 15, 2026; update it, and on Windows set CMAKE_CUDA_ARCHITECTURES explicitly.
  5. Cannot connect: check the port first — since b11521, llama-server defaults to 9931, not 8080.
  6. Answers cut off mid-stream: max_tokens is too small; reasoning and the answer share one budget, and official examples use 16,384.
  7. Long context plus image input on a 24GB card will crash: pick one, or move to more VRAM.
  8. Expectation management: 2-bit is a compromise. NVIDIA positions it as smart enough for most tasks but not its smartest model, and all rankings are vendor-reported — do not benchmark it mentally against flagship APIs.

6. The license: what OpenMDW-1.1 means

Bolt ships under OpenMDW-1.1 (published by the Linux Foundation on May 28, 2026; NVIDIA adopted it across the Nemotron family), and the model card states it is ready for commercial or non-commercial use. The clauses that matter:

  • Free, unrestricted use covering copyright, patent, database, and trade-secret rights — no field-of-use limits, no revenue thresholds.
  • Outputs are unrestricted: code and documents the model generates are yours to use.
  • Redistributing the model materials requires including the license copy and retaining all copyright and origin notices — this binds redistribution of the materials, not outputs.
  • Patent retaliation: initiating litigation claiming the materials infringe a patent or copyright (defensive countersuits excepted) terminates the grant.
  • No warranty; clearing third-party rights and due diligence are your responsibility.
  • Fine-tuning the BF16 weights and publishing the result is allowed, as is community requantization and repackaging.

The product line keeps expanding below Bolt: a smaller Bolt E4B (4GB class) is slated for October 23, 2026, targeting GeForce, Jetson, and RTX/DGX Spark for local voice assistants and embedded interfaces; its parameter counts are unpublished. The parent Super VL 120B targets enterprise-resident multimodal agents for OCR, tables, documents, and video, with 30+ companies post-training on it per NVIDIA — though claims that a fine-tuned version beats certain flagship closed models reflect specifically adapted tasks versus general models. Family division of labor: Bolt for everyday local agents, Lightning (30B-A3B) for high-throughput execution, Super for multimodal reasoning, Ultra for the hardest reasoning. Production serving also has a NIM container path with an OpenAI-compatible endpoint, outside this article's scope.

The repository: huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB, with the GGUF and drafter in the sibling -24GB-GGUF and -24GB-LiLiCorr-GGUF repos. Plain address once more for copy-paste: huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.

FAQ

Q1: Can 24GB of VRAM really run a 74B model, or is it a demo trick? A: It runs, but the math is tight: experts at 2.0625 bits come to about 19GB, total weights land at 22-24GB, and a 24GB card is left with almost no headroom. The recommended 24GB setup is the 22GB UD-IQ2_XXS quant with context at 16K-32K — agent tasks work, but forget opening the full 256K context too.

Q2: Does 2-bit quantization ruin the model? A: There is a capability trade-off; NVIDIA positions it as smart enough for most tasks but not its smartest model. QAD distillation recovers accuracy, and vendor-reported GPQA Diamond of 86.4 beats 4-bit smaller models in the same memory class — but no third-party replications exist, so run your own regression set first.

Q3: Can a Mac run it? A: The official path is CUDA-first and Ollama does not support Apple Silicon. Unofficial Mac options are Unsloth Desktop (macOS) and community MLX conversions, neither with official performance figures.

Q4: Does commercial use require paying NVIDIA or getting permission? A: No. OpenMDW-1.1 grants free, unrestricted use including commercial use, and outputs are unrestricted. The only obligation is including the license copy and retaining copyright and origin notices when redistributing the materials; mind the patent-retaliation clause, since suing over the materials terminates the grant.

Q5: Should a 24GB card start at 4K or 16K context? A: Both figures have reasons. The Unsloth route ships a draft model costing another 1-2GB, so official guidance starts at 4096; for bare GGUF or Ollama, where weights fill 24GB, 16K-32K is the start. The safe play is running at 4K first, then stepping up while watching VRAM headroom.

This article is AI-assisted and human-edited. Last updated: 2026-10-23

FAQ

Can 24GB of VRAM really run a 74B model, or is it a demo trick?
It runs, but the math is tight: experts at 2.0625 bits come to about 19GB, total weights land at 22-24GB, and a 24GB card is left with almost no headroom. The recommended 24GB setup is the 22GB UD-IQ2_XXS quant with context at 16K-32K — agent tasks work, but forget opening the full 256K context too.
Does 2-bit quantization ruin the model?
There is a capability trade-off; NVIDIA positions it as smart enough for most tasks but not its smartest model. QAD distillation recovers accuracy, and vendor-reported GPQA Diamond of 86.4 beats 4-bit smaller models in the same memory class — but no third-party replications exist, so run your own regression set first.
Can a Mac run it?
The official path is CUDA-first and Ollama does not support Apple Silicon. Unofficial Mac options are Unsloth Desktop (macOS) and community MLX conversions, neither with official performance figures.
Does commercial use require paying NVIDIA or getting permission?
No. OpenMDW-1.1 grants free, unrestricted use including commercial use, and outputs are unrestricted. The only obligation is including the license copy and retaining copyright and origin notices when redistributing the materials; mind the patent-retaliation clause, since suing over the materials terminates the grant.
Should a 24GB card start at 4K or 16K context?
Both figures have reasons. The Unsloth route ships a draft model costing another 1-2GB, so official guidance starts at 4096; for bare GGUF or Ollama, where weights fill 24GB, 16K-32K is the start. The safe play is running at 4K first, then stepping up while watching VRAM headroom.

Related

Field SOP

428B Weights Are Out, Budget 8 H100s: M3.1 Flash Local Deploy

A deployment log for M3.1 Flash (428B, 23B active). Cold water first: there is no consumer or single-GPU recipe anywhere official - the smallest validated tiers are 4x Blackwell (B200/B300/GB200/GB300, TP4), 4x AMD MI355X (TP4), or a full node of 8x H100/H200 (TP8, FP8 path). Weights come from HF or ModelScope (public, not gated); three size calibers coexist: community-count 252.4GB over 48 shards plus a 2.3GB DSpark head, and the vLLM recipe listing NVFP4 at 285GB, MXFP4 at 293GB, FP8 at 533GB; there is no official quantized repo. vLLM needs 0.33.0 or newer (absent from PyPI on release day, so use the vllm/vllm-openai:minimax-m31-flash Docker image or nightly); three official commands transcribed verbatim - H100/H200 TP8, GB200 TP4, MI355X TP4 - all forcing --block-size 128 and --disable-hybrid-kv-cache-manager (omit it and you get No common block size for 16); 1M context opens via --max-model-len (the Hopper sample caps at 262144). SGLang is not merged to main yet (Blackwell needs the MSA kernel at a pinned commit plus nvcc 13.4 minimum - cu130 does not count). The memory ledger: BF16 at roughly 856GB is an inference, not an official figure; 23B active lowers compute, not the VRAM floor - all 428B experts must stay resident, which is exactly why consumer cards are out; total KV cache for 1M context is unpublished. Domestic compute rides FlagOS Day-0 across nine chips (Huawei Ascend included, via SGLang-plugin-FL), numbers flagged as FlagOS official. Thirteen official error notes collected with fixes: HugeTLB 560GiB, B200 OOM, parser spelling, and more. License: MiniMax Community License, commercial attribution plus the 20-million-dollar threshold. One line of reality: 428B weights are out - budget 8 H100s.

Oct 24, 20269 min read
Field SOP

Xing4.0-29B-Intent Local Deploy: 4B Active, Which Command?

Getting Xing4.0-29B-Intent running hinges on a detail that trips most people up: the official deployment instructions do not live in the intent repository but in the base model repository. The intent README only says to replace the model path with the Xing4.0-29B-Intent weights and set --served-model-name to Xing4.0-29B-Intent, so the full vLLM and SGLang launch commands must be taken from the Xing4.0-29B-A4B base repo. Several flags are hard requirements: --trust-remote-code is mandatory because model_type is the in-house xing4_0 architecture, neither llama nor qwen; --tensor-parallel-size 2 implies at least two cards; --max-model-len 262144 matches the 256K context; --reasoning-parser xing4_0 and --tool-call-parser xing4_0 supply the matching parsers; --speculative-config uses the mtp method; and --gpu-memory-utilization 0.90 with --max-num-seqs 4 is the official example ratio. The weights ship as 41 safetensors shards totaling 62.43 GB, and the only official quantizations are FP8 (33.17 GB, e4m3) and GGUF (a single IQ4_NL file at 17.75 GB) - there is no AWQ, GPTQ, or INT8 safetensors. Minimum VRAM is not published at all; the official line is to configure GPU memory, parallelism, and context length for your environment. What the base repo does spell out: bfloat16 is required because the mHC path errors out on FP16 or quantized weights, --pipeline-parallel-size greater than 1 is not yet supported, the CUDA mHC path needs tilelang installed, and the host needs Docker 24.0 or newer with driver 535 or newer. The Ascend route explicitly wants 910B2 NPUs with at least two cards, with three official quay.io images (none of which bundle the weights). Consumer and CPU paths appear only in the base llama.cpp docs: int4 weights quantized with IQ4_NL come to about 19 GB and fit on a single consumer GPU, with an RTX 3090 24 GB as the example. The SDK is just an httpx HTTP client and is not on PyPI, and structured output comes in two layers - the model's native category plus args array, and SDK-guided JSON Schema constraints through vLLM response_format. The license is Apache-2.0, yet the weight side has no standalone LICENSE file, the repository LICENSE is an unfilled placeholder template, and the original terms of the 61 upstream datasets still need checking.

Oct 22, 20269 min read