Says #1, Its Own Table Says #2: Picking a 24GB Model
Running local agents on a single 24GB card: compress a big model to 2 bits, or let a natively small model stay at 4? The lead is NVIDIA's Nemotron 3.5 Bolt 74B-A9B 24GB, released 2026-10-15 and compressed from Nemotron 3.5 Super VL (120B-A12B) via Star Elastic, a router-guided architecture search that drives the MoE top-K down from 22 in the parent to 9 while targeting a deployment footprint below 24GB, followed by quantization-aware distillation to recover accuracy, with expert weights at IQ2_XXS, 2.0625 bits per element, plus a 1.13B LiLiCorr speculative drafter (seven drafted tokens, concurrency 64 in vLLM, mean acceptance length 4.68 greedy and 4.25 sampling). NVIDIA's design target is at least 80 tokens per second inside 24GB; AIPerf SPEED-Bench measured 84. Three comparisons: Qwen3.6-35B-A3B with 262K context, 3B active and Apache 2.0; Gemma 4 26B-A4B with 256K, about 4B active and Apache 2.0; and Nemotron 3.5 Lightning 30B-A3B, the vendor's own sibling, as the hallucination reference. Three curves contradict each other, and that is the spine of the piece. On intelligence per gigabyte Bolt leads: Artificial Analysis puts it at 21 on its Intelligence Index, fourth of 67 models in its comparison class, a second-hand account of AA's result and notably a measurement of the real 2-bit checkpoint rather than a BF16 reference build. On throughput it trails: NVIDIA's own table, taken with llama.cpp on Windows 11 ARM64 on a Surface Laptop Ultra with 128GB, shows 83.99 tokens per second on AIPerf SPEED-Bench-Coding against 96.63 for Qwen3.6 and 100.56 for Gemma 4, and end-to-end agent replay duration of 600.80 seconds against 453.97 and 480.68. On hallucination-linked metrics it also trails, with AA-Omniscience accuracy at 16.90 against Gemma 4's 22.17 and a non-hallucination score of 61.27 against its own Lightning 30B-A3B at 68.81. The leaderboard claim carries internal tension: the announcement calls Bolt the number one open model in its class on PinchBench, yet the same model card scores Qwen3.6-35B-A3B at 88.07 and Bolt at 87.82 - a quarter of a point, with the order coming from NVIDIA's own table. Two readings are laid out side by side rather than settled: if "in its class" means the 2-bit precision class then Qwen3.6's BF16 build is not a classmate, and if it means the 24GB memory footprint then the claim and the table disagree. The article does not accuse the vendor of fabrication. The licence is OpenMDW-1.1, granting use, modification and redistribution on condition of keeping the licence and notices, with rights ending if you sue claiming infringement. There is no hosted API, no per-token price and no third-party latency measurement. The conclusion splits by scenario: data that cannot leave the building goes to Bolt, throughput and concurrency go to Qwen3.6 or Gemma 4, and lower hallucination argues for re-evaluating Lightning. Division of labour with this site's October 10 local-coding comparison and October 12 VRAM-strategy piece: this one does not revisit memory tiers or VRAM-saving tricks, only the compression argument at 24GB.