On August 26, 2026, Qwen shipped two things on the same day: Qwen3.8-Flash-Next, a 125B multimodal MoE model, and FlashQLA, a kernel library written specifically for GDN. Put them side by side and the main battlefield of this year's architecture race comes into view. It is not parameters. It is not leaderboard scores. It is the attention cost of long context.
By August 2026, "supports 1M context" is table stakes for a flagship: Hy4 preview, Kimi K3, GLM-5.3-Flash, the DeepSeek V4 line, and the production Qwen3.8-Flash all claim 1M. But supporting 1M and affording 1M are two different things. Under dense attention, compute grows with sequence length and the KV cache piles up linearly with it -- running 1M used to double inference cost. This year the ways vendors compress that curve have split into three clear routes: picking tokens (sparse), compressing history (linear/recurrent), and reusing indexes (sharing sparse selection across layers). Which route a model takes decides the real VRAM and billing curve of your long-context workload.
Scope first: this is a representative comparison, not hands-on benchmarking. Architecture descriptions, parameters, and cost-reduction figures come from official blogs, READMEs, model cards, and public reporting, with the sourcing basis labeled per item (official release / media citation / our earlier fact-check / converted / engineering estimate), as of 2026-08-30. Derived figures such as activation ratios are converted or engineering estimates and exclude KV cache and runtime overhead. Prices and repo stats are snapshots; defer to official pricing pages and vendor sites. Any field the vendor has not disclosed is marked "Not collected" -- we do not infer or extrapolate. Not investment advice.
Division of labor first, so you do not mix up the ledgers: our lite flagship API price comparison covers the API unit-price and cache math of 320B-class models; our open-source flagship comparison covers the parameter, activation-ratio, and deployment-threshold math of 700B-2.8T models. This article covers the third ledger -- the attention architecture ledger: how each flagship drives down long-context attention cost, where the routes diverge, and what to look at when choosing. Three independent ledgers; read all three.
1. The Route Table: Five Flagships, Three Routes
| Model | Attention route | Mechanism | Disclosed cost reduction | Basis |
|---|---|---|---|---|
| Qwen3.8-Flash-Next (Alibaba) | Hybrid: linear compression + sparse selection | GDN compresses history; QSA uses a compressed lightweight indexer to pick important context at micro-block granularity, running full attention only on the selected part | Training cost about 1/9 of Qwen3.7-Plus (officially attributed to hybrid attention, Gated Residual, and Muon combined) | Official blog / README |
| Tencent Hy4 preview | Sparse selection + cross-layer index reuse | Gated DSA (64 heads, sparse index top-k 2048) plus IndexCache for cross-layer sparse index reuse; the README states it is inspired by DeepSeek and GLM | Not collected | Official README / press release |
| Zhipu GLM-5.3-Flash | Hybrid: sparse + linear attention | First open frontier model to adopt a hybrid sparse and linear attention architecture, paired with mHC manifold-constrained hyper-connections | Attention compute down 3.01x, KV cache down 4.44x (vs. GLM-5.3) | Official release / media citation |
| Kimi K3 (Moonshot) | Hybrid linear attention | KDA hybrid linear attention plus attention residuals | Not collected | Tencent Cloud Developer Community citation |
| DeepSeek V4-Flash | Not collected | Structure identical to the April Preview (284B / 13B MoE); this round reworked post-training only | Not collected | Official changelog |
Three first-glance takeaways:
- Pure sparse is down to one vendor; hybrids are the mainstream. Three models run explicit hybrids (Qwen's GDN+QSA, Zhipu's sparse+linear, Kimi's KDA), only Hy4 preview is pure sparse selection, and one has not disclosed. The reason is simple: sparse answers "which positions to compute," linear answers "how to store history," and the two cost curves do not overlap -- stacking them compresses both ends more easily than picking either alone.
- Only one vendor puts numbers on the savings. GLM-5.3-Flash reports 3.01x lower attention compute and 4.44x smaller KV cache (against its own GLM-5.3), the only verifiable quantitative basis among the five. Qwen's 1/9 is a training-side figure and the combined result of hybrid attention, Gated Residual, and Muon -- it does not mean "attention cost is 9x lower." Hy4 and Kimi K3 do not disclose attention savings. None of the numbers you can see is directly comparable across vendors.
- One architecture is still undisclosed. The DeepSeek V4-Flash changelog says only that structure and size are unchanged and post-training was reworked; the attention mechanism is not disclosed. That is a genuine transparency gap at selection time -- its price is very competitive (section 3), but you do not know which layer buys you that efficiency.
2. Mechanism Breakdown: Picking Tokens, Compressing History, Reusing Indexes
Route one: picking tokens (sparse attention) -- QSA and Gated DSA. Accept that not every historical token deserves the current step's full attention, use a lightweight indexer to pick the important ones, and run full attention only on those. Hy4 preview's Gated DSA uses a sparse index of top-k 2048 -- each step looks carefully at 2048 positions rather than all of 1M -- and IndexCache reuses the selection across layers so the layer below consumes what the layer above already picked. Qwen's QSA follows the same logic but pushes granularity down to micro-blocks and stresses that the indexer itself is compressed -- the picking step cannot be expensive either, or it eats back the attention it saved.
Route two: compressing history (linear/recurrent) -- GDN and KDA. Stop keeping the full KV sequence; compress the context that has already happened into a compact state that later steps read directly. That is what Gated DeltaNet (GDN) does. The cost is information loss, and quality depends heavily on the gating design; the benefit is that cost stops scaling with sequence length -- the only route that fundamentally sidesteps linear KV cache growth. Kimi K3's KDA belongs to the same family and adds attention residuals as compensation.
Route three: hybrids -- GDN+QSA, sparse+linear. Use the linear half to compress most of the history and keep a small slice on sparse or full attention to protect precision. This is the dominant engineering compromise today. Zhipu's numbers show why: compute down 3.01x, KV cache down 4.44x -- the KV cache drops more than compute does, which is the "compress history" half at work.
One sentence for all three routes: the activation ratio saves compute; the attention mechanism decides whether long context is affordable at all. The former sets the FLOPs per token; the latter sets how that cost grows with length.
3. The Supporting Columns: Parameters, Activation Ratio, Context, License, Price
Architecture sets the curve, but these five columns decide whether you can actually use it (snapshot 2026-08-30; activation ratios are converted):
| Model | Total / active per token | Activation ratio (converted) | Context | License | API price (per 1M tokens, snapshot) |
|---|---|---|---|---|---|
| Qwen3.8-Flash-Next | 125B (plus 51B N-gram embeddings) / 6B | ~4.8% (6B/125B); ~3.4% if the 51B N-gram table counts toward total | Native 262,144, extended to 1,000,000 via YaRN | Unconfirmed: GitHub API license field is None, no LICENSE file in the repo; check HF / ModelScope model pages for the license shipped with the weights | Production Qwen3.8-Flash: $0.16/$0.47 (official X account); $0.15/$0.47 (third-party aggregator, 08-28); 1 RMB / 3 RMB (RMB basis). Defer to the official QwenCloud site |
| Tencent Hy4 preview | 770B / 49B | ~6.4% | 1M | Apache 2.0 | $0.834/$2.501, cache read $0.042 (third-party OpenRouter snapshot 2026-08-28, not an official rate card) |
| Zhipu GLM-5.3-Flash | 320B / 18B | ~5.6% | 1M (max output 128K) | MIT | 0.8 RMB / 2.8 RMB, cache hit 0.23 RMB (official pricing, launched 2026-08-26) |
| Kimi K3 | 2.8T / Not collected | Not collected | 1M | Commercial use allowed (terms per official channels, our earlier reporting) | Output 100 RMB/M (official pricing via media) |
| DeepSeek V4-Flash | 284B / 13B | ~4.6% | 1M | MIT | $0.14/$0.28, cache hit $0.0028 (official pricing page); domestic pricing also uses peak/off-peak tiers |
Three things worth calling out:
- Every activation ratio lands in the 5%-6.5% band. Converted: Qwen ~4.8% (~3.4% counting the 51B N-gram table), DeepSeek ~4.6%, Zhipu ~5.6%, Hy4 ~6.4%. On this dimension the five cannot differentiate anymore -- it is table stakes, not a differentiator. The real split returns to attention architecture: at the same ~5% activation ratio, dense attention and GDN+QSA produce two entirely different cost curves at 1M.
- Qwen's 51B N-gram embeddings are the design outlier. They do not participate in per-token compute (activation is only 6B) and can be offloaded to host memory, scheduled via asynchronous prefetch overlapped with compute -- decoupling capacity from VRAM. The cost is a new PCIe bandwidth dependency; the benefit is that a 125B-class multimodal model becomes feasible on consumer GPUs.
- The license is the only red-flagged cell in this article. The Qwen3.8-Flash-Next repo has no LICENSE file, the GitHub API license field returns None, and the community is already asking. We do not assert that it is or is not Apache 2.0 -- we mark it Unconfirmed. For contrast: Hy4 preview is Apache 2.0; GLM-5.3-Flash and DeepSeek V4-Flash are MIT. Open weights do not equal an open-source license -- the former lets you download, run, and modify; the latter decides whether you can ship it in a commercial product. Read the license text on the model page yourself before you ship.
4. Engineering Reality: Working in a Paper Does Not Mean Fast on a GPU
This is the section most often skipped, and the one most likely to bite.
| Dimension | Qwen3.8-Flash-Next | Hy4 preview | GLM-5.3-Flash | Kimi K3 | DeepSeek V4-Flash |
|---|---|---|---|---|---|
| Kernel / supporting engineering | FlashQLA open-sourced the same day (GDN kernel library, MIT); install requirements SM90 and up, CUDA 12.8+, PyTorch 2.8+ | Official vLLM / SGLang images, TP8 minimum; native MTP speculative decoding layer, 3 draft tokens by default | In-house SGLang inference engine: W8A8 quantization + INT8/FP8/BF16 mixed cache quantization + EPD three-stage disaggregated scheduling; 3x end-to-end (official basis) | MXFP4 quantized weights (4 bits/param) | Not collected |
| Kernel speedup | 2-3x forward, 2x backward (vs. FLA Triton kernel; vendor self-reported, not third-party reproduced) | Not collected | Not collected | Not collected | Not collected |
| Context basis | Native 262,144, extrapolated to 1,000,000 via YaRN | Native 1M | Native 1M | Native 1M | Native 1M |
| Weights / quantization | Not collected | BF16 + FP8 dual format | Not collected | MXFP4 | Not collected |
Four ways to read it:
- "Native" and "extrapolated" are not the same thing. Only Qwen3.8-Flash-Next reaches 1M by extrapolating from a native 262,144 window with YaRN; the other four list native 1M. A native window is a length seen during training; extrapolation is positional-encoding interpolation and usually behaves differently in the tail of long documents. This does not mean extrapolation is unusable -- it means that before running a full 1M, do your own retrieval test instead of capacity planning on the spec number.
- Someone has to write the kernel to pay off the new architecture's speed debt. Qwen shipping the model and the GDN kernel library the same day is itself the answer: an architecture that holds in a paper is a math problem, and making it fast on a GPU is an engineering problem. But hold the 2-3x figure to its reference frame -- the baseline is the FLA Triton kernel, not every backend and not a hand-written CUDA version, and the vendor names pretraining and on-device agentic inference as the scenes where it pays off.
- The kernel floor can be higher than the model floor. FlashQLA requires SM90 and up, meaning a Hopper (H100/H800 generation) or Blackwell card; A100 (SM80) and consumer RTX 30/40 series (SM86/SM89) are not on the support list, and CUDA 12.8 plus PyTorch 2.8 are hard requirements. To get GDN running at the vendor's stated speed, the hardware bar is tougher than the bar for simply loading the model.
- Hy4 and GLM take the other path: official prebuilt images. Hy4 ships vLLM / SGLang images with deployment docs written at cluster scale; GLM ships its own SGLang engine with quantization and scheduling included. For teams with no kernel engineers, having an official image affects time-to-production more than how advanced the architecture is.
5. Pick by Scenario: Five Verdicts
- Want the most controllable long-context cost curve and can live with a pending license: Qwen3.8-Flash-Next. GDN+QSA holds down both ends -- compressing history and picking tokens -- and 6B activation (~4.8%) plus a 51B N-gram table that offloads to host memory make it the most self-hosting-friendly design here; FlashQLA shipping the same day shows the kernel is not an IOU. But the license is unconfirmed -- check the HF / ModelScope model pages before commercial use. For the hands-on path, see our Qwen3.8-Flash-Next local deployment SOP (same batch).
- Want a clean license plus quantified savings: Zhipu GLM-5.3-Flash. MIT, 1M context, 3.01x lower compute and 4.44x smaller KV cache (official basis, vs. GLM-5.3), 320B/18B. It is the only one of the five that reports architecture savings as a checkable number, and that saving is what funds the 0.8 RMB / 2.8 RMB pricing -- the low price is architecture, not a burn-rate subsidy.
- Want a clean license, complete official images, and the pure-sparse route: Tencent Hy4 preview. Apache 2.0, BF16/FP8 dual weights, Gated DSA + IndexCache, native MTP speculative decoding, vLLM/SGLang images. The representative of the token-picking route, suited to teams that want the sparse dividend without writing kernels; the price is 770B of resident weights, so self-hosting is a cluster-scale threshold.
- Want maximum scale, with attention savings as a secondary concern: Kimi K3. 2.8T total, KDA hybrid linear attention, MXFP4 weights to compress VRAM, 1M context. It solves the capability ceiling; output at 100 RMB/M is a premium tier, not a value pick.
- For every scenario: when choosing a 2026 flagship, treat attention architecture as a first-class citizen before parameters. Activation ratios have converged into a 5%-6.5% band and no longer differentiate; what actually drives your long-context bill is the attention route, whether the context is native or extrapolated, and whether kernels and official images ship with it. Read all three ledgers -- parameters and deployment thresholds and API price and cache math cover the other two. And the last point: for any model with an unconfirmed license, check the model page before you ship commercially.
One line to close: when activation ratios have converged around 5% and every flagship claims 1M context, flagships are no longer separated by parameter count but by whose attention makes long context genuinely affordable -- and the page that says whether you may ship it commercially deserves to be opened before any 3.01x figure.
FAQ
Q1: What is the actual difference between sparse and linear attention? A1: The difference is which line item each one saves. Sparse attention (QSA, Gated DSA) is "picking tokens": it keeps the full KV sequence but runs full attention only on positions the indexer selects, saving compute. Linear/recurrent attention (GDN, KDA) is "compressing history": it compresses past context into a compact state, saving KV cache and the overhead that grows with length, at the price of compression loss. One optimizes how much you compute; the other optimizes how much you store. Hybrids save both.
Q2: Does "supports 1M context" mean I can run a full 1M? A2: Not necessarily -- first check whether it is native or extrapolated. Hy4 preview, GLM-5.3-Flash, Kimi K3, and DeepSeek V4-Flash all list native 1M. Qwen3.8-Flash-Next has a native window of 262,144 and reaches 1,000,000 through YaRN interpolation (the production Qwen3.8-Flash defaults to 1M; defer to the official site). Extrapolation usually behaves differently from a native window in the tail of long documents, so run your own retrieval test before committing to a full 1M.
Q3: Which affects cost more, the activation ratio or the attention architecture? A3: It depends on how long your context is. The activation ratio sets the compute burned per token, but it has converged -- converted across these five it sits in a roughly 4.6%-6.4% band (~3.4% for Qwen when the 51B N-gram table is counted), too tight to differentiate. The attention architecture sets how that cost grows with sequence length: at short context (tens of thousands of tokens) the activation ratio dominates; at long context (hundreds of thousands to 1M) the attention architecture immediately becomes the primary variable. For agent loops, whole-repo code, and long research documents, ask about the route before asking about parameters.
Q4: Can Qwen3.8-Flash-Next actually be used commercially?
A4: This article marks it Unconfirmed and will not decide for you. The checkable facts: the GitHub API license field returns None; the repo root contains only README.md and tech_report.pdf; the README states that the license file ships with the weights on Hugging Face Hub or ModelScope; and there is already a discussion thread on HF asking about it. Open weights (download, run, modify) and an open-source license (commercial use, redistribution) are two different things. Read the license text shipped with the weights on the model page yourself before commercial use.
Q5: For self-hosting, is the architecture enough to go on? A5: No -- you also need to check the kernel, the hardware floor, and official images. Someone has to pay off the new architecture's speed debt: Qwen pays it with FlashQLA, shipped the same day, but that requires SM90 and up, CUDA 12.8+, and PyTorch 2.8+, with A100 and consumer RTX 30/40 series off the support list. Hy4 preview and GLM-5.3-Flash take the other path, shipping vLLM/SGLang images and an in-house inference engine. And remember that total parameters set resident VRAM: 770B and 2.8T are both cluster-scale thresholds.
References
-
Qwen official blog (2026-08-26): 125B + 51B N-gram embeddings, 6B active, GDN + QSA, Gated Residual, Muon, native 262,144 / YaRN 1M, training cost about 1/9 of Qwen3.7-Plus: https://qwen.ai/blog?id=qwen3.8-flash-next
-
GitHub repo QwenLM/Qwen3.8-Flash-Next (README plus API check, snapshot 2026-08-30: 259 stars / 11 forks, license field None): https://github.com/QwenLM/Qwen3.8-Flash-Next
-
Tech report tech_report.pdf (On the Design of Qwen3.8-Next Architecture): https://github.com/QwenLM/Qwen3.8-Flash-Next/blob/main/tech_report.pdf
-
Hugging Face model page Qwen/Qwen3.8-Flash-Next (license per the file shipped with the weights): https://huggingface.co/Qwen/Qwen3.8-Flash-Next
-
ModelScope model page Qwen/Qwen3.8-Flash-Next: https://www.modelscope.cn/models/Qwen/Qwen3.8-Flash-Next
-
DataCamp third-party coverage (benchmark and price basis, not the official table): https://www.datacamp.com/blog/qwen3-8-flash-next
-
QwenLM/FlashQLA (GDN kernel library, MIT; 2-3x forward / 2x backward vs. the FLA Triton kernel is vendor self-reported; SM90 and up, CUDA 12.8+, PyTorch 2.8+): https://github.com/QwenLM/FlashQLA
-
Tencent official press release (2026-08-28) and the Hunyuan Hy4 preview repo: 770B/49B, Gated DSA (top-k 2048) + IndexCache, 1M, Apache 2.0, BF16/FP8: https://www.tencent.com/tencent-releases-and-open-sources-tencent-hy4-preview ; https://github.com/Tencent-Hunyuan/Hy4-preview
-
Gated DSA paper: https://arxiv.org/abs/2512.02556 ; IndexCache paper: https://arxiv.org/abs/2603.12201
-
OpenRouter Hy4 preview page (third-party aggregator snapshot 2026-08-28; $0.834/$2.501, cache read $0.042): https://openrouter.ai/tencent/hy4-preview
-
Tencent Cloud Developer Community (2026-07-17): Kimi K3 at 2.8T parameters, KDA hybrid linear attention, MXFP4: https://cloud.tencent.com/developer/news/4268105
-
Zhipu AI documentation, GLM-5.3-Flash model card (320B/18B, sparse+linear hybrid, mHC, 1M, MIT): https://docs.bigmodel.cn/cn/guide/models/vlm/glm-5.3-flash
-
Cailian / STAR Market Daily (2026-08-27): attention compute down 3.01x, KV cache down 4.44x, pricing 0.8 RMB / 2.8 RMB, 100,000 domestic accelerators: https://baijiahao.baidu.com/s?id=1874642322871069438
-
Lanjinger News (2026-08-27): https://baijiahao.baidu.com/s?id=1874660001676882489
-
DeepSeek API official changelog (2026-07-31; V4-Flash final at 284B / 13B MoE, post-training reworked): https://api-docs.deepseek.com/zh-cn/updates
-
DeepSeek official pricing page (V4-Flash $0.14/$0.28, cache hit $0.0028): https://deepseek.ai/pricing
-
Morph (V4-Flash 284B MoE, 1M context, MIT): https://www.morphllm.com/deepseek-v4-flash
-
Further reading: our Qwen3.8-Flash-Next multimodal hotspot (the main subject, same batch), Qwen3.8-Flash-Next local deployment SOP (same batch), FlashQLA open-source deep dive (the kernel piece, same batch); earlier: open-source flagship comparison (parameters and deployment thresholds), lite flagship API price comparison (price and cache math), Zhipu GLM-5.3-Flash open-source hotspot, Tencent Hy4 preview open-source hotspot, DeepSeek V4-Flash launch hotspot
This is a representative comparison compiled from public reporting and official releases, snapshot 2026-08-30, not hands-on benchmarking and not investment advice. Architecture descriptions and cost-reduction figures defer to each vendor's official basis; activation ratios and other derived figures are converted or engineering estimates. The Qwen3.8-Flash-Next license status is unconfirmed -- verify against the license file shipped with the weights on the HF / ModelScope model pages before commercial use. Prices and repo data may change at any time; verify against official pricing pages before purchasing.