On October 20 DeepSeek announced on its official WeChat account that it was open-sourcing its inference framework, and pushed two repositories to GitHub the same day: DeepInfer, the framework itself, at github.com/deepseek-ai/DeepInfer
Repository: https://github.com/deepseek-ai/DeepInfer, and deepseek-recipe, the token encoding and decoding component, at github.com/deepseek-ai/deepseek-recipe. What is new is the layer that decides how fast and how cheaply a model runs.
The conclusion up front: what shipped is an engine, not the whole factory. Below are the repository numbers, the officially reported performance figures with their exact basis, the hardware path, and the parts still locked inside DeepSeek.
1. Two repositories: one does the math, one does the interface
DeepInfer is the inference framework. Its repository description reads "DeepInfer: High-performance, large-scale inference framework for DeepSeek models", and the README calls it the high-performance inference framework powering DeepSeek's large-scale production services. This release provides offline inference plus the core features needed for serving at scale.
deepseek-recipe does something else. It is Rust libraries with Python bindings that convert Chat Completions, Responses and Messages requests into one Conversation representation, render prompts for a given model version, encode them into token IDs, and parse model output back into the caller's protocol. The official statement is that its logic is fully aligned with the official DeepSeek API.
The two are released side by side, not wired together. DeepInfer's requirements.txt has no dependency on deepseek-recipe, none of the 342 repository files has recipe in its path, and DeepInfer renders prompts through the checkpoint's own encoding/encoding.py.
Licensing must be read separately. DeepInfer is Apache-2.0, its LICENSE the unmodified Apache License 2.0 template with the copyright line still a placeholder. deepseek-recipe is MIT, its LICENSE reading "Copyright (c) 2026 DeepSeek". Both are permissive, but the notice obligations differ, so closed-source commercial use must follow each license's own terms.
2. Repository numbers, from the GitHub API
Measured via the GitHub API on 2026-10-22. Star counts move over time:
| Repository | License | Stars | Forks | Created | Last push |
|---|---|---|---|---|---|
| deepseek-ai/DeepInfer | Apache-2.0 | 17,902 | 1,836 | 2026-10-16 | 2026-10-21 |
| deepseek-ai/deepseek-recipe | MIT | 1,947 | 244 | 2026-09-10 | 2026-10-19 |
DeepInfer was created on October 16, announced as fully open source on October 20, and reached 18,000 stars in four working days. deepseek-recipe existed a month earlier, so the encoding layer was prepared in advance and the engine was the headline act.
The release artifacts are equally early-stage: GitHub Releases and Tags are both 0, PyPI carries only the sdist deepinfer 0.1.0 uploaded on 2026-10-20, and the repository has no Dockerfile and no docker reference. The open-issue list was not verified item by item, so no issue counts are given here.
3. Performance numbers, all vendor-reported
Everything here comes from DeepSeek's own material: the online statistics table in the official WeChat post of 2026-10-20, the official Zhihu technical blog, and the deployment Q&A document in the repository. No third party has reproduced it.
Scale and revenue: a single cluster serves 155 trillion tokens per day, with overall revenue of 38 dollars per GPU-hour at valley pricing, and an official average input length of 77K tokens, which says the system is shaped for long documents and agent workloads.
Latency and generation: average time to first token of 400 milliseconds with a P99 of 1000 milliseconds, generation at 230 tokens per second and higher in code scenarios, and 14K tokens per second per card on decode.
Caching and prefill: a 96% average input cache hit rate, and prefill throughput per card rising from 60,000 tokens per second without cache hits to 1,500,000 with them, roughly 25 times.
One limitation is mandatory: all of these are NVIDIA SM100 cluster figures. The README performance charts are labeled "Measured on 4 SM100 GPUs for prefill and 64 for decode". Huawei Ascend has no official measured throughput or latency published at all, so none of these numbers may be pushed onto Ascend 950. Reading the official charts directly, prefill on EP4DP4 peaks near 89,700 tokens per second per GPU at a batch of 65,536, and decode on EP64DP64 near 26,300 per GPU at 256 requests.
Still not published: total cluster GPU count, per-request TTFT and TBT distributions, minimum card count and memory requirements, DeepInfer's own evaluation scores, and any standalone technical report.
4. SWA Bounded Replay and DSpark: the two designs that save compute
SWA Bounded Replay comes first. Some layers of V4.1 Flash use sliding window attention over only the 128 most recent tokens. When the cache is missing, the official system does not recompute over the full context: it replays only the last 128 tokens, truncates the attention window to that segment, and uses the result as an approximate SWA KVCache. Recomputation cost becomes a constant instead of growing with context length and layer count, independent of both in the official wording. The same trick applies to 20 more decoder layers, where cutting the reconstruction from the last 2560 tokens to 128 brings total prefill computation close to half.
DSpark speculative decoding comes second, pairing draft prediction with confidence-scheduled verification. The system predicts several likely next tokens, a Confidence Head estimates each candidate's acceptance probability, and limited verification compute goes to the candidates with the highest marginal return. In production the draft step produces 5 tokens at a time, with the maximum draft length set by DEEPINFER_USE_NEXTN_DECODE, defaulting to 5.
One nuance: the 6% throughput gain over a fixed 5-token verification scheme is the official WeChat summary, while the Q&A document breaks it down by concurrency, with verified tokens per step falling from 5.00 to between 3.2 and 4.4, the average accepted length essentially unchanged, and the gain ranging from plus 4% to plus 7%.
Stack DSpark, confidence scheduling and asynchronous scheduling together and the official result is 14K tokens per second per card in the online decode stage. Asynchronous scheduling lets CPU and GPU work at once: while the GPU runs step N, the CPU is already scheduling step N+1 and post-processing step N-1.
5. MoE expert replica reshuffling and the 80% balance figure
V4.1 Flash is a mixture-of-experts model with 384 routed experts. In decode the official deployment spreads them across 64 cards, six per card plus one extra replica of a hot expert, seven in total, so each step reads fewer parameters from memory.
The hot replicas move as well: load is sampled every 1000 steps and re-assigned every 10000 steps, and in production the busiest card runs only 9% above average load, the official moe_lb metric reading 1.090. Prefill uses 4 redundant experts per card, a 100-step statistics window and re-assignment every 1000 steps. One official self-contradiction is worth listing side by side: for prefill moe_lb the GitHub Q&A gives 1.074 at EP4DP4 and 1.045 at EP2DP2, while the official Zhihu blog gives 1.0618 and 1.0320; decode agrees at 1.090 on both sides. This article does not adjudicate between them.
6. What did not ship: an engine, not a factory
This is the part that most needs writing down. The README lists what a complete inference service requires but that this release does not include: a KV cache system, VE/Prefill/Decode disaggregation, and load balancing across data-parallel replicas for both prefill and decode. The deployment Q&A is blunter: the KV cache management and load balancing components are tightly coupled to DeepSeek's internal infrastructure, so that code is not yet fully open-sourced. In plain terms this release ships offline inference plus the request-to-result path and its scheduling primitives: optimized kernels and the model and sampler runtime, attention data parallelism and balanced context parallelism, MoE expert parallelism with expert load balancing, DSpark with confidence-scheduled verification, SWA bounded replay, asynchronous scheduling and structured outputs. The operations layer, meaning multi-machine cache storage, cross-replica routing and cross-datacenter scheduling, is not included.
Two more boundaries. The official release provides no head-to-head benchmark against vLLM, SGLang or TensorRT-LLM; the eval directory measures model capability (AIME 2026, HLE, GPQA Diamond, Pass@1) and ships no scores, not throughput between frameworks. And there is no official Docker image: no Dockerfile, no docker reference anywhere, and PyPI carries only the sdist.
7. Hardware and deployment: two stacks, a high bar
The README support list puts two platforms on the same level, NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. That is not a plan to support them later; it is the same code base compiled and run separately against two ecosystems, and the Ascend side is no placeholder: CANN 9.2 and torch-npu==2.13.0rc1, vendor kernels DeepEP-Ascend and DeepGEMM-Ascend, an operator layer with tilelang_ops/asc/ holding 30 files against 30 in cuda/, and a ci_npu marker running the NPU pipeline in CI. DeepSeek's official WeChat post of 2026-09-30 on open-sourcing its Ascend base components also mentions joint work on a 128-card supernode solution based on Ascend 950.
What is missing on Ascend is stated by the vendor: batch-invariant inference and deterministic sampling are both marked "(NVIDIA only)" in the official feature list and need DEEPINFER_BATCH_INVARIANT=1. Anyone expecting identical output across batch compositions on Ascend should not count on these two.
Build prerequisites: Linux, GCC 13.0 or newer, Python 3.12, and either CUDA 13 for NVIDIA or CANN 9.2 for Ascend. The official steps are to create and activate an empty Python 3.12 virtual environment, then export DEEPINFER_ACCELERATOR=nvidia (use ascend on Ascend) and run ./build.sh from source, keeping that variable exported where DeepInfer later runs. Weights are handled on first run: the default checkpoint deepseek-ai/DeepSeek-V4.1-Flash-1017 is downloaded into the Hugging Face cache, or you set DEEPINFER_HF_MODEL_DIR to a local copy. The first example is python examples/offline_inference.py.
The system shape explains the design. Prefill runs on ordinary 8-GPU servers, decode on 64 to 128-GPU supernodes, and the two stages get separate compute, communication and scheduling matched to their load characteristics. The smallest efficient deployment unit the official Q&A names is 64 GPUs of prefill plus 64 GPUs of decode, with the practical prefill to decode card ratio close to 1:1. The KVCache target is retention of more than 72 hours.
8. Which claims are second-hand and which you must verify
The vLLM bloodline comes first. The official README acknowledgments state that DeepInfer was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM, and setup.py carries the comment "Derived from vLLM; modified by DeepSeek". So the claim that it started life on a vLLM base has official backing, while the community phrase "a deeply modified fork" is second-hand framing.
Whether it only serves DeepSeek models holds for the current release: the supported models list names only DeepSeek-V4.1-Flash and the description says "for DeepSeek models". The official statement is that inference code for future DeepSeek models will keep being open-sourced in DeepInfer.
The Hacker News score has two figures from different measurement times, so any citation must state when it was taken, and this article gives no number. Third-party reproduction is also absent: every performance number above has exactly one source, the official one, which leaves whether they reproduce on other hardware unanswered.
9. Who this is for, and how it divides with our other DeepSeek piece
DeepInfer fits two groups: teams that already own SM100 or Ascend 950 clusters and need to build their own large-scale inference service, and engineering teams prepared to write the cache and load balancing layers themselves. It is not a hobbyist download, and it is not a turnkey serving framework, because the open-source release ships offline inference.
This site previously published "DeepSeek Weights Are Free, Yet Agent Task Bills Fall 60 Percent", which covers the model itself: DeepSeek-V4.1-Flash-1017, its parameter counts, benchmark results and API pricing. This article stays on the inference engine layer: the two repositories and their licenses, the officially reported serving numbers with their hardware basis, SWA Bounded Replay and DSpark, MoE load balancing, what was not open-sourced, and the deployment path.
FAQ
Q1: What exactly is the relationship between DeepInfer and vLLM?
A: The official README says DeepInfer was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM, and setup.py notes "Derived from vLLM", so the built-on-vLLM claim has official backing while "deeply modified fork" is community framing. No official head-to-head benchmark against the three was provided.
Q2: What was open-sourced, and what did the official release leave out?
A: What shipped is offline inference plus the engine core: optimized kernels, attention data parallelism, MoE expert parallelism with expert load balancing, DSpark, SWA bounded replay, asynchronous scheduling and structured outputs. The README lists three items a complete inference service needs but this release lacks: a KV cache system, VE/Prefill/Decode disaggregation, and load balancing across data-parallel replicas.
Q3: Can Ascend run it, and do those numbers apply to Ascend?
A: Ascend 950 NPUs are supported at the same level as SM100 GPUs, with CANN 9.2, torch-npu, Ascend vendor kernels and a dedicated NPU CI pipeline. But every performance figure here is an SM100 cluster number, since the README labels the charts as measured on 4 SM100 GPUs for prefill and 64 for decode, and Ascend has no official measured throughput or latency published.
Q4: Is the hardware bar high? Can an ordinary team run it?
A: Building needs Linux, GCC 13 or newer, Python 3.12, and either CUDA 13 or CANN 9.2, on SM100 GPUs or Ascend 950 NPUs. The path is a clean Python 3.12 virtual environment, export DEEPINFER_ACCELERATOR=nvidia, ./build.sh, then the weights download DeepSeek-V4.1-Flash-1017 unless DEEPINFER_HF_MODEL_DIR points at a local copy.
Q5: Does it support models other than DeepSeek ones?
A: The supported models list in the official README names only DeepSeek-V4.1-Flash and the repository description says "for DeepSeek models", so the community claim that it serves only the DeepSeek model family holds for this release. The official statement is that inference code for new DeepSeek models will keep being open-sourced in DeepInfer.