DeepInfer – In-Depth Evaluation of DeepSeek's Production-Grade Open-Source Inference Framework

1. What is DeepInfer

DeepInfer is a production-grade large model inference framework open-sourced by DeepSeek, derived from the vLLM codebase. It is specifically designed for the DeepSeek-V4.1-Flash model and supports both the NVIDIA SM100 GPU and Huawei Ascend 950 NPU computing ecosystems. The framework utilizes a PD-separated deployment architecture, placing the Prefill and Decode stages on different hardware. It relies on the 3FS distributed caching cluster to achieve a 96% input cache hit rate, increasing the Prefill throughput by approximately 25 times. The synchronously open-sourced deepseek-recipe component ensures API protocol alignment, guaranteeing that self-deployed models behave consistently with the official API.

Technical Positioning and Domain: DeepInfer belongs to the domain of large model inference and service optimization, focusing on ultra-high throughput and low-cost inference services in production environments. It targets enterprise private deployment and large-scale online inference scenarios. Its differentiating positioning lies in "specialized optimization for specific models" – rather than pursuing generality, it deeply customizes optimization paths for DeepSeek's own models, pushing inference efficiency to its limits.

Development Background: This framework was open-sourced by the official DeepSeek team, originating from real-world production validation in their large-scale online services. During the deployment and operation of V4.1 Flash, DeepSeek accumulated vast amounts of real traffic data and performance tuning experience. They have now fully open-sourced previously internally used inference optimization technologies, reflecting their strategic intent to expand into the infrastructure domain.

Core Value: Addressing the pain point of performance discrepancies between self-deployed DeepSeek models and the official API. Previously, developers struggled to replicate the low latency and high throughput of the official API when building their own inference services. DeepInfer, combined with the deepseek-recipe component, aligns the interface behavior and performance levels of self-deployed models with the official API, significantly lowering the barrier for enterprises to adopt open-source models.

Technical Features: The core innovations are concentrated in four areas: PD-separated deployment enables optimal hardware division of labor; SWA Bounded Replay fixes the cost of long context recalculations at 128 tokens; DSpark speculative decoding, combined with confidence scheduling, increases generation throughput by approximately 6%; and the 3FS prefix caching cluster achieves a 96% hit rate. According to official data, a single cluster processes up to 15.5 quadrillion tokens per day, with a first-character latency of just 400ms.

2. Key Features

• PD Separation Deployment: DeepInfer separates the inference process into two independent stages: Prefill (pre-filling) and Decode (decoding), deploying them on machines with significantly different hardware configurations. The Prefill stage uses standard 8-GPU servers to handle inputs of tens of thousands of Tokens with high computational power. The Decode stage uses 64–128-GPU super nodes to meet the intensive memory bandwidth requirements of per-Token generation. The two types of machines are configured in a roughly 1:1 ratio, allowing each stage to operate optimally based on its hardware characteristics.

• 3FS Prefix Cache Cluster: The system writes data in blocks of 128 Tokens to an independent 3FS distributed storage cluster, caching only prefixes that are exactly the same starting from the first Token. When a prefix is hit, only the newly added content is computed, and the existing KV Cache is reused directly. In Agent scenarios, the complete history is resent with each tool call, and this mechanism significantly reduces redundant computation. The cache retention policy varies by traffic type: Agent traffic is retained for 3 days, and chat traffic is retained for 7 days, balancing hit rate and storage cost.

• SWA Bounded Replay: A custom optimization for the sliding window attention mechanism used in the Flash module of V4.1. Since each Token only focuses on the most recent 128 Tokens, its KV Cache is not written to disk but remains in the memory of the computing machine. In traditional approaches, when a cache miss occurs, the computation must be recalculated precisely for "number of layers × window length." DeepInfer instead only recalculates the last 128 Tokens, with attention computed only within this window, achieving an approximate window cache. This nearly halves the total prefill computation and decouples cost from context length and number of layers.

• DSpark Speculative Decoding and Asynchronous Scheduling: A lightweight draft module speculates the next 5 Tokens continuously, while the target model verifies them in batches. Correct guesses are directly adopted, and the first incorrect guess is corrected by the large model, allowing multiple Tokens to be advanced with a single computation. A confidence head is used to estimate the probability of each guess being accepted, and low-probability guesses are no longer verified, freeing up computational resources for other requests. This results in a 6% increase in online throughput compared to a fixed verification of 5 Tokens. Additionally, CPU scheduling for the N+1 step overlaps with GPU computation for the N step, eliminating GPU idle time during scheduling gaps.

• MoE Expert Load Balancing: To address the uneven distribution of experts in a Mixture-of-Experts (MoE) model, the system dynamically redistributes expert replicas across GPUs every 10,000 steps, achieving balanced load distribution between experts and requests. According to online data, the busiest GPU only exceeds the average load by 9%, and the overall balance reaches 80% of the ideal state, effectively avoiding throughput degradation caused by the "bucket effect."

• Dual Hardware Platform Compatibility: Supports both NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, covering the two major mainstream computing ecosystems. Through a unified abstraction layer, hardware differences are masked, allowing the same codebase to be compiled and run on different platforms without modification. It is currently one of the very few production-level inference frameworks that provide dual-stack support for CUDA and CANN.

3. How to Use

1. Environment Prerequisites Verification: The operating system must be Linux, the compiler must be GCC ≥ 13.0, and the Python version must be 3.12. Choose one of the two hardware platforms: the NVIDIA platform requires CUDA 13 to be installed, and the Ascend platform requires CANN 9.2 to be installed. Dependency differences between hardware platforms are significant, so ensure the environment is complete before compiling.

2. Clone the Repository: Execute git clone https://github.com/deepseek-ai/DeepInfer to clone the main repository. If API alignment capabilities are required, clone the component repository https://github.com/deepseek-ai/deepseek-recipe simultaneously. After cloning, enter the DeepInfer directory to view the project structure.

3. Create a Clean Python Environment: Create and activate an empty Python 3.12 virtual environment (e.g., using python3.12 -m venv venv to create and source venv/bin/activate to activate), to avoid dependency conflicts with existing Python packages on the system. This step directly affects the compilation success rate, so it is recommended to perform this operation in a new environment.

4. Select the Hardware Platform and Compile: Set the target hardware using export DEEPINFER_ACCELERATOR=nvidia (change to ascend for the Ascend platform), then run ./build.sh to build from source. Note that this environment variable not only affects the compilation phase but must also remain effective in the subsequent runtime environment; otherwise, hardware recognition may fail during execution.

5. Prepare Model Weights: By default, the first run will automatically download the model weights from Hugging Face: deepseek-ai/DeepSeek-V4.1-Flash-1017. For offline deployment, use the environment variable export DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017 to point to a local copy of the model, avoiding repeated downloads.

6. Run Offline Inference Verification: Execute python examples/offline_inference.py to start the inference test and verify whether the environment configuration and model loading are functioning properly. For more usage methods, refer to the various example scripts in the examples directory of the repository. Modify the appropriate example based on your specific use case.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: If an enterprise's primary goal is to self-deploy DeepSeek models and achieve official-level performance, DeepInfer is the most suitable choice. Its DSpark speculative decoding and SWA Bounded Replay optimizations are specifically tailored for the V4.1 Flash architecture, which general frameworks struggle to replicate. Combined with deepseek-recipe, it enables seamless integration with the official API, making it ideal for teams with existing DeepSeek API calling experience who wish to migrate to private deployment.

If an enterprise needs to serve multiple open-source models simultaneously or wants to quickly launch a general-purpose inference service, vLLM remains the top choice due to its model compatibility and mature ecosystem. SGLang is worth considering in scenarios requiring structured output or high cache hit rates for long contexts, while TensorRT-LLM is suitable for teams that are tightly integrated with NVIDIA hardware and seek maximum single-card performance. Whether DeepInfer will expand its model support in the future is something to keep an eye on.

6. Editor's Summary

The greatest value of DeepInfer lies in demonstrating the feasibility and effectiveness of a "specialized model for specific tasks" inference framework. DeepSeek has fully open-sourced the optimization techniques accumulated from its V4.1 Flash large-scale online service—PD separation deployment, SWA Bounded Replay, DSpark speculative decoding, and 3FS prefix caching—enabling external developers to reproduce inference performance at the official API level. According to official data, a single cluster can process 15.5 quadrillion tokens per day, with prefill throughput increased by approximately 25 times and computational load for long contexts nearly halved. These metrics are difficult to achieve simultaneously in general-purpose inference frameworks, highlighting the practical benefits of deep customization and optimization.

In terms of practical value, DeepInfer precisely addresses two key pain points: first, the performance gap faced by enterprises when deploying DeepSeek models privately. The deepseek-recipe component ensures that the interface performance is fully consistent with the official API, significantly reducing migration costs. Second, the demand for domestic computing power compatibility. Native support for Huawei Ascend 950 gives it a differentiated competitive advantage in information innovation scenarios, freeing it from reliance on NVIDIA hardware.

The primary users of DeepInfer include three main categories: first, startups and mid-sized companies that need a high-cost-performance inference service, using its high throughput per card and generation speed of 230–300 tokens/s to replace the pay-as-you-go official API; second, government and enterprise organizations, as well as financial institutions, with domestic computing power deployment needs; third, technical teams that pursue extreme inference performance, who can draw inspiration from its advanced optimization strategies such as PD separation and sliding window caching.

The main limitations of DeepInfer currently are its narrow model support and restricted functionality in the open-source version. Its future development will depend on whether DeepSeek expands model compatibility and gradually opens up online service capabilities. Considering the continuous investment from the official DeepSeek team and the market influence of the V4.1 series models, DeepInfer is likely to become the standard inference infrastructure within the DeepSeek ecosystem and hold an important position in the field of domestic inference frameworks.

7. Application Scenarios

• Enterprise Private Deployment: Enterprises deploy the V4.1 Flash model in their on-premise data centers, using the deepseek-recipe component to align with the official API behavior, enabling compliant inference with data staying within the enterprise domain. This is ideal for data-sensitive industries such as finance, healthcare, and legal, offering an interface experience consistent with the official API while ensuring data security. Business systems can be migrated with minimal modifications.

• Domestic Computing Power Adaptation: Natively supports Huawei Ascend 950 NPU, meeting the requirements for self-reliant and controllable computing power in government-enterprise, defense, and energy sectors under the national information innovation initiative. IT teams can compile and run DeepInfer on the Ascend platform, achieving full-chain domestication from chip to framework, reducing reliance on NVIDIA's CUDA ecosystem, and aligning with national information innovation policies.

• Large-scale Agent Services: In Agent scenarios, each tool call retransmits the full historical conversation. The 96% prefix cache hit rate significantly reduces redundant computational costs across multiple rounds of calls. This is suitable for high-frequency tool calling Agent platforms such as intelligent customer service, code assistants, and automated workflows, enabling low-latency responses while substantially lowering inference costs.

• Processing of Ultra-long Context Documents: The SWA Bounded Replay mechanism nearly halves the prefill computational load for long texts, with costs independent of context length. This is ideal for scenarios involving ultra-long inputs such as contract reviews, codebase analysis, and academic paper reading, allowing for rapid prefill processing of massive text volumes at controllable costs and improving overall response efficiency.

• High-cost-performance Ratio Self-built Inference Service: With a high throughput of 14K tokens per card per second and a generation speed of 230–300 tokens/s, it is suitable for startups and technical teams with large-scale inference needs. Compared to the official API's pay-as-you-go model, building a self-hosted DeepInfer service offers lower marginal costs, significantly reducing inference expenses in high-volume usage scenarios and achieving cost savings at scale.

8. FAQ

Q: What is the relationship between DeepInfer and vLLM?
A: DeepInfer's codebase originates from vLLM and is a production-level inference framework specifically customized for the DeepSeek V4.1 Flash model. vLLM is a general-purpose inference framework that supports hundreds of models, while DeepInfer focuses on a single model, integrating optimizations tailored to the V4.1 Flash architecture, such as DSpark speculative decoding and SWA Bounded Replay, to achieve optimal inference performance for this model.

Q: Which hardware platforms does DeepInfer support?
A: DeepInfer supports two major computing platforms: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. Before compilation, you must specify the target platform (nvidia or ascend) using the DEEPINFER_ACCELERATOR environment variable. The NVIDIA platform requires CUDA 13, while the Ascend platform requires CANN 9.2. The hardware requirements are relatively high.

Q: What is PD separation deployment? Why does it improve performance?
A: PD separation deployment runs the Prefill (pre-filling) and Decode (decoding) stages on different hardware. Prefill processes tens of thousands of Tokens at once and is a compute-intensive task, which can be efficiently handled by a standard 8-GPU server. Decode generates Tokens sequentially and is limited by memory bandwidth, requiring a large amount of memory bandwidth from 64–128 GPU super nodes. By configuring both types of machines approximately in a 1:1 ratio, each stage can operate on hardware optimized for its specific characteristics, avoiding the efficiency loss caused by resource contention in traditional hybrid deployments.

Q: What is the core advantage of SWA Bounded Replay?
A: SWA Bounded Replay optimizes the sliding window attention mechanism used in certain layers of the V4.1 Flash model. It fixes the recalculation cost to the last 128 Tokens when there is a cache miss, decoupling the recalculation cost from the traditional "number of layers × window length" computation. This means that regardless of the context length, the cost of fixing a cache miss is fixed, nearly halving the total Prefill computation in long context scenarios.

Q: What is the difference between the open-source version of DeepInfer and the official online service?
A: The open-source version of DeepInfer provides the same core inference engine and optimization technologies as the online service (such as PD separation, cache reuse, and speculative decoding), but only includes offline inference capabilities. The official online service also includes complete online API services, cluster scheduling, monitoring, and operations and maintenance components necessary for production environments, which users must build themselves in the open-source version. For API protocol alignment, it can be used in conjunction with the deepseek-recipe component.

Q: Is the deepseek-recipe component required?
A: It is not required, but strongly recommended. The DeepInfer core engine is responsible only for inference computation, while deepseek-recipe handles protocol-level logic such as unified encoding/decoding for the three API formats (Chat Completions, Responses, Messages), injection protection, parallel tokenization, and streaming output. If you are deploying your own service and need full compatibility with the official API, deepseek-recipe can significantly reduce the workload of interface adaptation.

Q: Where can the model weights be obtained?
A: By default, the model weights will be automatically downloaded from Hugging Face as deepseek-ai/DeepSeek-V4.1-Flash-1017 during the first run. If you need to deploy offline or use an existing model copy, you can specify a local path using the DEEPINFER_HF_MODEL_DIR environment variable to avoid repeated downloads. For specific acquisition methods, it is recommended to refer to the official DeepSeek model release documentation.

9. Project Links

• Official GitHub repository: https://github.com/deepseek-ai/DeepInfer
• deepseek-recipe component repository: https://github.com/deepseek-ai/deepseek-recipe
• Hugging Face model weights: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017