Watch the 1-minute version: five tricks inside DeepSeek’s engine and what is still missing.
DeepSeek just gave away the engine it uses to serve its own chatbot and API. It is called DeepInfer, and DeepSeek says one cluster running it handles 155 trillion tokens a day. I read the official DeepInfer repository and DeepSeek’s long deployment Q&A end to end, because the headlines were mixing up what is released and what is not.
Short version: DeepInfer is a real, Apache 2.0 licensed engine, but it is one piece of the puzzle. The caching system, the traffic balancers and the prefill/decode split are not in the release. This guide explains what you get, what you do not, and who should care.
- What it is: the inference engine behind DeepSeek V4.1 Flash, released as open source under Apache 2.0.
- What it runs: only DeepSeek V4.1 Flash for now, on NVIDIA SM100 GPUs or Huawei Ascend 950 chips.
- What is missing: the KV cache system, the load balancers and the full prefill/decode disaggregation.
- Smallest efficient setup: 64 GPUs for prefill plus 64 GPUs for decode, per DeepSeek.
- Who it is for: cloud providers and big teams. Most people should keep using the API or a hosted copy.
What Is DeepInfer?
A model is only half of an AI service. The other half is the software that loads the model on thousands of chips, feeds it requests, and gets answers back fast and cheap. That software is called an inference engine. DeepInfer is DeepSeek’s.
DeepSeek says it was originally built mostly on top of vLLM, with ideas borrowed from SGLang and TensorRT-LLM. It runs on two hardware families, so DeepSeek is not locked to one chip maker.
It pairs with a second release, deepseek-recipe. That one converts between popular API formats (Chat Completions, Responses and Messages) and the raw token sequences the model reads. DeepSeek says its logic matches the official API, so a third party hosting the model should behave the same way. For the model itself, see our guide to DeepSeek V4.1 Flash.
The Numbers DeepSeek Shared
| Claim | Number | What it means |
|---|---|---|
| Daily volume | 155 trillion tokens per day on a single cluster | The scale this engine was built for |
| Revenue | US$38 per GPU-hour at valley (off-peak) pricing | Each GPU earns this even in the cheapest price window |
| Cache retention | More than 72 hours | Repeated context is reused, not recomputed |
| Cache lifetime | 3 days for agent traffic, 7 days for chatbot traffic | Chat users come back to old context more often |
| Cache storage | About 100 PiB of SSD, about 14.4 TB per GPU (logical size) | Caching is a huge storage system, not a small add-on |
| Load balancing result | Roughly 80% of a perfectly balanced offline benchmark | Real traffic is messy, and this is how close they get |
These are DeepSeek’s own figures from its own documents. I have not run DeepInfer, and nobody outside DeepSeek has checked these numbers yet.
How DeepInfer Works in Plain English
Think of a restaurant. Reading your order takes one kind of effort, and cooking each dish takes another. DeepSeek splits the two jobs onto different machines so neither slows the other down.
1. Prefill and decode run on separate machines
Prefill is the model reading your prompt. Decode is the model writing the answer one token at a time. They stress hardware differently, so DeepSeek runs prefill on ordinary 8-GPU servers and decode on large 64 to 128 GPU groups. A separate vision encoder handles images, usually one GPU per instance.
2. Experts are spread wide across many GPUs
V4.1 Flash is a mixture-of-experts model, so each token only uses a few of many small expert networks. For decode, DeepSeek spreads those experts across 64 to 128 GPUs, which lets each GPU do bigger, more efficient batches. It also keeps spare copies of busy experts and reshuffles them as demand changes.
3. A giant shared cache avoids rereading the same text
If you send the same long document twice, the model should not read it twice. DeepSeek stores the processed result (the KV cache) on a dedicated storage cluster built on its own 3FS file system, so any prefill machine can reuse it. The data is stored with 10+2 erasure coding, so two storage nodes in a group can fail and nothing is lost.
4. SWA bounded replay saves almost half of prefill work
Instead of recomputing a long stretch of history, the engine replays only the last 128 tokens to rebuild the small sliding window it needs. DeepSeek says this nearly halves total prefill computation on the decoder side.
5. DSpark guesses ahead, then checks only the good guesses
Speculative decoding means a small draft step proposes several tokens, and the main model verifies them. DSpark adds a confidence score for each guess and verifies only the guesses most likely to be right. That cuts the tokens checked per step by about 30% (from 5 down to between 3.2 and 4.4) while the average accepted length stays about the same.
| Sequences per GPU rank | Fixed 5-token check | DSpark (variable) | Gain |
|---|---|---|---|
| 16 | 4,891 tok/s | 4,816 tok/s | -1.5% |
| 32 | 8,964 tok/s | 9,111 tok/s | +1.6% |
| 48 | 12,032 tok/s | 12,467 tok/s | +3.6% |
| 60 (production level) | 13,730 tok/s | 14,466 tok/s | +5.4% |
| 72 | 14,962 tok/s | 15,961 tok/s | +6.7% |
| 96 | 16,817 tok/s | 18,660 tok/s | +11.0% |
The gain is small at low load and grows with traffic. At the level DeepSeek runs in production (about 60 sequences per GPU), DeepSeek reports roughly 4 to 7% more throughput. That is a modest win, not a miracle.
6. The GPU never waits on the CPU
Normally the GPU sits idle while the CPU plans the next step. DeepInfer plans step N+1 while the GPU is still running step N, which DeepSeek calls a zero-overhead batch scheduler. It is on by default.
7. Smart load balancing for long and short requests
Prefill requests are sorted into four buckets by length (under 16K, 16K to 64K, 64K to 256K, and 256K plus), and each bucket gets its own share of machines. Decode balancing weighs both how many requests a GPU holds and how long their cached context is, so short and long jobs get mixed sensibly.
8. Low-precision math to save memory
The engine uses 4-bit formats (FP4) for the main KV cache, the indexer and the expert weights, and an 8-bit format for the sliding-window cache. Smaller numbers mean less memory moved per token, which is where much of the speed and cost saving comes from.
9. Structured outputs and exact repeatability
You can force output to match a regex, a JSON schema or a grammar. DeepInfer also offers batch-invariant inference and deterministic sampling, so the same input and seed give the same output (NVIDIA only).
10. Recovery from hardware failures
A single bad GPU can break a big expert-parallel group. DeepSeek’s fix is to restart the group with the healthy GPUs it still has, and to keep spare capacity so requests can be rerouted. Restarts take minutes, so the safety net is spare capacity, not fast restarts.
What Is and Is Not in the Release
| Included in the open-source release | Not included |
|---|---|
| Optimized kernels, model and sampler runtime | The KV cache system (kvcache-store on 3FS) |
| Attention data parallelism and expert parallelism with expert load balancing | Load balancers for prefill and decode data-parallel replicas |
| DSpark speculative decoding | Full vision/prefill/decode disaggregation (DeepSeek says most of the engine code for it exists) |
| SWA bounded replay and async scheduling | The autoscaling and fault-tolerance platform |
| Structured outputs, deterministic sampling | Support for any model other than V4.1 Flash |
DeepSeek says the cache and balancer code is tied to its internal infrastructure. It describes the design in its deployment Q&A and promises future model code and performance work will land in DeepInfer on GitHub, but the missing pieces have no release date.
What You Need to Run It
- Linux, GCC 13 or newer and Python 3.12.
- CUDA 13 for NVIDIA (SM100 class GPUs) or CANN 9.2 for Huawei Ascend 950.
- Build with
./build.shafter settingDEEPINFER_ACCELERATORtonvidiaorascend. - The default checkpoint is DeepSeek-V4.1-Flash-1017 on Hugging Face, downloaded on first run.
- Run the offline example with
python examples/offline_inference.py.
This release gives you offline inference, meaning you run prompts through the engine yourself. It is not a ready-made chat server. DeepSeek’s own published benchmarks were measured on 4 SM100 GPUs for prefill and 64 for decode.
Hardware Reality Check
DeepSeek says a 1 prefill + 1 decode setup can run, but the smallest efficient unit is 64 GPUs for prefill plus 64 for decode. For caching, it recommends storage servers with 20 SSDs of 30.73 TB each and at least 40 GB/s read speed, plus about 400 Gbps of networking per 8 GPUs if you split prefill and decode across buildings.
That is cloud-provider territory. If you only want to use V4.1 Flash, you do not need this. You can call it through the API, or read about the earlier generation in our DeepSeek V4 features guide.
Who Should Care
| You are | Should you care? | Why |
|---|---|---|
| Cloud or hosting provider | Yes | You can serve V4.1 Flash with the same engine DeepSeek uses, then add your own cache and routing |
| AI infrastructure engineer | Yes | Real production design choices to study: wide expert parallelism, caching and speculative decoding |
| Startup with a few GPUs | Maybe | Good to learn from, but you will not get the full cost advantage without the missing pieces |
| Developer using the API | Not directly | Providers may get faster and cheaper over time, but nothing changes for you today |
| Hobbyist at home | No | It targets data-center NVIDIA and Huawei chips |
Frequently Asked Questions
Is DeepInfer free and open source?
Yes. DeepInfer is released under the Apache 2.0 license, which allows commercial use. The model weights have their own license on Hugging Face.
Which models does DeepInfer support?
Only DeepSeek V4.1 Flash today. DeepSeek says code for future models will be added to the same project.
Which hardware does it run on?
NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. Batch-invariant inference and deterministic sampling work on NVIDIA only.
Can I run DeepInfer on my laptop or a single gaming GPU?
No. It targets data-center chips, and the model itself is very large. It is not meant for consumer hardware.
Is the full DeepSeek production system included?
No. The KV cache system, the data-parallel load balancers and the full vision/prefill/decode disaggregation are not in the release.
Does DeepInfer make DeepSeek cheaper for me?
Not directly. DeepSeek says the engine lets it serve more requests with less compute, which supports low prices, but your API price is set by DeepSeek.
How does DeepInfer compare with vLLM or SGLang?
DeepSeek built it on vLLM and drew ideas from SGLang and TensorRT-LLM. No independent head-to-head benchmarks exist yet, so I will not claim it is faster.
What is DSpark?
DSpark is DeepSeek’s speculative decoding method. It scores each drafted token by how likely it is to be accepted and verifies only the best ones, saving verification work.
Where do I start?
Read the DeepInfer repository on GitHub, build it on a supported machine, and try the offline example with the V4.1 Flash checkpoint.
My Take
The most useful part for most readers is not the code, it is the explanation. DeepSeek is openly showing how a frontier-scale model is served cheaply: split the work, spread the experts, cache aggressively and only verify the guesses worth verifying. If you build or buy AI infrastructure, study that Q&A. If you just use the model, DeepInfer is a good sign that serving costs keep falling, but it changes nothing in your workflow today. I would wait for independent benchmarks before treating any speed claim here as settled.
