How does Mooncake's KVCache-centric architecture improve LLM serving throughput?

Why KVCache reuse sets the throughput limit

Mooncake is the serving platform behind Kimi, and its starting point is an optimization problem rather than a single bottleneck: maximize overall effective throughput while satisfying latency-related Service Level Objectives (SLOs). The two SLOs named in the paper are time to first token (TTFT), which the prefill stage determines, and time between tokens (TBT), which the decoding stage determines.

The paper argues that KVCache scheduling sits at the center of this problem because the two usual ways to raise throughput pull against those SLOs. Reusing KVCache reduces the computation a request needs, but fetching that cache from a remote location lengthens TTFT. Increasing the number of tokens in each batch raises Model FLOPs Utilization (MFU), but a larger batch lengthens TBT. Mooncake is therefore organized around deciding where cached state lives and which instance handles each request, not around a single faster kernel.

Separating prefill from decoding protects TBT

The architecture splits the cluster into a prefill pool and a decoding pool, connected by RDMA for inter-node KVCache transfer. Each pool has its own objective: prefill aims to maximize cache reuse subject to the TTFT SLO, a minimum MFU, and KVCache fitting within DRAM; decoding aims to maximize throughput subject to the TBT SLO and KVCache fitting within VRAM. A global scheduler called the Conductor selects a prefill and decoding pair for each request.

This separation matters because the paper's simulated experiment shows long-context requests severely disrupting vLLM's decoding stage, forcing vLLM to process requests individually rather than in batches. Mooncake keeps batching while its two-stage design limits prefill's impact on decoding, and the paper reports that it never breaks the TBT SLO in that experiment. The cost is a coordination problem: the Conductor pre-selects a decoding node based on current load, but the local scheduler rechecks the TBT SLO after prefill, and a rejection at that point wastes the prefill work already done.

Idle CPU, DRAM and SSD become a cache tier

Mooncake uses the underutilized CPU, DRAM and SSD resources of the GPU cluster to build a disaggregated KVCache pool, so cached blocks can outlive the VRAM of a single instance. The paper's trace analysis shows why placement decisions matter: over 50% of cache blocks remained unused while certain blocks were accessed tens of thousands of times, which the authors say makes replicating hot blocks necessary to avoid transfer congestion.

The same trace, sampled from a one-hour period and released as 23,608 entries with timestamp, input_length, output_length and hash_ids, gives the workload shape Mooncake is tuned for. Average input length is 8,590 tokens against an average output length of 182, an input-output ratio of roughly 720. In a single global cache pool, raising capacity from 1,000 to 50,000 blocks lifted the hit ratio from 34% to 55%, with little further gain beyond that in this sample; the authors caution that the sample is only a subset of real workloads and that required capacity should scale proportionally in practice. LRU performed best on these patterns.

Chunked pipeline parallelism targets long prefill

The paper keeps a separate prefill pool even though chunked prefill could inline prefill work into decoding batches, for two stated reasons: prefill nodes need different cross-node parallelism for long contexts, and separation creates an opportunity to save VRAM. For long inputs, extending tensor parallelism across more than one 8-GPU node requires two RDMA-based all-reduce operations per layer, which cuts prefill MFU, while sequence parallelism still communicates across nodes frequently and competes with KVCache transfer for network resources.

Mooncake instead implements chunked pipeline parallelism (CPP): every X nodes in the prefill cluster form a pipelined group, each request's input is split into chunks no longer than prefill_chunk, and different chunks are processed simultaneously by different nodes. The paper argues this needs cross-node communication only at pipeline-stage boundaries, which can overlap with computation, and that it fits both short and long contexts without frequent dynamic repartitioning. It also overlaps KVCache loading and storing with layer-by-layer computation, since prefill is computation-bound.

The 75% gain comes from TBT compliance

The headline real-workload result is easy to misread, because the two systems differ in how they meet the two SLOs. With 10 prefill and 10 decoding instances for Mooncake and 20 vLLM instances, and with TTFT capped at 30 seconds and TBT at 0.1 seconds per token, the TTFT distributions of the two systems were nearly identical, with almost 100% of requests meeting the TTFT SLO. The split appears in TBT: approximately 100% of Mooncake requests satisfied it, versus only 57% for vLLM, with some vLLM requests showing extremely high TBT.

That difference is what produces the reported capacity gain: Mooncake processed approximately 75% more requests while adhering to the SLOs. The number is a comparison of served requests under the same SLO targets in this replay setup, not a claim that Mooncake is 75% faster per request. The simulated long-context experiment is a separate comparison with a different baseline configuration, where vLLM processed requests individually to avoid decoding disruption and Mooncake's advantage ranged from 50% to 525% across 16k, 32k, 64k and 128k prompts.

Early rejection avoids wasted prefill work

Because Mooncake operates under heavy overload rather than assuming every request will be processed, admission control becomes part of throughput. A baseline that rejects requests based on load before either stage starts wastes resources by turning away requests that prefill could have handled. The paper's Early Rejection strategy considers system load more comprehensively, and Early Rejection based on Prediction goes further by forecasting decoding load after the prefill stage.

In an overload experiment with 8 prefill and 8 decoding instances replaying 23,000 real requests at 2x speed, the baseline rejected 4,183 requests, Early Rejection rejected 3,771, and prediction-based rejection rejected 3,589. The paper attributes the improvement to avoiding unnecessary prefill computation and to smoothing load fluctuations between the two pools. Its prediction is system-level rather than request-level: it assumes a uniform decoding time per request and estimates batch counts or TBT status, because predicting individual output lengths is described as costly or inaccurate under scarce resources.

Where the throughput claim stops

The evidence supports a specific claim: for long-context, prefill-heavy workloads resembling Kimi's trace, where input tokens vastly outnumber output tokens, disaggregating prefill from decoding and scheduling around KVCache raises the number of requests served within fixed TTFT and TBT targets. The 525% figure belongs to simulated scenarios with particular prompt lengths and a vLLM baseline forced to process requests individually; the 75% figure belongs to a real-trace replay at a specific cluster size and SLO setting. Neither number transfers automatically to other model sizes, other SLO thresholds, or workloads with a different input-output ratio.

The paper also notes that its cache-capacity findings come from a one-hour sample, and that request-level output-length prediction is left for future work, which is why the deployed rejection policy relies on coarser system-level estimates. The practical priority the paper establishes is therefore not raw batch size but the pairing of cache reuse with SLO compliance: the TBT gap, 100% versus 57% of requests meeting the target, is the mechanism behind the extra served requests, and any deployment decision should be read against that latency target rather than against throughput alone.

Sources used in this answer