Disaggregated inference serving Market Outlook 2026–2033: Growth Drivers, Key Companies, Strategic Developments & Adoption Forecast — A Meticulous Next™ Foresight Brief

This Meticulous Next™ brief examines disaggregated inference serving, in which the two stages of answering a prompt, reading the prompt and writing the reply, run on separate pools of hardware. It tests how far this design becomes the standard way to serve large AI models by 2033, and what that means for chipmakers, cloud providers, serving software vendors and the companies that pay for inference.

The idea moved from research to product in two years. The DistServe paper of 2024 showed that a single large prompt could slow the writing of other replies by 2 to 30 times, and by 2025 nearly every major serving framework supported separating the stages, according to Hao AI Lab. The conclusion is that the design is no longer an experiment, and buyers now ask which stack runs it best.

The largest chip supplier released an open framework. NVIDIA announced Dynamo at GTC on 18 March 2025 as an open source framework that separates prefill and decode, routes requests by where the cache sits and moves cache between memory tiers, and it claims up to 30 times more requests served for DeepSeek-R1 on GB200 NVL72 and more than double the throughput for Llama 70B on Hopper chips. The conclusion is that the chip supplier gives away the software to sell more hardware, and the figures are the company own projections.

Open stacks compete with it. SGLang reported DeepSeek-R1 serving on 96 H100 chips with 24 GPUs for prefill and 72 for decode in May 2025, and later reported 3.8 times the prefill and 4.8 times the decode throughput on GB200 NVL72 in September 2025, while vLLM reported about 2,200 tokens per second per H200 chip with the llm-d 0.3 stack in October 2025. The conclusion is that several open stacks reach similar designs, so value moves from the basic split to cache management, routing and operations.

Different chips for each stage are next. AWS and Cerebras announced in March 2026 a service on Amazon Bedrock in which Trainium handles prefill and the Cerebras CS-3 handles decode, and NVIDIA announced Groq based racks to pair with the Vera Rubin platform at GTC 2026. The conclusion is that disaggregation lets buyers mix chip types by stage, which opens the market to specialised silicon and raises the value of software that places work across chips.

The brief treats disaggregated inference serving as an accelerating signal with a seven year horizon to 2033. It identifies which vendors capture value as inference becomes the largest AI cost and as stages move to separate hardware. It is a focused 99-page decision brief that presents an indicative trajectory rather than a segmented market model.

The two stages stress hardware differently. Prefill reads the whole prompt and is limited by computing power, while decode writes one token at a time and is limited by memory bandwidth. Running both on the same chip makes each interfere with the other. The conclusion is that separating them lets each pool be sized and tuned for its own job, which raises the number of requests served for the same hardware.

The cache connects the pools. The result of reading a prompt is stored in a key value cache that must be moved to the decode pool, so fast links, cache aware routing and offloading to host memory or storage decide the benefit. NVIDIA Dynamo provides a planner that shifts GPUs between pools against latency targets, a router that tracks where the cache sits and a transfer library for moving data. The conclusion is that the software that places and moves cache carries much of the value and is more than a thin layer on top of the models.

Open and vendor stacks overlap. Dynamo supports TensorRT-LLM, vLLM, SGLang and PyTorch as backends, and projects such as llm-d, Mooncake and LMCache cover scheduling and cache storage. The conclusion is that buyers can mix components, which keeps switching costs low and pushes vendors to compete on operations, support and results.

Hardware can differ by stage. SambaNova uses GPUs for prefill and its SN50 chips for decode, AWS pairs Trainium for prefill with Cerebras CS-3 for decode and NVIDIA plans Groq based racks beside Vera Rubin. The conclusion is that specialised decode chips gain a way into data centres that are built around general GPUs, and the serving software decides which chip handles each stage.

The limits are network and cache transfer cost, the complexity of running several pools and figures that are reported by the vendors themselves. The brief tests each limit against the evidence to 2033.

The disaggregated inference serving market, covering orchestration software, key value cache management, request routing and managed services that run the prefill and decode stages of AI model inference on separate hardware pools, and excluding GPU and accelerator hardware, rented compute capacity, per-token model access fees and inference optimisation software that does not separate the two stages, is valued at USD 0.8–1.2 billion in 2026 and is projected to reach USD 7.25–10.87 billion by 2033, at a CAGR of 37%.

The 2026 base rests on subscriptions, licences and managed services for orchestration, cache management and routing sold to cloud providers, model developers and enterprises, alongside open source stacks that carry support fees. Growth then follows the volume of inference tokens served, the share of deployments that separate the stages and the share of operators that buy software rather than build it.

The forecast is sensitive to the pace of reasoning and agent workloads, the spread of mixed chip deployments and the degree to which open source stacks reduce paid demand.

Scenarios.

The base case assumes steady growth in reasoning and agent workloads and steady support from open stacks, bringing the inflection to about 2028 and a 2033 value near the middle of the range. An accelerated case assumes fast adoption of mixed chip deployments and long context agents that make cache the main cost, moving the inflection to 2027 and lifting the 2033 value to the top of the range. A delayed case assumes that open source stacks meet most needs for free and that new chips remove the need for separation, moving the inflection to 2030 and holding growth at the bottom of the range.

Growth drivers

• Growth of reasoning and long context agent workloads.
• Higher requests per chip from separating prefill and decode.
• Pressure on cost per token at large providers.
• Entry of specialised decode and prefill chips.

Enablers

• Open source frameworks such as Dynamo, vLLM and SGLang.
• Fast interconnects and transfer libraries.
• Cache offloading to host memory and storage.
• Cloud services that expose disaggregated serving.

Restraints and barriers

• Network and cache transfer cost.
• Complexity of running several pools.
• Open source stacks that reduce paid demand.
• Vendor-reported results without shared benchmarks.

The direction of disaggregated inference serving is set by the growth of reasoning and agent workloads, the gain in requests per chip, pressure on cost per token, the supply of specialised chips and the readiness of operators to run several hardware pools.

The key participants in disaggregated inference serving and the software that splits prefill and decode include NVIDIA Corporation, Amazon Web Services, Inc., Cerebras Systems Inc., SambaNova Systems, Inc., Alphabet Inc., Microsoft Corporation, Meta Platforms, Inc., Oracle Corporation, International Business Machines Corporation, CoreWeave, Inc., Perplexity AI, Inc., Advanced Micro Devices, Inc., Databricks, Inc., Hugging Face, Inc., Dell Technologies Inc., Super Micro Computer, Inc., Cisco Systems, Inc., Arista Networks, Inc., Micron Technology, Inc., Lambda, Inc., Groq, Fireworks AI, DeepInfra, Modular, WEKA, DeepSeek, Intel Corporation, Broadcom Inc., Marvell Technology, Inc., Samsung Electronics Co., Ltd., SK hynix Inc., Red Hat, Snowflake Inc., Baseten, Together AI, the vLLM project, the SGLang project, the llm-d project, the Mooncake project, LMCache, LMSYS, Hao AI Lab and SemiAnalysis.

The brief profiles representative participants in each archetype and assesses which are positioned to lead as disaggregated serving becomes the standard design for inference.

NVIDIA Corporation leads among chip suppliers with a full serving framework, Amazon Web Services, Inc., Microsoft Corporation and Alphabet Inc. lead among clouds that run inference at scale, and Cerebras Systems Inc. and SambaNova Systems, Inc. lead among specialised chip makers that target the decode stage. Competitive intensity is rising as open source stacks, chip suppliers and clouds all offer serving software.

Where value migrates.

In 2026 value sits with chip suppliers that give away serving software to sell hardware and with clouds that run the largest deployments, while open stacks provide the basic split for free. By 2028 it moves toward cache management, routing and placement software and toward network and memory suppliers, because the gain from the split depends on moving cache fast and placing work well. By 2033 it settles with platforms that mix chips by stage and run the lowest cost per token, where scale, cache efficiency and support carry the margin. Margins hold for chip suppliers with serving stacks and clouds with scale, and narrow for vendors that sell only the basic split that open stacks provide.

Support in all major serving stacks, an open framework from the largest chip supplier, clouds that expose the design as a service and growth in reasoning workloads support growth, while network and cache transfer cost, the complexity of several pools and open source alternatives hold back the pace.

A new chip that removes the cost of mixing stages, an open stack that meets all needs or a slowdown in long context agents could change the pace. The brief therefore tracks four indicators: the share of inference served on separate pools, the cache hit rate, the cost per million tokens and the number of mixed chip deployments.

Disaggregation at large providers first, then mixed chips by stage, then separated stages and shared cache tiers as the default design is the expected order of change.