Why inference latency is the next networking problem
A client of ours, an esports organization running a live broadcast operation, called us in last year to fix what they thought was a model problem. Their AI-driven overlay was lagging behind the stream. By the time the system said a player was about to push, the push was over. The broadcast team had been told the model needed more optimization. They'd already burned a quarter on prompt engineering and a smaller model. Nothing helped.
The model was fine. Every inference call was making a round trip to a GPU cluster three regions away.
I keep seeing this. A team builds something that works in dev, ships it, and watches the user experience degrade in ways no model evaluation catches. The instinct is to optimize what you know how to optimize: the prompt, the model, the cache. The network is treated as background. It's been background for a decade, and most engineering teams have not had to think about routing since their last CDN integration.
That assumption now costs you users.
Where the milliseconds actually live
Walk through the math on a single text generation request. The user-to-edge hop is ten to thirty milliseconds. The edge-to-inference-provider hop is fifty to a hundred. The model itself, if it's an 8B parameter model, takes five hundred to two thousand. Cloudflare's own breakdown of a typical inference request lines up with those numbers.
The model is the slowest part. Of course it is. The boring answer is that this doesn't matter as much as you think.
Watch what happens when you chain inference calls. Agents. Multi-step reasoning. Retrieval pipelines. Anything where the model calls the model. Each round trip to the inference endpoint adds fifty to a hundred milliseconds that has nothing to do with the model thinking. For a five-step agent loop, that's half a second of pure network time before the model has done any work. For the broadcast overlay, it was enough to make the output arrive after the moment it described.
Inference latency is a budget. The network keeps taking from it.
Why the assumption persists
For most of the last fifteen years, the network has been good enough to ignore. Your hyperscaler had a network. Your CDN had a network. Users got reasonable latency to a region of your choosing, and if they didn't, there was a region closer. The work of routing was abstracted away, and that abstraction was a gift. It let teams focus on what was on top of the network instead of underneath it.
AI workloads break the abstraction in two ways. Inference endpoints are not where your application is; most enterprises run inference against a small number of GPU-heavy regions, often a different cloud than the rest of their stack. And the workloads are chatty in a way traditional web traffic isn't. A single user interaction can fan out into twenty inference calls. Each one pays the full network cost.
The network was solved enough to ignore for static content and most APIs. It is not solved for this. Teams treating inference latency as a model problem are tuning the wrong variable.
Carrier-agnostic, in practice
When we engaged on the broadcast project, the first thing we did was map where their inference traffic was actually going and which carriers it was traversing to get there. They had never seen this map. It was worse than they expected: traffic doubling back across the country, peering arrangements that made sense in 2019 routing around a problem that no longer exists, inference calls going through carriers that had no business handling them.
Carrier-agnostic is the posture of not being married to any of those carriers. We design the path from your application to your inference endpoint to be the path that's actually fastest today, not the path your cloud provider's defaults prefer. The work is partly diagnostic (what is the traffic doing, where is it going, why) and partly negotiated (which carriers, which peering, what failover). Done well, it produces latency reductions in a range that no amount of model optimization can match. Cloudflare and JD Cloud's recent partnership work projects up to 80 percent latency reductions on AI workloads from network routing changes alone.
For the broadcast client, the fix was moving inference to a closer region and renegotiating two peering arrangements. The overlay caught the stream the next week.
The trade-off
This is slower to set up than picking a hyperscaler and accepting their defaults. It requires the client to have an opinion, or to develop one, about routing decisions they have historically delegated. It requires a more senior team on their side to evaluate the options we put in front of them, because the questions are no longer "which region" but "which path, with what failover, through whom."
Some teams are not ready for that conversation. That is a real answer. If your AI workload is internal-facing, low-traffic, or latency-tolerant, the defaults are probably fine, and you should keep your attention on the model. We have told clients this.
The teams that should be having this conversation are the ones whose product experience depends on the speed of inference. Real-time interfaces. Agentic workflows. Anything streaming. The broadcast overlay didn't need a smarter model. It needed a shorter path, and the only people who could give it one were the people willing to look at the network.