Something unusual is happening in enterprise infrastructure.
For the past decade, the dominant narrative was simple: move everything to the cloud, shut down your datacenters, and let hyperscalers worry about the hardware. CIOs who were slow to adopt cloud were considered behind the curve.
That narrative has not reversed — but it has become significantly more complicated. AI workloads are creating a class of infrastructure demand that is forcing IT leaders to rethink decisions they thought were already made.
What Has Changed
The original cloud migration wave was driven by web applications, databases, and productivity tools. These workloads have predictable compute patterns, tolerate network latency, and scale gracefully. Cloud is genuinely well-suited for them.
AI training and inference workloads are different in almost every relevant dimension.
GPU density requirements are unlike anything traditional infrastructure planning anticipated. Training a large language model or running inference on a frontier model at scale requires GPU clusters that consume megawatts of power in a single rack. The physical density — power, cooling, and space — required for serious AI compute is creating infrastructure constraints that neither enterprise datacenters nor some cloud regions are equipped to handle today.
Data gravity is becoming a decisive factor. AI models need to be trained and fine-tuned on enterprise data. When that data is large, sensitive, or subject to regulatory requirements, moving it to cloud for AI processing creates compliance exposure, egress costs, and latency that directly affect model performance. The closer the compute is to the data, the better the outcome.
Inference latency matters for real-time AI applications in ways that batch workloads never required. A customer-facing AI agent that takes three seconds to respond because of round-trip latency to a distant cloud region is not commercially viable. Latency requirements are pushing inference workloads closer to the edge — and in many cases, back on-premises.
Cost unpredictability at AI scale is a genuine shock for organisations that adopted cloud partly for its pay-as-you-go model. GPU compute on hyperscale cloud is expensive, and AI workloads that run continuously at scale can generate monthly bills that dwarf traditional cloud spending. Several large enterprises have publicly disclosed that AI cloud costs exceeded projections by multiples.
The On-Premises Renaissance
"On-premises" is not the same word it was in 2015. The modern on-premises AI infrastructure stack — purpose-built GPU servers, liquid cooling, high-speed interconnects, and NVMe storage — bears little resemblance to the legacy datacenters that enterprises were being encouraged to decommission.
The economics are shifting. When an organisation runs sustained AI workloads — inference serving, continuous model fine-tuning, embedding generation at scale — the total cost of ownership for on-premises GPU infrastructure can be significantly lower than equivalent cloud capacity over a three to five year horizon. The crossover point depends on utilisation rate, but organisations running AI workloads at 60% or more GPU utilisation consistently are finding on-premises economics compelling.
Control is the other driver. On-premises infrastructure gives organisations complete control over their data, their model weights, their compliance posture, and their upgrade cycle. For industries like financial services, healthcare, and government, this is not a preference — it is frequently a requirement.
Several hyperscalers have responded to this by offering dedicated infrastructure options — private cloud zones, dedicated hosts, and co-location partnerships — that attempt to combine on-premises control with cloud management abstractions. These models are gaining traction but introduce their own complexity and cost considerations.
The Cloud Case Remains Strong
On-premises is not the answer for everyone, and the cloud case for AI workloads is genuinely strong in several scenarios.
Burst compute for training runs is where cloud remains unmatched. Training a new model version or running a large-scale fine-tuning job is inherently episodic. Provisioning on-premises GPU capacity sized for peak training demand means expensive hardware sitting idle between training runs. Cloud's elasticity solves this problem cleanly.
Speed to experimentation is a real cloud advantage. Spinning up a GPU cluster for a proof of concept takes minutes on cloud and weeks or months on-premises. For organisations in the exploration phase of AI adoption, cloud accelerates the learning cycle in ways that matter competitively.
Geographic distribution for global inference serving is another cloud strength. Serving AI inference with low latency to users across multiple continents is operationally complex on-premises. Cloud's global footprint handles this naturally.
Managed AI services — foundation model APIs, vector databases, embedding services, RAG infrastructure — are maturing rapidly on cloud platforms. Organisations that want to build on top of AI rather than operate AI infrastructure will find cloud's managed service ecosystem significantly more developed than on-premises alternatives.
The Emerging Answer: Hybrid by Design
The most sophisticated IT leaders are not choosing between on-premises and cloud — they are designing hybrid architectures that assign workloads to the environment best suited for them.
The pattern that is emerging across large enterprises looks something like this. Sensitive data and regulatory workloads stay on-premises, with fine-tuning and inference for models trained on that data running locally. Burst training, experimentation, and globally distributed inference run on cloud. Foundation model APIs are consumed from cloud for use cases that don't require proprietary data. Edge inference handles latency-sensitive applications at the point of consumption.
This architecture requires more sophisticated governance than a pure cloud or pure on-premises approach. Network architecture, data pipelines, model deployment, and cost management all become more complex when workloads span environments. The operational burden is real.
But the organisations that get this architecture right are building infrastructure that is both economically competitive and operationally resilient — not locked into a single vendor's pricing decisions or capacity constraints.
What IT Leaders Should Be Doing Now
The worst position to be in is making infrastructure decisions reactively — waiting until AI workloads are in production before figuring out where they should run.
Three things IT leaders should be doing now:
Build an AI infrastructure inventory. Catalogue every AI workload running or planned — training, inference, embedding, retrieval — and characterise each by compute intensity, data sensitivity, latency requirement, and usage pattern. This inventory is the input to every infrastructure decision that follows.
Model the economics at your actual utilisation. Cloud pricing calculators and vendor TCO models are starting points, not answers. Build your own model based on your actual workload patterns, your organisation's cost of capital, and realistic utilisation assumptions. The crossover point between cloud and on-premises economics is highly sensitive to these inputs.
Design for portability. Container-based deployment, standardised model serving interfaces, and infrastructure-as-code practices make it significantly easier to move workloads between environments as requirements evolve. Organisations that built cloud-native AI infrastructure with proprietary dependencies are finding it expensive and slow to adapt as the economic and technical landscape shifts.
Conclusion
The infrastructure decisions that IT leaders make in the next 12 to 18 months will shape their organisation's AI capabilities for the next five years.
The era of simple answers — just move everything to cloud — is over for AI infrastructure. The workloads are too diverse, the economics too variable, and the stakes too high for a single-environment strategy.
The IT leaders who will deliver the best outcomes are those who resist the pull of both the legacy on-premises mindset and the reflexive cloud-first instinct, and instead design infrastructure architectures that match the actual characteristics of the workloads they need to run.
That is harder than following a trend. It is also the job.
