Energy, not intelligence, is the real bottleneck in AI. Why compute per joule, datacenter power, cooling, and on device inference decide what actually ships.
Follow any AI roadmap far enough down the arrows and you don't arrive at a smarter model. You arrive at a power substation in a field somewhere, humming, with a three-to-five-year queue of datacenters waiting on it. That queue is the real roadmap. Everything drawn above it is optimism.
I run engineering at a startup in Dubai, so I feel this from the small end of the telescope. My bills are cloud invoices, not grid interconnects, and I have never personally negotiated a megawatt. But the shape is identical all the way up the stack: the thing that stops us is almost never a shortage of intelligence in the system. It is the cost of running that intelligence, paid in watts, in dollars, and in heat we have to shove somewhere else. I have come to believe that for the next decade energy is the binding constraint on computing, and the engineers who plan around thermodynamics will ship things that the ones planning around benchmarks never will.
This post is my argument for treating energy as a first-class engineering input — the same way you already treat latency, memory, and dollar cost — and a concrete playbook for building AI features that survive an energy-constrained world.
Here is the uncomfortable chain, one link at a time. Better models want more compute. More compute wants more chips. More chips want more power and, the part everyone skips, more cooling. Power and cooling want land near cheap electricity and water, plus a grid connection that utilities are now quoting in years rather than months. Somewhere in the Gulf, in Virginia, in Ireland, a utility engineer with a spreadsheet is the actual gatekeeper of your product timeline. Not your model provider. Not your CTO. A grid planner you will never meet.
We got used to treating compute as elastic and effectively free at the margin. For a decade that was true enough to build on. You provisioned more, the bill nudged up, you moved on with your day. That era is closing. The marginal unit of AI compute is now priced against a physical resource that does not scale on a credit card: electrical power delivered to a specific patch of ground, plus the ability to reject the heat that power turns into.
When a resource stops being elastic, it stops being a line item and becomes an architecture decision. That is the whole thesis of this post in one sentence. Energy has crossed that line, and most of us are still writing code as if it hasn't.
For most of computing history, efficiency rescued us. Performance per watt roughly doubled on a comfortable cadence, so you could ignore energy entirely and let the next process node bail you out. Dennard scaling handed us free lunches: transistors shrank and drew proportionally less power, so clock speeds climbed without the chip melting. You did nothing, and next year's hardware ran your same code faster and cooler.
Dennard scaling ended around 2006. That is not a hot take, it is the reason your CPU stopped getting meaningfully faster per core and started sprouting core counts instead. We have lived on architectural cleverness ever since: wider vectors, specialized units, GPUs, then tensor cores, then whole accelerators that do exactly one kind of matrix math and do it ferociously well.
Each of those jumps bought a large one-time efficiency gain. None of them restarted the smooth exponential we used to lean on. And they are getting harder and more expensive to find:
So the compute-per-joule curve is still climbing, but it is bending toward flat at exactly the moment demand goes vertical. When a rising line meets a flattening one, you don't get a graceful graph. You get a wall. And the wall is not built out of algorithms. It is built out of joules.
Somewhere in the last few years, "the cloud" quietly became heavy industry, and almost nobody updated their mental picture. A modern AI datacenter has more in common with a smelter or a chemical plant than with the server closet you are probably imagining. It draws power on the scale of a small city, drinks serious water for cooling, and gets sited by the same brutal logic as an aluminium plant: go where energy is cheap and abundant, and where the climate or the coast helps you dump the heat.
That has three consequences most application engineers never think about, because for our whole careers the datacenter was an abstraction with a green checkmark next to it.
Compute clusters near cheap power. Cheap power is not evenly distributed, and it is frequently nowhere near your users. The Gulf is racing to build here precisely because energy is cheap and land is plentiful, but the same physics that makes a site perfect for training a model makes it lousy for serving a user in Jakarta. The map of where compute wants to live and the map of where your users actually are have started to diverge, and no amount of clever caching fully closes that gap.
If the efficient place to run a big model is a thousand kilometres from your user, every single request pays for that distance in milliseconds you cannot refactor away. You do not get to code past the speed of light; nobody has patched that one yet. So "where is the cheap energy" and "where is the low latency" pull in opposite directions, and you are forced to choose per workload instead of quietly assuming you can have both. That choice used to be a config value. Now it is a strategy.
Cooling is not a footnote you can hand to the facilities team. A meaningful fraction of a datacenter's total energy is spent purely on moving heat around, and the newest accelerators run so hot that air cooling is surrendering to liquid. If you cannot reject the heat, you cannot run the chips, however many you bought and however good your model is. I watched a small on-prem rack for a client throttle itself into uselessness one August afternoon because the room's AC could not keep pace with what the GPUs were dumping into it. The model was fine. The silicon was fine. The room lost. That is the entire industry's problem, rendered in miniature in one overheated office in the Emirates.
The usual argument for running models on the phone is privacy. Real, but secondary. The deeper reason on-device inference matters is that the device is the single most efficient place you can spend a joule, because the joule is already sitting there in a charged battery and the network round trip has vanished entirely.
Trace where energy actually goes in a typical cloud inference call:
Now compare that against a small model running on the phone's neural engine, sipping from a battery the user already charged overnight for free. For a large class of tasks — summarization, classification, autocomplete, on-device search, simple extraction — the on-device path wins on total energy by a wide margin, even though the phone chip is objectively far weaker than the datacenter accelerator. Weaker hardware, cheaper answer. That inversion is the whole point.
As a Flutter developer this is where it stops being philosophy and becomes a code review comment. The pattern I keep reaching for looks like this:
Future<Summary> summarize(String text) async { // Cheapest joule is the one already in the user's battery. // Try the device first, and only escalate on a real failure. if (await OnDeviceModel.isAvailable && text.length < 4000) { try { return await OnDeviceModel.summarize(text); } on OnDeviceModelException { // Genuine failure only. Never fall through by default. } } // The datacenter is the escalation path, not the front door. return await CloudModel.summarize(text);}The load-bearing idea is the ordering, not the code. The datacenter is the escalation, never the default. Ship as much intelligence as you can out to the edge where the energy is already spent, and reserve the expensive, hot, far-away compute for the cases that genuinely earn it. That is fundamentally an energy decision that happens to also be a privacy win, a cost win, and a latency win all at once. The privacy story is what marketing puts on the slide. The joules are the actual reason the pattern is correct.
One caveat worth naming: a device-first path is not free of complexity. You now maintain two inference code paths, you ship and version model weights on the client, and you have to decide what "real failure" means precisely enough that you never silently degrade quality. Get that boundary wrong and you either waste the battery you were trying to save or you route everything to the cloud and gain nothing. The discipline is in the escalation condition, not the fallback.
There is a quiet reordering happening in what counts as the best model for a job. For years the honest answer was "the biggest one you can afford to call." When compute felt free, more capability was strictly better, and you paid a rounding error for it.
Once energy is the binding constraint, the question changes to capability per joule, and a lot of the old ranking simply inverts. A model that is 90% as good but runs at a tenth of the energy is not a compromise you apologize for. It is usually the correct engineering choice, full stop. The missing 10% almost never justifies a 10x energy bill on a per-request basis, and at scale that bill is precisely what decides whether the product has a business or a very impressive demo.
I have lived a smaller version of this on Firebase, and it left a mark. On one project our per-screen read counts had crept up to roughly 40 Firestore reads just to paint a single dashboard, because fanning out reads was easy and each one felt free. We restructured the data model, denormalized the parts that were read together, and cut it to 3. The feature did not get one bit smarter. It got affordable, and affordable is exactly what let it ship to every user instead of a lucky few on a generous plan. Energy-bound AI is that same lesson with a much larger blast radius: the thing that kills you is rarely the capability you lack, it is the running cost of the capability you already have.
A useful way to reframe the whole shift is to notice which questions you are actually asking in design reviews:
| Old question | New question |
| --- | --- |
| Is this the most capable model? | What is the capability per joule? |
| Can we afford more compute? | Can we afford the heat and the grid slot? |
| How do we scale up? | How do we scale down without users noticing? |
| Is latency acceptable? | Is latency acceptable given where the cheap power is? |
"Good enough, locally, now" is quietly becoming a stronger position than "best, in a datacenter, eventually." Not always, and I am not pretending it is a universal law. But often enough that it deserves to be your default hypothesis rather than the fallback you reach for after the frontier model blows the budget.
Run a thought experiment. Impose a hard rule: for the next five years, the energy budget for your product's inference is effectively frozen. You cannot buy your way out with more racks, because there are no more racks and no more megawatts on offer at any price you can stomach. What do you actually do?
You relearn a set of skills the free-compute decade quietly let us forget.
None of this is exotic or new. It is the everyday discipline of an embedded engineer or an early-web engineer, back when resources were tight and you felt the weight of every byte and every cycle. That discipline atrophied across the industry because for fifteen years it genuinely stopped mattering. It is about to matter enormously again, and the teams still carrying it as muscle memory will move faster than the ones rediscovering it in a panic when the invoice arrives.
I will make this concrete, because vague futurism is worthless and easy. Here is what I actually expect to hold up over the next decade, stated plainly enough that you can hold me to it.
The trap to avoid is assuming intelligence is the scarce thing. Intelligence, in the narrow sense of capable models, is getting cheaper and more abundant at a genuinely startling pace. What is getting scarce, expensive, and slow to provision is the raw physical capacity to run it. If your plan quietly assumes energy stays elastic and free at the margin, then your plan has a hole in it precisely where the substation is, and you will find that hole at the worst possible time.