You can run trillion-parameter open-weight models for around $3m per year, providing frontier output quality to 10,000 or more users concurrently, without any data leaving your corporate network.

I once led infrastructure teams at a major US telco, deploying rack-scale infra all over the US. On the build vs. buy continuum, I’m a strong build - that goes for hardware and software. So when I read news that agent pricing is a loss-leader and the frontier labs are pulling more and more agentic capability into the cloud, I start evaluating options. Looking at the latest generation of rack compute from Nvidia, I think the moment has arrived for big players to start repatriating their AI workflows.

Modern Inference Hardware

Let’s start with the hardware - the Vera Rubin NVL72 from Nvidia.¹ This is a beast of a machine, basically a network-connected chassis for distributing workload onto 72 monster GPUs. Each chassis has 20TB of GPU memory and an outrageously fast network. The latest revision supports a 100MW “AI Factory” rack-scale-out architecture that has been sold to CoreWeave and Google Cloud and others. That factory provides 40,000 GPUs at a total draw power of 2.5kW per GPU. Which is a handy figure to have, because we can back that into the TDP for a single rack - 180kW. This matters because AI workloads are measured at tokens per second per megawatt. And Vera Rubin has been benchmarked to deliver 37 million tok/s/MW.²

Translate that back to a single rack, and we’re talking 6.6 million tokens per second per rack, fully optimized and at full power draw. I took a look at my own usage, and over the past month my typical daily token usage sits around 350k tokens, centered around the workday. This is writing code, evaluating engineering designs, analyzing business data, running personal agents to review emails and meeting transcripts.

But looking at a single day the picture is a little more unusual. I have moments where usage spikes up near 30k tok/s, and a lot of moments where nothing happens, and a longer period of sustained work. For an individual the average tok/s is really low (10-20 tok/s over a workday), but usage is split into little bursts. Across thousands of engineers, agentic AI usage looks like an average. Discuss, develop, test, review, repeat. What this means is that we can easily support thousands of engineers on a single NVL72 rack. That’s engineers running design sessions, writing code, farming out workloads to agents with goals, performing analysis, or integrating LLMs into pull request and operational processes.

Scaling Hardware to Employees

Fermi estimation is useful for this - 10x the average consumption and halve the rack capacity and a rack can still support at least 20k engineers. These things are astounding. What does the rack cost? These racks are available at reported prices peaking above $8.8m apiece.³ Space, power, network switches, companion servers, and labor opex are all negligible in relation to the GPUs themselves. Call it $500k per year. And before you complain to me that systems engineers cost more than that - we’re talking about the Fortune 500 here, where there are already significant infrastructure teams deployed and engaged in meaningful work. Depreciate the server over three years in three almost-equal tranches, why not? (n.b. nobody depreciates computer equipment on a 3-yr schedule)

Average cost per engineer is a lie though - we all know engineering output follows a pareto distribution. At my company today, our top engineers are spending $10k+ per month on tokens, at API prices. So a system like this funds the unbridled activity of 30 or 40 engineers. This investment is within reach of pretty much every company in the Fortune 500. Companies are already having trouble with AI budgets, as top employees teach other employees how to use these tools effectively. And the initial clumsy attempts to manage costs are already in⁴ the⁵ news⁶. CFOs are setting tight budgets and in the process are forcing their staff to purchase personal subscriptions or submit to a cumbersome manage-by-exception model.

More Signals that Inference is Commodity

So what’s stopping every Fortune 500 from doing this today? Well, it’s a software problem and an expertise problem. Running multiple instances of DeepSeek/GLM/etc on an expensive machine, tuning the cache params and timeouts to support 10k concurrent users without blowing up your input processing and tanking capacity, building robust APIs and routing for accessing the service, security controls for users and for the agents themselves - these are all hard problems. Most of the hard problems have software to solve them already - vLLM and others address the core challenge of running LLMs, but the enterprise-ready stack doesn’t have a contender yet. While the tech press focuses on frontier models and benchmarks, the actual product is the harness around the model and the orchestration of models and users onto distributed equipment.

For evidence of this, look no further than the behavior of the frontier labs with their agents. Claude recently began encrypting the exchange between their agents and their servers to prevent their customers from inspecting agent traffic.⁷ And they attempt to move your workload to their cloud workers at every opportunity. The recent narrative around these agents being dangerous is as much about regulatory capture as it is about influencing CIOs to mistrust new investment in repatriated AI. But the trend is obvious - these companies are overinvested in training in pursuit of AGI (or whatever we’re calling it this week) and are transitioning towards an extractive business model that maximizes switching costs and minimizes the ability of competitors to replicate their offering.

We’re at the very beginning of on-prem AI. Models with sufficient quality didn’t exist until this year - Kimi, GLM, and DeepSeek are all outstanding (and all Chinese, which doesn’t matter when you’re self-hosting). The latest generation of hardware is finally at a point where it exceeds the needs of a large enterprise in a single chassis. Even if CFOs aren’t ready to pull the trigger on $10m worth of hardware, knowing the all-in hardware and opex for a self-hosted system helps to create a floor for negotiations with the big AI vendors. It’s hard to justify $50/Mtok API pricing when a rack pays that bill in less than a day of sustained use, with no surprises.

References