The Four Hidden Surcharges in Your GPU Bill

We’ll use one workload the whole way through: an inference API serving an open 70B-class model on four H100s, behind a customer-facing feature where a human is waiting on the response. It’s a setup that is very common these days, and is a very good example, because it gets hit by all four surcharges at once.

H100 on-demand pricing has fallen — VentureBeat, citing Cast AI’s tracking, puts it down from roughly $7.57 per GPU-hour in September 2025 to around $3.93 today. So four H100s running around the clock is about 4 × $3.93 × 24 × 30, or roughly $11,300 a month. That’s the “fare”. Everyone compares this number, and it’s almost never the number that decides your bill.

A note on the comparison before we start: $3.93 is a commodity on-demand rate. Named hyperscalers often charge more per H100, but they also publish the egress, NAT, and cross-AZ rates we use below. To make the strongest version of the argument, we’re pairing the cheapest plausible compute fare with a realistic hyperscaler data-movement and resilience posture.

Uploading data into a cloud is free, getting it back out is not, and that is deliberate. As of mid-2026, first-tier internet egress is about $0.09/GB on AWS, $0.087 on Azure, and $0.12 on GCP Premium: roughly four to six times what the same provider charges to store a gigabyte for a month.

The headline rate is accurate. Two line items quietly stack on top of it:

• The NAT Gateway- If your inference pods sit in a private subnet and reach the internet through a managed NAT, the default secure pattern on AWS every gigabyte gets an extra $0.045/GB data-processing fee on top of egress. That roughly doubles your effective rate before traffic leaves the building.
• Cross-AZ traffic- Spread replicas across availability zones for resilience and AWS bills $0.01/GB in each direction, landing under “EC2-Other,” where nobody goes looking.

Now let’s put numbers on our workload. Say it handles 30 million requests a month with an average response payload around 250KB once you count the completion, the echoed context, and the logs you ship out. That’s 30M × 250KB, or about 7.5 TB of internet egress:

• 7,500 GB × $0.09 ≈ $675
• NAT Gateway: 7,500 GB × $0.045 ≈ $338
• Cross-AZ chatter: assume replicas are split across two availability zones behind a load balancer, so roughly half the request/response volume about 4 TB, crosses a zone boundary. At $0.01/GB in each direction that’s $0.02 round-trip, or ≈ $80

That’s roughly $1,090 a month in data movement, about 10% on top of compute. For data-heavy workloads, transfer routinely lands in the 10-15% range, and on distributed multi-AZ architectures, higher still.

Cast AI’s 2026 State of Kubernetes Optimization Report measured production telemetry across about 23,000 clusters and found average GPU utilization of 5%. Organizations are provisioning roughly 20 times the GPU capacity their workloads use at any given moment. Co-founder Laurent Gil’s line is the whole problem in one sentence: an idle CPU core costs cents per hour; an idle GPU costs dollars.

And the thing you’re wasting is getting more expensive, not less. The same report notes AWS raised H200 Capacity Block prices about 15% in January 2026, the first time since EC2 launched in 2006 that a hyperscaler meaningfully raised GPU pricing rather than cutting it.

That 5% is the fleet average, and it includes a lot of forgotten dev clusters. The report’s best-tuned example: a 136-H200 cluster sustained 49%. So let’s be generous and assume our real-time API runs at 30%: well above the fleet average, still short of a well-managed cluster. That still means 70% of the four GPUs we’re paying for sit idle. Of the ~$11,300 in compute, only about $3,400 is doing useful work, and roughly $7,900 is the idle tax.

And you can’t just “turn off the idle ones.” Scaling a GPU replica isn’t like scaling a web server. A stateless pod comes up in a second; a GPU replica has to schedule onto a node, pull a multi-gigabyte image, initialize the runtime, and load model weights into VRAM. Which, as the next section shows, takes tens of seconds. You can’t scale reactively at the speed traffic arrives, so to protect tail latency you keep a buffer of warm capacity ahead of demand.

Noisy-neighbor tax shows up as your service needing more hardware.

In a multi-tenant GPU environment you share physical silicon, and a GPU has two things that are easy to starve: VRAM and memory bandwidth. Token generation is memory-bandwidth-bound, you stream the whole model out of VRAM for every token, so when a co-tenant saturates that bandwidth or expands its KV-cache footprint, your throughput drops and your tail latency spikes, and you didn’t touch your own code.

This is measured, not hand-waved. The iGniter study co-located inference workloads on a shared GPU using NVIDIA’s MPS spatial sharing and watched average inference latency climb from under 1% to roughly 35% as co-tenants grew from two to five. The damage is concentrated exactly where it hurts: medians and throughput barely move, while the p99 tail and jitter blow up, worst for the small, latency-critical service. A dedicated H100, by contrast, gives you the full 3.35 TB/s of memory bandwidth with no interference and a predictable p99.

Production SLAs live and die on p99, not the median. When you can’t see or control the neighbor degrading your tail, you do the only thing you can: over-provision. Add a GPU of headroom, run at lower utilization on purpose, just to absorb variance you don’t own. On our workload, one extra H100 of defensive headroom is about $2,840 a month paid to protect against a workload that isn’t yours.

Scale-to-zero is the obvious answer to the idle tax: drop to zero GPUs when traffic stops, pay nothing. The catch is the next request, the cold start.

When a request hits a cold endpoint, the platform creates the container, initializes the ML runtime and CUDA context, fetches weights from object storage, loads them into VRAM, and warms up CUDA graphs and the KV cache. For a large model that’s 30 to 90 seconds, dominated by fetching weights. One arXiv breakdown clocks a 130GB Llama-2-70B checkpoint at ~26 seconds just to pull from storage over a 5GB/s link, then ~84 seconds to load onto 8 GPUs, while the first token itself generates in about 100 milliseconds. It gets worse under load: when several cold containers land on the same node, they fight for network bandwidth, fetching weights, and starting to stretch further.

For an async batch job, a minute of warm-up is irrelevant. For a real-time API a human is waiting on, a 45-second first response is a failed request. The user is gone.

To be precise: cold starts aren’t what they were two years ago. Memory snapshotting captures and restores the whole VRAM state, weight caching and prefetching overlap the fetch with container creation. RunPod and Modal have pushed best-case starts toward sub-second Modal cites a vLLM model dropping from ~118s to ~12s, with best cases in the low single digits. But the cost hides inside the fix: the reliable way to guarantee no cold start is to keep a minimum pool of workers warm. And a warm worker is an idle GPU you’re paying for. Scale-to-zero traded the idle tax for the cold-start tax, and the standard mitigation trades it right back. Keeping a one-GPU warm baseline on our workload is another ~$2,840 a month.

The bill: ~$18,050 shared vs ~$12,800 dedicated

Same workload: four H100s, ~7 TB of egress, a real-time SLA.

On a hyperscaler, shared, with the defensive posture you actually need to stay up:

• ~$11,300 compute (at $3.93/GPU-hr)
• ~$1,070 egress, NAT, and cross-AZ
• ~$2,840 for one GPU of noisy-neighbor headroom
• ~$2,840 for a one-GPU cold-start warm baseline

That’s about $18,050 a month, and only ~30% of the base compute is doing useful work. Strip the two insurance GPUs and you’re still at ~$12,390. Now the part that we tend to miss. Single-tenant dedicated capacity for the same four H100s isn’t cheaper per GPU. On DigitalOcean’s Dedicated Inference it’s $4.41 a GPU-hour against the hyperscaler’s $3.93- so four is about $12,700 a month in compute, roughly $1,400 more on the fare alone.

And the total still comes out around $5,000 lower, because dedicated, single-tenant capacity deletes the surcharges the cheap fare quietly required:

• Egress collapses. Your frontend isn’t co-located, so responses still leave DigitalOcean but at DigitalOcean’s flat $0.01/GB outbound rate, with no NAT-processing tax and no cross-AZ tax. That same ~7.3 TB is roughly $73, not $1,070.
• No headroom GPU, because there’s no neighbor to defend against.
• No warm-pool GPU, because the capacity is already provisioned and predictable.

When teams finally look at this, the first instinct is one of two reflexes, and both aren’t helpful:

Reflex one: “Go serverless, scale to zero, stop paying for idle”. You just moved the cost into cold-start latency, and on a real-time endpoint. Add a warm pool to fix it and you’re paying for idle GPUs again under a different name.

Reflex two: “Buy bigger reserved commitments to get the rate down”. Reserved pricing lowers the fare. It does nothing about egress, nothing about the neighbor, and it makes the idle tax worse: you’ve now committed, multi-year, to capacity you’ve measured running at 5-30%.

Notice the pattern- Three of the four surcharges aren’t optimization problems you tune your way out of, they’re architecture problems. You don’t FinOps your way out of NAT-Gateway egress, you keep traffic on a private path. You don’t out-clever a noisy neighbor, you stop sharing silicon. You don’t beat a cold start on an endpoint that keeps scaling to zero, you run on capacity that’s already there.

The real question is the same one that governs most infrastructure decisions: build it yourself, or use a layer that already solved it.

Building it yourself is real work. You’re standing up dedicated nodes, configuring a serving stack like vLLM, owning the autoscaler and the warm-pool policy, wiring private networking so egress doesn’t route through a NAT, and re-tuning all of it every time your traffic distribution shifts.

The managed option is DigitalOcean’s Dedicated Inference. It maps onto the same list:

• It’s managed LLM hosting on single-tenant, dedicated GPUs, which takes the noisy-neighbor tax off the table because there are no neighbors.
• It runs an opinionated production stack underneath (vLLM with a prefix-cache-aware router) behind an OpenAI-compatible API, so warm-pool and routing complexity is handled rather than billed back as idle insurance.
• It exposes private VPC endpoints, so traffic between your app and your model stays off the metered internet path.
• It’s reserved per-GPU-hour pricing with a bring-your-own-model path, the predictable side of the idle-versus-cold-start trade, where you keep your own weights.

Two things worth flagging. First, this is dedicated, single-tenant silicon, surcharge three is eliminated by definition: no co-tenant saturating your memory bandwidth, no KV-cache contention spiking p99, no buying a whole extra GPU to absorb someone else’s burst. Second, the VRAM: 192GB on a single AMD MI300X card is enough to hold a model that would otherwise need three or four 80GB H100s sharded together, so you get the full memory bandwidth of one chip with no inter-GPU split, its own quiet win on tail latency. Plans run from a single GPU up to 8-GPU AMD or NVIDIA configs.

Under the hood it’s a Kubernetes-native stack DigitalOcean operates for you: the vLLM serving engine, ingress networking, model storage, autoscaling, and prefix-aware routing. The router tracks KV-cache affinity, so a request reusing a prompt prefix is sent to the replica that already holds those tensors instead of recomputing from scratch. That stack is exactly what, if you built it yourself, became your warm-pool problem, your cold-start problem, and your on-call pager. Here it’s operator-owned. You can still set the node count, including scaling replicas to zero when you want to drop idle time.

Once it’s active, you call it like any OpenAI-compatible endpoint: a POST to the chat-completions URL, a Bearer token, and a model string of the form dedicated:<your-deployment-name>:<model-slug>. If your app sits in the same VPC and you disable the public endpoint, that call never leaves the private network.

There’s a lever below even that. If your model fits, a single 192GB AMD MI300X at $2.59 a GPU-hour (about $1,865 a month) holds a model that would otherwise need three or four sharded 80GB H100s, giving you one card’s full memory bandwidth with no inter-GPU split. That’s not the same workload as four H100s, so benchmark your throughput before assuming it’s a drop-in, but for a lot of 70B-class serving, it’s the cheapest path on this list.

Dedicated only wins if you keep it busy. If your traffic is genuinely spiky with long dead zones and no real-time SLA, serverless with per-second billing can be the cheaper answer.

Measure before you migrate. Pull your actual data-transfer-out and NAT line items. Measure real GPU utilization: used cycles, not provisioned capacity. Check whether you’re single or multi-tenant today. And time a cold start on your own endpoint with a stopwatch. You can’t price the four surcharges on your specific workload until you’ve logged them.

Egress and GPU rates move fast. Commodity H100s are still getting cheaper while frontier-tier H200 reserved pricing went up. A number that’s true this quarter may not be true next quarter. Re-run the math at renewal.

The fare is the one number everyone compares, and it’s the one that hides the other four. Egress and its NAT and cross-AZ riders, the idle tax, the noisy-neighbor tax, and the cold-start tax routinely turn a sub-$12K compute fare into an $18K bill and three of the four are architecture problems, not optimization problems. Keep traffic on a private path, stop sharing silicon, and run on capacity that’s already warm, and the surcharges don’t get optimized down. They stop existing.

Want to see the numbers on your own workload? Start with our LLM inference benchmarking methodology, then pull your transfer-out and utilization line items and run the four-surcharge math against your current setup or spin up a Dedicated Inference deployment and compare directly.