Serverless vs. Dedicated vs. Self-Hosted LLM Inference: When Self-Hosting Actually Gets Cheaper

• Serverless Inference is cheaper than self-hosted GPU Droplet until the GPU is 22-48% busy.
• Dedicated Inference is 30% more expensive than a self-hosted GPU Droplet, moving its break-even to ~29% duty cycle.
• Default to Serverless Inference, and move to a GPU Droplet once it will actually stay busy.
• Don’t guess your inference bill, measure it carefully.
• Buying or reserving a GPU “to save money” and running it near-idle is 2-4x more expensive than serverless, plus the ops burden on top.

Every number here comes from our own runs on DigitalOcean, using Qwen3-32B across two realistic workload shapes: a chat shape (1,000 tokens in / 500 out) and a RAG shape (2,000 in / 200 out). Latency was measured over the public internet from a single client; the max-throughput numbers were measured on the box itself so the network couldn’t cap them.

Two honest caveats up front. First: the self-hosted arm ran full bf16 precision on a 192 GB MI300X, while the managed arms serve their own build, most likely FP8. Precision is therefore not matched, and that’s a real limitation of this comparison, not a deliberate design choice. It cuts against self-hosting, not for it, since a matched FP8 self-host would likely be cheaper still. Second: the numbers at the highest concurrency tiers (peak throughput, the ~40s tail latency) come from single runs, not averages. Treat those tail figures as approximate rather than a solid measurement you can bank on.

All raw data and scripts are published so you can rerun this yourself, pricing referenced throughout is as of the time this experiment was run. Check DigitalOcean’s current rates before making a purchasing decision, since they can change.

Serverless (DigitalOcean Serverless Inference). You call an API and pay per token. The provider owns the GPUs, the scaling, all of it. You manage nothing.

Dedicated (DigitalOcean Dedicated Inference). You rent a whole GPU’s worth of capacity, kept warm for you, as a managed endpoint. You get isolation; the provider still runs the box.

Self-hosted (DigitalOcean GPU Droplets). You rent a raw GPU and run the model server yourself (or, at the extreme, you buy the hardware). Most control, all of the responsibility.

Judge all three on four things: cost, latency, how they scale, and how much operational work they put on you.

We fixed the model (Qwen3-32B) and two realistic workloads: a chat shape with lots of generated text, and a retrieval shape with a large document stuffed in for a short grounded answer. Same prompts, same streaming client, same metrics. The only thing that changed was the deployment model.

• Arm A: DigitalOcean Serverless Inference
• Arm B: DigitalOcean Dedicated Inference (managed dedicated endpoint)
• Arm C: self-hosted on a DigitalOcean GPU Droplet, renting a raw GPU and standing up vLLM ourselves, both from a ready image and fully by hand

The first difference shows up before a single user arrives: how long until you can serve, and when the meter starts running.

Serverless was live in minutes. Grab an API key, paste it into your app, you’re live, and you pay only when tokens flow.

DigitalOcean Dedicated Inference took about 25 minutes to provision before it answered a single request, and you’re billed for that entire warm-up. That’s normal for a large model; managed endpoints across the industry typically take 5 to 30 minutes because the platform has to pull and load tens of gigabytes of weights.

Renting a raw GPU Droplet ourselves was actually faster: roughly 4 minutes to a serving endpoint. But then you own everything: the firewall, the drivers, the container, restarts, teardown. We learned that the hard way twice. A port was silently firewalled off from our self-hosted endpoint (the 1-click vLLM image ships ufw with only 22/80/443 open by default), and a “deleted” managed endpoint left an orphan GPU that kept billing until we hunted it down manually.

For a single request, a GPU all to yourself is clearly faster. Measured the same way over the public internet, one stream on our own self-hosted MI300X GPU Droplet ran about 50 tokens/second, with first token in roughly 0.6 seconds. Serverless Inference, sharing a pool, was slower per stream, around 22 tokens/second, with first token closer to 1.5 seconds. A GPU you control is about 2.5x snappier to first token.

But that’s only while it’s lightly loaded. Push that same GPU harder and throughput climbs while latency falls apart.

At 32 parallel requests it still felt fine, around 15 seconds end to end. By 100-plus concurrent requests, end-to-end crossed 30, then 40 seconds, and the slow tail got brutal. Strip the network out entirely and measure on the box itself, and first-token latency drops to tens of milliseconds, but that’s the machine’s raw ceiling, not what users actually feel.

One MI300X peaks at roughly 2,389 tokens/second at 192 concurrent requests, but end-to-end latency at that point blows past 40 seconds. Push to 256 concurrent and it over-saturates: throughput actually drops.

Hold that thought, because it changes where self-hosting actually pays off.

Here’s the number everyone actually wants, and it’s easy to get wrong, so we’re computing it per answer, not per raw token. A typical chat answer here is 1,000 tokens in, 500 out. On Serverless Inference, that costs about $0.0005, and it never changes with how busy the GPU is.

A rented GPU is the opposite. You pay for it by the hour, whether it’s working or not, so its price per answer is entirely about utilization. Two things get mixed up here constantly:

• Duty cycle: what share of the day the GPU is actually working. This is what sets your cost.
• Concurrency: how many requests you run at once. This is the knob that trades latency for throughput.

Run the GPU flat out at high concurrency, and one MI300X handles almost 5 answers a second at about $0.0001 each, roughly a fifth of serverless. But flat out means 40-second responses.

So there are two break-even points, not one:

• If you saturate the GPU (accepting ~40s responses), you break even at only about 22% duty cycle.
• If you keep it snappy (~15s end-to-end, the c32 operating point), you break even closer to 48% duty cycle.

Counterintuitively, saturating breaks even at a lower utilization than staying snappy, because higher concurrency squeezes more tokens out of every working hour.

Either way, the message is the same: once a single GPU is steadily busy (somewhere between a quarter and half of the day), it beats serverless on price. Well above that, it’s several times cheaper. And keeping it busy is not the same as maxing it out; you can sit at the snappy operating point and still win on price, as long as the GPU isn’t idle.

The trap runs the other direction. At 10% duty cycle (where a lot of early-stage GPUs actually sit), self-hosting is more than 2x the cost of serverless. At 5%, it’s over 4x. And that idle GPU still needs someone to babysit it.

These are two different operating points, not one curve. Pick the row for the latency you’re actually willing to accept, don’t mix them.

Dedicated Inference and a self-hosted GPU Droplet run the exact same silicon, so they perform identically. Dedicated Inference just cost about 30% more per hour, which moves its break-even from roughly 22% up to about 29% duty cycle. Still a low bar, but that premium is purely for not having to manage the box yourself.

As for buying hardware outright: modeled honestly (capital spread over three years, plus power), an owned accelerator is cheaper still, a small fraction of serverless once you keep it busy. But two big asterisks. First, you can’t actually buy a single MI300X; they ship in eight-GPU servers, so this is an illustrative model, not a real price you can put on a card. Second, it only pays off if you sustain that load for years, have the capital up front, and (the part nobody costs) have someone on call for it. For almost every startup, that someone is too expensive.

One pricing note underlying all of this: the self-host figures above use on-demand rates. A reserved GPU Droplet on a longer commitment runs cheaper still (roughly $1.49 per GPU-hour on a 12-month MI300X term, versus $1.99 on-demand), which pushes every break-even lower again. It only helps if you can commit to the term, so it rewards exactly the steady base-load case where self-hosting already wins.

Always confirm the rates with DigitalOcean’s current rates before making a purchasing decision, since they can change.

Default to Serverless Inference. Pre-product-market-fit, with spiky and unpredictable traffic, it’s both the cheapest option and the least work. You pay essentially nothing when idle, and it bursts for you automatically.

Move to Dedicated Inference or a self-hosted GPU Droplet when you need something serverless can’t sell you: tight, predictable latency; isolation and guaranteed capacity; or data that has to stay in your own boundary. Notice this is a control decision, not a savings decision.

Chase the cost crossover once you have a steady, predictable base-load that keeps one GPU at least a quarter to half busy around the clock. That’s when it genuinely undercuts serverless. Self-host before you buy: renting is the same speed for roughly 30% less and no capital outlay.

Put a thin gateway in front of everything, so when you move a hot path to be dedicated later, it’s a config change, not a rewrite.

Buying or reserving a GPU “to save money” and running it near-idle. At the 5-10% utilization most early GPUs actually see, that’s 2-4x more expensive than serverless, plus the ops burden on top.

Forgetting the warm-up and idle bills. Dedicated Inference charged us for 25 minutes before it served anything. An orphaned GPU kept billing after we thought we’d shut it down: the endpoint disappeared from the API, but its backing MI300X droplet stayed active and kept accruing charges until we deleted it manually.

Measuring throughput wrong. Our first test, run over the internet from a laptop, under-reported the GPU’s true throughput by roughly 3x. The client and WAN were the bottleneck, not the GPU. We had to re-run the load generator on the droplet itself (localhost) to find the true ceiling.

Comparing costs on the wrong basis. This one is easy to fall into. Serverless hands you a price sheet with input and output tokens billed at separate rates, while a rented GPU just hands you an hourly bill and no token accounting at all, so you have to build the per-token number yourself. Early on we built it from output tokens only, then compared that against a serverless price that billed both input and output. The denominators didn’t match, which loaded the GPU’s full hourly cost onto a fraction of the tokens it actually processed and made self-hosting look more expensive than it is. Fixed by pricing one whole answer end to end (all input plus output tokens) under each option, so both sides count the same thing.

The fix, every time, was the same: line the numbers up honestly. If you benchmark, saturate from close to the box, and compare cost per answer, not cost per raw output token.

Optimize your model choice and your utilization before you touch the deployment model. Default to DigitalOcean Serverless Inference, and move to a GPU Droplet once it will actually stay busy. And don’t guess your inference bill, measure it carefully.

If you want the broader conceptual framework for choosing an inference mode before diving into numbers, DigitalOcean’s inference mode comparison guide covers serverless, dedicated, batch, and inference router at a high level. And for a deeper look at the throughput/latency/cost tension that shapes all of this, The LLM Inference Trilemma is worth a read.