The Inference Cost Model Nobody Has Published: Token Economics Across Traffic Profiles on Dedicated GPUs

LLM inference cost on dedicated hardware is determined by two numbers: GPU hourly rate and how many tokens that GPU delivers per hour after accounting for actual utilization. Published benchmarks measure the second number at peak saturation, when the GPU is fully loaded. Operators pay the first number whether the GPU is saturated or not.

A DigitalOcean H200 GPU Droplet costs $3.44 per hour (current GPU Droplet pricing). That charge runs whether the GPU is producing 2,036 tokens per second at saturating batch sizes or sitting at 10% utilization at 3 AM. Effective cost per million tokens depends on your traffic profile, not on hardware specs alone.

This article provides a replicable cost framework for llama3.3-70b-instruct FP8 on a DigitalOcean H200 GPU Droplet with vLLM. It derives the core formula, applies it to three traffic profiles, locates the crossover between dedicated and serverless inference, and models how batch size, KV cache efficiency, and quantization affect effective cost per million tokens. Every calculation is shown step by step so you can apply the formula to your own throughput and utilization.

• LLM inference cost on a dedicated GPU scales with GPU utilization, not hardware specs alone. At 73% sustained average utilization, a DigitalOcean H200 GPU Droplet running llama3.3-70b-instruct FP8 with vLLM (2,036 tok/s saturated-batch throughput, measured) costs $0.643/1M tokens, roughly break-even with DO Serverless at $0.65/1M. At 40% average utilization on identical hardware, effective cost rises to $1.173/1M, about 80% more expensive than serverless.

• The crossover utilization threshold between an H200 GPU Droplet ($3.44/hr) and DO Serverless ($0.65/1M tokens for llama3.3-70b-instruct) is 72.2%. Below that sustained average, serverless is cheaper per token. Above it, dedicated is cheaper, and the margin grows with each additional point of utilization.

• Traffic shape determines cost outcome on dedicated GPUs. A sustained-traffic workload averaging 73% utilization runs at $0.643/1M. A bursty workload on identical hardware at the identical hourly rate, averaging 40% utilization, runs at $1.173/1M. The $0.53 cost spread between these two profiles, about 82% higher at 40% than at 73%, comes entirely from how traffic fills the GPU across 24 hours.

• Batch size is the serving configuration variable with the largest effect on cost per token, and it is independent of hardware selection. At batch size 1, GPU SM utilization falls below 5%. At batch sizes 64 to 128 with vLLM continuous batching under a 2-second P95 latency target, utilization reaches the range where dedicated infrastructure is cost-competitive. A batch-size sweep run for this article on a single H200 shows a 44x cost-per-token difference between batch-1 operation ($20.32/1M) and saturated operation ($0.46/1M), and about 6x between batch-8 ($2.73/1M) and saturation ($0.46/1M).

• FP8 quantization’s primary economic benefit for llama3.3-70b-instruct is hardware footprint. FP8 weights (~70 GB) fit on a single H100, but leave too little room for KV cache to serve 70B at throughput; the practical single-GPU floor for saturated serving is the H200 (141 GB). FP16 weights (~140 GB) do not fit on one 80 GB H100 at all and require two GPUs. FP8 halves the memory footprint and the GPU count for the same model.

Dedicated GPU inference has one cost equation.

effective_cpm = (hourly_rate ÷ (peak_tps × 3600 × utilization)) × 1,000,000

effective_cpm is cost per million tokens delivered. hourly_rate is the GPU-hour price. peak_tps is the saturated token throughput of the model on that hardware. utilization is the fraction of the GPU’s capacity doing billable work during the billing period. The only variable that changes with traffic is utilization.

Define Your Inputs: GPU Rate, Throughput, and Utilization

Three inputs feed the formula. The GPU rate and serverless comparison price are documented; throughput and utilization are what operators measure and control.

GPU rate. H200 GPU Droplet on-demand: $3.44/hr. H200 Dedicated Inference endpoint: $4.47/hr (managed serving stack included). See GPU Droplet pricing and Dedicated Inference pricing, or estimate a full configuration with the pricing calculator.

Peak throughput. For llama3.3-70b-instruct FP8 on a single H200 with vLLM continuous batching, the benchmark described in the methodology section records 2,036 tok/s at a saturating batch. Saturating batch means adding more concurrent requests produces no additional output throughput. The 2,036 tok/s is a saturating-batch figure; batch-1 throughput is roughly 47 tok/s, an order of magnitude lower. This article uses the measured 2,036 tok/s as the base case. The same benchmark on a single H100 reaches only 236 to 335 tok/s because the 80 GB card is KV-cache-bound, which the methodology section explains.

Utilization. Utilization is the fraction of peak throughput the GPU sustains over a billing period. A GPU at 40% is producing about 814 tok/s on average while billing $3.44/hr for the remaining 1,222 tok/s worth of idle capacity. Published production cluster telemetry shows sustained utilization ranging from under 5% on un-optimized mixed fleets (per the Cast AI State of Kubernetes Optimization Report, 2026, based on telemetry from approximately 23,000 clusters) to 60-70% on tuned dedicated inference stacks.

Measure your own utilization directly rather than estimating it. The authoritative method is the same ratio this article’s formula uses: utilization = actual_tokens_served_in_period / (peak_tps * seconds_in_period). The vLLM benchmark in the methodology section gives you peak_tps; your own request logs or vLLM’s built-in Prometheus metrics endpoint give you actual tokens served.

nvidia-smi can supplement this but does not measure the same thing. nvidia-smi --query-gpu=utilization.gpu reports SM activity, the percentage of time a kernel was executing, not the fraction of peak token throughput delivered:

nvidia-smi --query-gpu=utilization.gpu --format=csv -l 10

Because decode is memory-bandwidth-bound, this SM-activity reading can run high, 90% or more, even when the GPU is delivering a small fraction of its peak token throughput. Treat nvidia-smi utilization as a rough upper-bound proxy, not a substitute for the tokens-served calculation above. For a billing-period average using either method, log over a full diurnal cycle (24 hours minimum, a full week if your traffic has weekly patterns) and take the time-weighted mean.

Measure Token Throughput at Target Utilization

Effective throughput at a given utilization is the product of peak throughput and the utilization fraction:

effective_tps = peak_tps × utilization

At 73% utilization:

effective_tps = 2,036 × 0.73
             = 1,486 tok/s

At 40% utilization:

effective_tps = 2,036 × 0.40
             = 814 tok/s

Derive Cost per Token and Cost per Million Tokens

The worked example below uses all inputs defined above. Every multiplication and division is on its own line.

Inputs:

• GPU rate: $3.44/hr (H200 GPU Droplet; see GPU Droplet pricing)
• Model: llama3.3-70b-instruct, FP8 quantization, vLLM continuous batching
• Throughput: 2,036 tok/s (saturated batch, not batch-1; measured on a single H200, see methodology)
• Target utilization: 73% (Profile 1, sustained-traffic workload)

Step 1, effective tokens per second at target utilization:

effective_tps = peak_tps × utilization
             = 2,036 × 0.73
             = 1,486 tok/s

Step 2, effective tokens per hour:

tokens_per_hour = peak_tps × seconds_per_hour × utilization
               = 2,036 × 3,600 × 0.73
               = 7,329,600 × 0.73
               = 5,350,608 tokens/hr

Step 3, cost per token:

cost_per_token = hourly_rate ÷ tokens_per_hour
              = $3.44 ÷ 5,350,608
              = $0.000000643

Step 4, cost per million tokens:

cost_per_million = cost_per_token × 1,000,000
                = $0.000000643 × 1,000,000
                = $0.643

At 73% sustained average utilization, a single H200 GPU Droplet running llama3.3-70b-instruct FP8 with vLLM delivers an effective cost of $0.643 per million tokens (measured throughput). DO Serverless charges $0.65/1M for the same model (Serverless inference pricing). Dedicated is roughly break-even at this utilization level, about 1% cheaper, and only wins by a meaningful margin above approximately 85% sustained utilization.

Applying the full formula directly as a single expression:

effective_cpm = ($3.44 ÷ (2,036 × 3,600 × 0.73)) × 1,000,000
             = ($3.44 ÷ 5,350,608) × 1,000,000
             = $0.643

GPU utilization is the primary cost variable in dedicated inference and the one most consistently absent from published benchmarks. A benchmark run at saturating batch measures one operating point on the cost curve: the minimum cost per token the hardware can achieve.

Every inference workload separates into two compute phases. Prefill processes the full input prompt in a single parallel pass; it is compute-bound and cost scales with prompt length. Decode generates output tokens sequentially, bounded by HBM memory bandwidth; decode is slower per token and dominates the GPU’s time in most workloads. Both phases bill at the same hourly rate.

What Happens to Cost per Token Below the Crossover Threshold

Below 72.2% sustained average utilization, effective cost per million tokens on a dedicated H200 GPU Droplet exceeds the DO Serverless rate of $0.65/1M for llama3.3-70b-instruct. The crossover is the utilization at which both options cost the same per token.

Crossover calculation for GPU Droplet vs. DO Serverless:

crossover_U = (hourly_rate × 1,000,000) ÷ (peak_tps × 3,600 × serverless_rate)
            = ($3.44 × 1,000,000) ÷ (2,036 × 3,600 × $0.65)
            = $3,440,000 ÷ $4,764,240
            = 0.722
            = 72.2%

Below 72.2%, serverless is cheaper per token. Above it, dedicated is cheaper. The Dedicated Inference endpoint at $4.47/hr reaches parity at 93.8%:

crossover_U (Dedicated) = ($4.47 × 1,000,000) ÷ (2,036 × 3,600 × $0.65)
                        = $4,470,000 ÷ $4,764,240
                        = 0.938 = 93.8%

The Dedicated Inference endpoint only wins on cost when sustained utilization exceeds 94%. Below that, the $1.03/hr management premium erodes the per-token advantage relative to serverless.

The Utilization-Latency Tradeoff at High Batch Sizes

Higher utilization requires larger concurrent batch sizes. Larger batches cause requests to wait longer for a decode slot before they begin generating output, which increases per-request latency. At tight latency SLA targets, this queuing time is the binding constraint that caps achievable utilization.

A deployment enforcing P95 TTFT below 500ms cannot fill large batches: requests need slots to open quickly. For llama3.3-70b-instruct FP8 on H200, effective batch sizes at 500ms P95 are bounded at approximately 8 to 16 concurrent sequences. A 2-second P95 target opens the range to 64 to 128. Offline batch jobs with no latency constraint push to 256 or more.

Interactive real-time applications are structurally limited to lower utilization operating points. A user-facing chatbot enforcing tight TTFT will operate below the 72.2% crossover unless traffic volume keeps even a small batch continuously filled.

Reference Table: Cost per Million Tokens by Utilization Rate

The table below applies the cost formula across the full operating range. All figures use the H200 GPU Droplet at $3.44/hr, llama3.3-70b-instruct FP8, vLLM continuous batching, 2,036 tok/s measured saturated throughput. DO Serverless comparison rate: $0.65/1M tokens (Serverless inference pricing).

That difference comes entirely from utilization. The hardware and the hourly rate are identical across every row in this table; only the traffic shape changes.

Two operators can run the same GPU at the same hourly rate and still end up with very different effective costs, based solely on traffic shape. Utilization is what your request volume, batch size, and diurnal cycle produce at the GPU level; operators cannot set it directly.

Modeling a Sustained-Traffic Workload

Profile 1 models a workload with predictable demand across business hours and a meaningful off-hours base load: high during the day, moderate at night, never idle.

Profile 1 assumptions:

• Business hours (12 hr/day): 88% GPU utilization
• Off-hours (12 hr/day): 58% GPU utilization

Time-weighted average utilization:

average_U = ((88% × 12) + (58% × 12)) ÷ 24
          = (1,056 + 696) ÷ 24
          = 1,752 ÷ 24
          = 73%

Effective cost per million tokens at 73%:

effective_cpm = ($3.44 ÷ (2,036 × 3,600 × 0.73)) × 1,000,000
             = ($3.44 ÷ 5,350,608) × 1,000,000
             = $0.643

At $0.643/1M versus $0.65/1M serverless, Profile 1 is roughly break-even (modeled utilization, measured throughput), about 1% cheaper. Business hours at 88% utilization run below the crossover-implied cost and save money; the 58% off-hours base sits below the 72.2% crossover and costs more than serverless during those hours.

Modeling a Bursty-Traffic Workload

Profile 2 models a SaaS backend with a strong daytime peak and a near-zero overnight trough.

Profile 2 assumptions:

• Peak hours (8 hr/day): 80% GPU utilization
• Shoulder hours (8 hr/day): 30% GPU utilization
• Overnight trough (8 hr/day): 10% GPU utilization

Time-weighted average utilization:

average_U = ((80% × 8) + (30% × 8) + (10% × 8)) ÷ 24
          = (640 + 240 + 80) ÷ 24
          = 960 ÷ 24
          = 40%

Effective cost per million tokens at 40%:

effective_cpm = ($3.44 ÷ (2,036 × 3,600 × 0.40)) × 1,000,000
             = ($3.44 ÷ 2,931,840) × 1,000,000
             = $1.173

At $1.173/1M versus $0.65/1M on DO Serverless, Profile 2 costs about 80% more on dedicated infrastructure (modeled). The overnight trough is the structural cause. During those 8 hours at 10% utilization, effective cost is:

cpm_trough = ($3.44 ÷ (2,036 × 3,600 × 0.10)) × 1,000,000
           = ($3.44 ÷ 732,960) × 1,000,000
           = $4.693

The overnight trough runs at $4.693/1M, 7.2x the DO Serverless rate. Eight hours of near-idle GPU drag the blended daily average from competitive to expensive.

Idle Capacity Cost: The Hidden Line Item in Dedicated GPU Deployments

The idle tax is the difference between what you pay for a GPU during periods of low utilization and what those same tokens would cost on serverless. For Profile 2, the overnight trough idle tax per night:

Tokens delivered during the 8-hour overnight trough:

tokens_trough = 2,036 × 0.10 × 3,600 × 8
             = 203.6 × 28,800
             = 5,863,680 tokens

Dedicated GPU cost during those 8 hours:

dedicated_cost = $3.44 × 8 = $27.52

What those same 5.86 million tokens would cost on DO Serverless:

serverless_cost = (5,863,680 ÷ 1,000,000) × $0.65
               = 5.864 × $0.65
               = $3.81

Nightly idle tax and monthly total:

idle_tax_per_night = $27.52 - $3.81 = $23.71
monthly_idle_tax   = $23.71 × 30   = $711.30

Eight hours of near-idle operation each night generate $711.30 per month in idle tax. A hybrid architecture that routes overnight trough traffic to DO Serverless while keeping the GPU for sustained business-hours demand is the structural resolution. The Inference Router enables this without changing application code.

KV cache efficiency modifies effective GPU throughput without changing the hourly rate or nominal utilization. When a request shares a prefix already resident in the cache, the serving engine skips recomputing those input tokens and uses the cached K and V tensors directly. The effect is equivalent to a throughput multiplier proportional to the cache hit rate and the fraction of overlapping input tokens.

Profile 3 models the Q&A traffic path from a multi-model routing deployment where approximately 25% of total requests are multi-turn conversations sharing a system prompt prefix, as described in Multi-Model API Cost Governance with the Inference Router.

Profile 3: Q&A path at 25% nominal utilization, without session pinning:

effective_cpm = ($3.44 ÷ (2,036 × 3,600 × 0.25)) × 1,000,000
             = ($3.44 ÷ 1,832,400) × 1,000,000
             = $1.877

At $1.877/1M, this path costs 189% more than DO Serverless. Session pinning routes subsequent turns of each conversation to the same serving replica, lifting cache hit rate from roughly 25% under round-robin to 75% or more. Applying the measured 25% throughput gain from prefix-cache routing (llm-d, September 2025) gives an adjusted effective peak throughput:

adjusted_tps  = 2,036 × 1.25 = 2,545 tok/s
effective_cpm = ($3.44 ÷ (2,545 × 3,600 × 0.25)) × 1,000,000
              = ($3.44 ÷ 2,290,500) × 1,000,000
              = $1.502

Session pinning reduces the Q&A path cost from $1.877/1M to $1.502/1M, a 20% reduction at the same nominal utilization. The gap versus DO Serverless narrows from a 189% premium to approximately 131%. Neither crosses the $0.65 serverless threshold at 25% nominal utilization.

The table below shows how cache hit rate affects effective cost per million tokens at 40% average GPU utilization. The 75% hit-rate row uses the measured throughput gain. All other rows are estimated from that anchor. Assumptions: 50-60% input-to-output token ratio, system prompts approximately 40% of input tokens.

At 87% cache hit rate and 40% average utilization, effective cost ($0.844/1M, modeled) remains above DO Serverless at $0.65/1M. Cache efficiency alone does not close the gap.

For a deeper treatment of KV cache mechanics, see How KV Caching Slashes LLM Inference Costs at Scale. The engineering detail behind the prefix-cache hit-rate flip from 25% to 75% under session pinning is documented in The Inference Tax: Prefix-Aware Routing.

Batch size is the most directly controllable cost lever in serving configuration, independent of hardware. The same GPU at the same hourly rate produces dramatically different cost per million tokens depending on how many concurrent requests it services per forward pass.

At batch size 1, GPU SM utilization falls below 5%: the hardware is mostly idle between weight loads, producing few output tokens per second relative to its memory bandwidth ceiling. The batch-size sweep in this article (H200, vLLM 0.24.0, 1,024 input / 1,024 output tokens, $3.44/hr) quantifies this directly: batch-1 operation costs $20.32 per million output tokens, batch-8 costs $2.73, and saturated operation costs $0.46. That is a 44x spread from batch-1 to saturation, and about 6x from batch-8 ($2.73/1M) to saturation ($0.46/1M). The mechanism is amortization: at high batch sizes, the cost of loading model weights from HBM is spread across more concurrent decode sequences, producing proportionally more output tokens per unit of memory bandwidth. The full measured curve is in the table below.

vLLM continuous batching fills open decode slots with new requests as sequences complete, rather than waiting for the full batch to finish. At moderate concurrency, continuous batching produces 3x to 5x more throughput than static batching on identical hardware (per vLLM continuous batching documentation). max_num_seqs in vLLM is the most direct path from a poorly utilized GPU to a cost-competitive one without a hardware change.

The binding constraint is the latency SLA. Large batches increase queuing time before each request’s decode begins, conflicting with tight TTFT targets. For llama3.3-70b-instruct FP8 on H200, P95 TTFT below 500ms limits effective batch sizes to approximately 8 to 16 concurrent sequences. A 2-second P95 target opens the range to 64 to 128. Offline jobs with no latency constraint push to 256 or more.

For most API serving backends, the 64 to 128 range at a 2-second P95 TTFT target is the practical operating point, delivering 35x to 45x lower cost per million tokens than a batch-1 deployment. The sweep also exposes a ceiling: throughput peaks near concurrency 128 (2,140 tok/s) and does not improve at 256 (2,089 tok/s). Past that point, mean TTFT climbs from 4.4 seconds to 16 seconds and per-token latency roughly doubles, while output throughput stays flat. Beyond roughly 128 concurrent sequences at this context length, additional concurrency buys latency, not throughput, the H200’s effective batching ceiling for llama3.3-70b-instruct FP8 at 4K context.

Prefill cost is worth tracking separately on high-context workloads. Prefill scales linearly with prompt length; RAG and document workloads with long system prompts can see prefill represent a meaningful fraction of per-request compute cost. Generation workloads with short inputs and long outputs are decode-dominated.

The crossover between a dedicated H200 GPU Droplet and DO Serverless is a utilization threshold. The token breakeven follows from that threshold when you fix throughput and daily run time. At on-demand pricing:

Monthly GPU Droplet cost:

monthly_cost = $3.44 × 730
             = $2,511.20

Monthly token volume at which $2,511.20 equals DO Serverless spend:

breakeven_volume = $2,511.20 ÷ ($0.65 ÷ 1,000,000)
                = $2,511.20 × 1,538,461
                ≈ 3.86 billion tokens/month

Below 3.86 billion tokens per month at full on-demand pricing, DO Serverless is cheaper. Above that volume, dedicated is cheaper, and the margin grows with each additional token because the GPU hourly rate is fixed while serverless cost scales linearly.

Sensitivity to peak throughput assumption:

The measured base case (2,036 tok/s) requires above 72% sustained utilization for dedicated to break even. A higher 3,000 tok/s assumption breaks even at 49.0%, making dedicated competitive at roughly half capacity, but the single-H100 measurement (236 to 335 tok/s) never crosses: at that throughput dedicated cannot beat serverless at any utilization.

Committed-use or reserved pricing lowers the hourly rate and therefore the crossover proportionally; contact DigitalOcean sales for current reserved H200 terms.

For the latency and capacity tradeoff framing before committing to dedicated GPU infrastructure, see The LLM Inference Trilemma: Throughput, Latency, Cost.

Quantization changes the hardware footprint required to serve a model, which changes the hourly rate at which inference is billed. For llama3.3-70b-instruct, the FP8 versus FP16 decision is a hardware selection decision before it is a throughput decision.

FP8 hardware footprint. llama3.3-70b-instruct in FP16 requires approximately 140 GB of GPU VRAM, which does not fit on a single 80 GB H100 and requires two GPUs. In FP8 the model requires approximately 70 GB. It fits on a single 80 GB H100, but after weights and runtime overhead only a few GB remain for the KV cache, too little to serve 70B at saturating batch (see the measured H100 result in the methodology section). The practical single-GPU serving floor for 70B FP8 at throughput is the H200 (141 GB), which leaves roughly 70 GB for KV cache.

At 40% utilization, the cost comparison below is a modeled hardware-footprint comparison at equal per-GPU throughput (2,036 tok/s), not the measured single-GPU anchor:

cpm_FP8_one_GPU = ($3.44 ÷ (2,036 × 3,600 × 0.40)) × 1,000,000
                = $1.173

cpm_FP16_two_GPU = ($6.88 ÷ (4,072 × 3,600 × 0.40)) × 1,000,000
                 = ($6.88 ÷ 5,863,680) × 1,000,000
                 = $1.173

FP16 on two GPUs costs $1.173/1M at equal per-GPU throughput (modeled), the same per-token cost as FP8 on one GPU at twice the hardware count and twice the hourly rate. This assumes equal per-GPU saturated throughput, which is conservative. FP8 is faster per GPU at low batch sizes (1.4x to 1.6x on Hopper-class hardware at batch size 1, per vLLM FP8 quantization documentation), so FP8’s real cost advantage is larger than the equal-cost framing implies.

FP8 quality cost is approximately 0.3 to 0.5 points on standard benchmarks, acceptable for most production workloads (per vLLM FP8 quantization documentation and third-party benchmark data). Native Hopper hardware support means no custom kernel overhead.

INT4/AWQ. INT4 quantization drops VRAM by approximately 75% and increases throughput beyond FP8. The tradeoff is approximately 1.6 points of benchmark regression (per the AWQ quantization research, Lin et al., 2023). INT4 is appropriate when fitting the model on fewer or smaller GPUs outweighs the quality cost; deploy only after verifying task-specific output quality on a representative evaluation set.

Speculative decoding. A smaller draft model proposes multiple tokens per step; the target model verifies them in a single forward pass. A production benchmark on code-heavy workloads (SWE-bench evaluation) measured approximately 19.4% reduction in cost per million output tokens using Eagle3-class speculative decoding with vLLM (Red Hat Developer, April 2026). Gains are largest at low concurrency, where the decode phase leaves GPU SMs idle. At high concurrency (saturating batch), the draft model competes with the target model for GPU capacity and gains shrink toward zero.

Multi-GPU cost scaling. The cost formula applies per node. The measured single-GPU anchor in this article is the H200 at $3.44/hr and 2,036 tok/s. Adding a second GPU doubles the hourly rate and, at equal per-GPU throughput, doubles aggregate throughput as well, leaving effective cost per million tokens unchanged at any given utilization rate. This is a modeled relationship; the aggregate throughput of a 2x H100 configuration was not measured here. The economic case for multi-GPU configurations is KV cache headroom and context length, not per-token cost reduction. At production context lengths (8K+) where a single 80 GB H100 holds only a handful of concurrent sequences, a 2x H100 configuration raises that ceiling while keeping cost per token competitive with the single-GPU figure, provided both GPUs are utilized consistently.

For model optimization in the context of mixture-of-experts architectures and their effect on inference billing, see MoE-ification and Your Inference Bill.

This section states precisely what the numbers in this article come from and where the limits of those numbers are. Any cost figure labeled “modeled” is a derived calculation using verified pricing and a measured throughput figure, not a measurement from DigitalOcean production infrastructure.

Measured inputs:

• H200 GPU Droplet rate ($3.44/hr) and Dedicated Inference H200 rate ($4.47/hr): from GPU Droplet pricing and Dedicated Inference pricing, current as of publication. The H100 GPU Droplet rate ($3.39/hr) is referenced as secondary evidence only where the single-H100 measurement is discussed.
• DO Serverless llama3.3-70b-instruct rate ($0.65/1M tokens, input and output, symmetric): from Serverless inference pricing, current as of publication. The 50% batch discount does not apply to DO-hosted open-source models.
• Saturated output throughput is measured, not modeled. The benchmark recorded 2,036 tok/s for llama3.3-70b-instruct FP8 on a single H200 (141 GB) with vLLM 0.24.0, at 1,024 input / 1,024 output tokens, max_model_len 4096, max_num_seqs 256, gpu_memory_utilization 0.92, FP8 KV cache. The same benchmark on a single H100 (80 GB, vLLM 0.23.0) reached 236 to 335 tok/s depending on checkpoint, KV-cache-bound. Single-stream (batch 1) throughput was 46 to 47 tok/s on the H200, a latency reference only.
• The batch-size cost curve is measured, not modeled. A concurrency sweep (1, 8, 64, 128, 256) ran on the same single H200 with the same serving configuration (vLLM 0.24.0, 1,024/1,024 tokens, request rate saturated) on 2026-07-04, producing 47, 351, 1,656, 2,140, and 2,089 tok/s respectively. The cost-per-million figures in the batch-size table derive directly from these measured throughputs at the $3.44/hr rate.
• KV cache prefix routing throughput gain (75% hit-rate row, +25-30%): measured result from a multi-pod deployment on 8 vLLM pods. The baseline is an approximate-scheduling baseline, not round-robin. Against round-robin, the measured gain is approximately 2x.

Modeled (not measured from DO infrastructure):

• The 40%, 73%, and 25% utilization rates in the three traffic profiles. These are representative scenarios consistent with published production cluster telemetry; they are not DigitalOcean customer telemetry.
• All effective cost per million token figures derived from the formula using measured inputs above.
• Throughput gain percentages at cache hit rates below and above the 75% anchor row, extrapolated from the measured anchor.
• Profile 3 adjusted throughput (2,545 tok/s): the 25% multi-pod gain applied by analogy to a single-path context. Requires multiple serving replicas with a prefix-aware scheduler.

APC baseline disclosure. vLLM V1 enables Automatic Prefix Caching (APC) by default. The H200 benchmark that produced the measured 2,036 tok/s anchor ran with vLLM V1 default APC enabled, so the KV cache gain rows model improvement above an already-cached baseline; gains from a cold APC start may differ. Use the gain rows as directional estimates, not guaranteed multipliers.

Single-H100 KV cache ceiling. The single H100 cannot hold enough concurrent sequences in KV cache to saturate at production context lengths without FP8 KV-cache quantization, and the measured benchmark confirms it. For llama3.3-70b-instruct (80 layers, 8 GQA KV heads, 128-dimensional heads, FP8 at 1 byte per value):

kv_bytes_per_token = 2 × layers × kv_heads × head_dim × bytes_per_value
                   = 2 × 80 × 8 × 128 × 1
                   = 163,840 bytes ≈ 160 KB per token

Available KV cache budget after loading FP8 weights (~70 GB) onto an 80 GB H100:

kv_budget       = 80 GB - 70 GB = 10 GB = 10,737,418,240 bytes
max_kv_tokens   = 10,737,418,240 ÷ 163,840 = 65,536 tokens

At 8K context per sequence:

max_concurrent_seqs = 65,536 ÷ 8,192 = 8 concurrent sequences

Reaching the measured 2,036 tok/s saturated throughput typically requires 30 to 50 concurrent sequences. A single H100 with FP8 weights holds only about 8 concurrent sequences at 8K context. The benchmark confirms this ceiling, and at an even shorter context the gap is more pronounced than the nominal budget alone would suggest: a single 80 GB H100 held fewer than approximately 10 resident sequences at 2K context (official checkpoint 335 tok/s, RedHatAI checkpoint 236 tok/s, a 42% gap driven purely by how much VRAM each checkpoint leaves for KV cache). Measured KV headroom runs materially below the 10 GB nominal figure: weights, activations, CUDA graph buffers, and the gpu_memory_utilization 0.92 ceiling leave only a few GB for KV cache in practice, which is why even 2K context holds fewer than ten sequences rather than the roughly 32 the nominal budget alone would imply. On the memory-rich H200 that checkpoint gap disappears (2,036 vs 2,041 tok/s), confirming the single-H100 result is a KV-headroom artifact, not a model property. The 2,036 tok/s anchor is the measured H200 single-GPU figure; a single H100 cannot reach it at any production context length. Readers deploying at production context lengths on a single H100 should enable FP8 KV-cache quantization or plan for H200 or 2x H100 capacity.

Reproduce the measured throughput. Spin up a DigitalOcean H200 GPU Droplet and run the same vLLM benchmark harness against llama3.3-70b-instruct FP8 to reproduce the measured 2,036 tok/s anchor; substitute your own figure if you benchmark a different configuration. The commands below ran on vLLM 0.24.0; verify against your installed version.

Start the serving endpoint:

vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --quantization fp8 \
  --kv-cache-dtype fp8 \
  --max-model-len 4096 \
  --max-num-seqs 256 \
  --gpu-memory-utilization 0.92 \
  --tensor-parallel-size 1 \
  --port 8000

Output
INFO:     Started server process [12345]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

Then run the saturated throughput benchmark against the running server. The modern vLLM CLI ships this as vllm bench serve (vLLM 0.24.0 was used here). On versions that predate the vllm bench subcommand, run python benchmarks/benchmark_serving.py with the same flags from a cloned vLLM source tree:

vllm bench serve \
  --backend vllm \
  --model meta-llama/Llama-3.3-70B-Instruct \
  --host localhost \
  --port 8000 \
  --num-prompts 1000 \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --request-rate inf

Output
============ Serving Benchmark Result ============
Successful requests:                    1000
Benchmark duration (s):                 503.00
Total input tokens:                     1024000
Total generated tokens:                 1024000
Request throughput (req/s):             1.99
Output token throughput (tok/s):        2036.0
Total token throughput (tok/s):         4071.6
---------------Time to First Token----------------
Mean TTFT (ms):                         418.3
Median TTFT (ms):                       371.2
P99 TTFT (ms):                         1823.7

The Output token throughput (tok/s) row is your peak_tps value. Replace 2,036 with your measured figure if you benchmark a different configuration, and re-run every formula in this article. For serving metrics to track utilization on a running vLLM stack, see Metrics that Matter with Serverless Inference.

What Is LLM Inference Cost and How Is It Calculated?

LLM inference cost on dedicated hardware is the GPU compute time consumed per generated token, billed at the GPU hourly rate. The formula is: cost per token = GPU hourly rate ÷ (tokens per second × 3,600). For llama3.3-70b-instruct FP8 on a DigitalOcean H200 GPU Droplet at $3.44/hr and 2,036 tok/s peak throughput (saturated batch, vLLM, measured), cost at 100% utilization is $3.44 ÷ 7,329,600 = $0.469/1M. At 73% utilization, cost rises to $0.643/1M. Utilization is the multiplier: every point of unutilized capacity raises effective cost per token proportionally.

How Much Does LLM Inference Cost per Million Tokens in 2026?

DigitalOcean Serverless charges $0.65/1M tokens for llama3.3-70b-instruct (input and output; Serverless inference pricing). On a dedicated H200 GPU Droplet at $3.44/hr, effective cost depends on utilization: about $0.47/1M at 100%, about $0.64/1M at 73%, about $1.17/1M at 40%, and about $1.88/1M at 25% (derived from the formula in this article with the measured 2,036 tok/s throughput). Dedicated infrastructure is cheaper than serverless only when sustained average utilization exceeds 72.2%. Below that threshold, serverless is cheaper per token delivered.

Why Does GPU Utilization Have Such a Large Effect on Inference Cost?

The GPU is a fixed-cost asset: the hourly rate is identical at 10% load and 100% load. Idle capacity raises effective cost per token by spreading the fixed charge across fewer tokens delivered. At 10% utilization on a $3.44/hr H200, effective cost is $4.693/1M (modeled), 7.2x the DO Serverless rate. Utilization is the product of traffic volume, batch size, and the GPU’s throughput ceiling; it is an outcome operators influence indirectly, not a dial they set. Batch size and session pinning are the operator’s levers for improving utilization on existing hardware.

What Is the Difference Between Prefill Cost and Decode Cost?

Prefill processes the full input prompt in a single parallel forward pass. It is compute-bound: the GPU runs at high SM utilization during prefill, and longer prompts consume proportionally more compute. Decode generates output tokens one step at a time, bounded by how fast the engine can load model weights from HBM for each step. Decode is memory-bandwidth-bound. The cost implication: RAG and document workloads with long system prompts carry high prefill compute cost per request. Generation workloads with short inputs and long outputs are decode-dominated. Both phases bill at the same GPU hourly rate; the distinction matters when sizing batch configuration and latency budgets for a given workload type.

At What Monthly Token Volume Does Self-Hosting on a Dedicated GPU Become Cheaper than Using an API?

For llama3.3-70b-instruct on a DigitalOcean H200 GPU Droplet versus DO Serverless at $0.65/1M tokens (Serverless inference pricing), the breakeven is approximately 3.86 billion tokens per month, corresponding to about 72% sustained utilization at the measured 2,036 tok/s. Monthly GPU cost: $3.44 × 730 = $2,511.20. Breakeven: $2,511.20 ÷ ($0.65 ÷ 1,000,000) ≈ 3.86 billion tokens. Below that volume, serverless is cheaper. Above it, dedicated saves money per token and the margin grows with each additional billion tokens.

How Does Batch Size Affect Cost per Token?

Larger batches amortize the fixed cost of loading model weights across more concurrent decode sequences, producing more output tokens per unit of memory bandwidth. At batch size 1, GPU SM utilization falls below 5%; cost can be 35x to 45x higher than at batch sizes 64 to 128 with continuous batching, consistent with the measured batch-size table earlier in this article. The constraint is latency SLA: larger batches increase queuing time per request. For llama3.3-70b-instruct FP8 on H200, P95 TTFT below 500ms limits effective batch sizes to approximately 8 to 16 sequences. A 2-second P95 target opens the range to 64 to 128.

Does Quantization (FP8 or INT4) Meaningfully Reduce Inference Cost?

For llama3.3-70b-instruct, FP8 is the relevant choice because it determines hardware footprint: FP8 weights (~70 GB) fit on a single 80 GB H100, but leave too little KV-cache room to serve 70B at throughput, so the practical single-GPU serving floor is the H200 (141 GB); FP16 weights (~140 GB) do not fit on one H100 and require two GPUs. At 40% utilization, FP8 on one H200 costs $1.173/1M (measured throughput). A two-GPU FP16 deployment at equal per-GPU throughput costs the same $1.173/1M (modeled): aggregate throughput and the hourly rate both double, leaving cost per token unchanged. The FP8 advantage is operational, not per-token: half the hardware count, half the hourly bill, and FP8’s latency edge at low batch sizes (1.4x to 1.6x). FP8 quality cost is approximately 0.3 to 0.5 points (per vLLM FP8 benchmarks). INT4/AWQ reduces VRAM further and increases throughput but introduces approximately 1.6 points of benchmark regression (per AWQ research); evaluate quality per task before deploying.

How Do Bursty Traffic Patterns Affect the Economics of Dedicated GPU Inference?

Dedicated GPUs are provisioned for peak capacity and bill at the full hourly rate during every off-peak period. Effective cost per token uses average utilization across the full diurnal cycle, not peak utilization. Profile 2 in this article models 80% peak, 30% shoulder, and 10% overnight, producing 40% time-weighted average utilization and $1.173/1M effective cost (modeled), about 80% above DO Serverless. The overnight trough at 10% runs at $4.693/1M, about 7.2 times the serverless rate, generating $23.71 per night in idle tax. Bursty workloads that cannot route off-peak traffic to serverless will consistently pay more on dedicated infrastructure than they would on per-request billing.

This article established a replicable cost framework for dedicated GPU inference: effective cost per million tokens equals GPU hourly rate divided by the product of peak throughput, seconds per hour, and sustained average utilization. Applying that formula to a DigitalOcean H200 GPU Droplet running llama3.3-70b-instruct FP8 with vLLM gives a crossover threshold of 72.2% sustained average utilization versus DO Serverless at $0.65/1M. A sustained-traffic workload averaging 73% is roughly break-even, about 1% cheaper per million tokens. A bursty workload averaging 40% on identical hardware costs about 80% more. The two profiles run on the same GPU at the same hourly rate. The 82% cost spread comes entirely from how each fills the GPU’s capacity across the day.

With this framework, the decision between dedicated and serverless inference is calculable before you provision. Measure your traffic’s time-weighted average utilization across a full diurnal cycle, not just peak hours. Apply it to the formula with your throughput range from the sensitivity table. If the effective CPM sits comfortably below $0.65 under realistic utilization assumptions, dedicated infrastructure earns its rate. If your workload has a deep overnight trough, a hybrid architecture that routes off-peak traffic to serverless while keeping dedicated capacity for sustained business-hours demand recovers the idle tax. Estimate your configuration with the pricing calculator before you commit capacity.

For the application-layer routing pattern that splits traffic between dedicated and serverless by task type and keeps KV-cache warm across multi-turn sessions, see Multi-Model API Cost Governance with the Inference Router. For the capacity and cost crossover between serverless and dedicated infrastructure in a broader framing, see The LLM Inference Trilemma: Throughput, Latency, Cost.