How Does Prompt Caching Work and When Does It Actually Cut LLM Costs?

• Prompt caching is workload-dependent, not provider-dependent. The same provider’s caching can be a clear win or a net loss depending on how your traffic is shaped.
• Three variables decide the outcome: prefix stability, session depth, and interarrival time. This article defines each one precisely and shows how to measure it.
• Anthropic charges a write premium; most OpenAI models on DigitalOcean do not, though OpenAI’s own documentation says the newest ones (GPT-5.6) now do. That difference changes the math, and this article derives the exact break-even point for each case. The Anthropic and most-OpenAI rates below are confirmed on DigitalOcean’s own pricing page; the GPT-5.6 cache-write rate is confirmed on OpenAI’s pricing page but not yet independently verified on DigitalOcean’s, since that model launched on DigitalOcean after the pricing page was last verified.
• A structural gate matters more than any percentage: for OpenAI models, DigitalOcean will not cache a prompt below 1,024 tokens. Open-source models on DigitalOcean have no such floor, and neither does a self-hosted engine, a difference that this article measures directly.
• Open-source models on DigitalOcean’s Serverless Inference support prompt caching automatically. It is in public preview and Prompt caching is automatic (no cache_control or prompt_cache_retention needed) for open-source models on DigitalOcean, per-account isolated, and available on models like DeepSeek V3.2. Use prompt caching with the serverless inference chat completion and responses APIs to cache context and use it in future requests. If part of your request is already cached, you are charged a lower price for those cached tokens and the standard price for the remaining input tokens. This significantly reduces the cost for inference. Open-source models cache context automatically. For a list of models that support prompt caching, see Foundation Models. For more details, use Prompt Caching in the serverless inference documentation.
• Measured on a DigitalOcean GPU Droplet (H200, vLLM 0.24.0, Llama 3.3 70B FP8): a cache hit cut time-to-first-token from 462 ms to 39 ms on a 3,110-token prompt, a 92 percent reduction. A 12-turn agent session with a stable 2,080-token prefix sustained a 98.7 percent token hit rate after the first turn.
• One line of dynamic content destroys everything, and this piece measured exactly how much: placing a timestamp and request ID at the top of the system prompt dropped the measured token hit rate from 98.7 percent to 0.7 percent and returned every turn to full-price prefill latency.
• Self-hosting on a GPU Droplet is the fallback for any open model without managed caching. vLLM enables automatic prefix caching by default, with no write premium and no 1,024-token minimum. The measured run below confirms caching on a prompt as short as 155 tokens on Llama 3.3 70B, which does not have a managed serverless caching rate.

If you’re new to prompt caching, here are some simple definitions for terms used in this article:

When you send a request to a model, the platform has to convert every input token into an internal representation before it can generate a response. This step is called prefill. For a long system prompt, a set of tool definitions, or a large retrieved document, this is the most expensive part of a request in both cost and latency, because the platform reprocesses that same content from scratch every single time, even if it saw the identical text one second ago.

Prompt caching stores the processed representation, the key-value state the attention layers use, so that a later request with the same prefix can skip reprocessing and read the stored state instead. This only applies to input tokens. Output tokens are always generated fresh, since the response depends on what you are asking this time, not what you asked last time.

The mechanism depends on exact matching. Providers cache based on a hash of the prompt prefix. If a single character changes anywhere before the cache boundary, including a timestamp, a request ID, or a reordered tool list, the entire prefix fails to match and the request falls back to full-price processing. This is why prompt structure matters as much as prompt content.

A cache hit skips the expensive prefill step for the matched prefix. A cache miss processes the full prompt and, on providers with explicit caching, writes a new entry for next time.

The cost structure, provider by provider

Every major provider prices caching differently, and the differences are not cosmetic. They change which workloads caching helps.

Anthropic models on DigitalOcean use explicit caching. You mark a boundary in your request with a cache_control field of type: ephemeral and a ttl of either 5m (default) or 1h. The first request that writes that boundary is billed at a premium over the standard input rate, and DigitalOcean’s own pricing page confirms the multipliers: for Claude Haiku 4.5, standard input is $1.00 per million tokens, 5-minute cache creation is $1.25 (1.25x), 1-hour cache creation is $2.00 (2.0x), and cache read is $0.10 (0.10x, a 90 percent discount). Those same 1.25x / 2.0x / 0.10x ratios hold across the Claude family on DigitalOcean. The response usage object reports cache_creation_input_tokens on the writing request and cache_read_input_tokens on subsequent hits.

Confirm this against the current docs for your specific model. Sources: DigitalOcean Prompt Caching how-to and Inference pricing, last verified on July 14, 2026; Anthropic prompt caching documentation.

OpenAI models on DigitalOcean cache prompts of 1,024 tokens or more, and on DigitalOcean you opt in per request with a prompt_cache_retention parameter set to either in_memory or 24h. Caching is best-effort: it applies when a request’s input tokens match the prefix of a previous response, but a hit is not guaranteed. The cache-read discount is not a single number, and this is the detail worth getting right, because DigitalOcean’s own pricing page shows it varies sharply by model. On the GPT-4o family the cached-read rate is exactly half the input rate (GPT-4o: $2.50 input, $1.25 cached read), a 50 percent discount. On GPT-4.1 and o3 it is 75 percent off. On the GPT-5 family it is 90 percent off (GPT-5: $1.25 input, $0.125 cached read). These figures are confirmed directly on DigitalOcean’s own Inference Pricing page, verified July 14, 2026. Separately, OpenAI’s own pricing documentation states that the newest GPT-5.6 models (Sol, Terra, Luna) add a cache-write rate of 1.25 times the input price, which would make them behave more like Anthropic’s explicit model than older OpenAI models. I could not confirm this specific line item on DigitalOcean’s own pricing page, since GPT-5.6 became available on DigitalOcean after that page’s last verification date. Verify the current rate for your specific model on the Inference pricing page rather than assuming a flat percentage. Source: DigitalOcean Prompt Caching how-to and Inference pricing, verified July 14, 2026.

Open-source models on DigitalOcean’s Serverless Inference Support prompt caching. As per DigitalOcean’s current documentation, prompt caching for open-source models is in public preview, and models such as DeepSeek V3.2 cache automatically. You do not set cache_control or prompt_cache_retention at all; caching is applied best-effort whenever a request’s tokens match the prefix of a previous request. Caches are isolated per customer account, derived from a per-account key, so one account can never read or infer another’s cached content. Two properties make the open-source path behave differently from the commercial models above. First, there is no 1,024-token minimum: DigitalOcean’s own documented example shows a 214-token DeepSeek request serving 128 tokens from cache. Second, there is no fixed TTL; the docs state caches are held for an unbounded period but should be treated as opportunistic, so you should not depend on a cache hit for predictable cost. Cached tokens are reported in two places on the usage object, cache_read_input_tokens and prompt_tokens_details.cached_tokens, and are billed at the model’s discounted cache-read rate. That discount, from DigitalOcean’s pricing page, runs roughly 46 to 80 percent off input depending on the model (DeepSeek V3.2: $0.425 input, $0.15 cached read; Qwen3 Coder Flash: $0.45 input, $0.09 cached read), with no separate write premium. Sources: DigitalOcean Prompt Caching how-to, Inference Features reference, and Inference pricing, verified July 14, 2026.

Note that not every open-source model carries a managed caching rate. DigitalOcean’s pricing page lists a prompt-caching rate for models like DeepSeek V3.2, the Qwen 3 series, GLM, Kimi, and several NVIDIA and MiniMax models, but not for others, including Meta’s Llama 3.3 70B, which is the model this piece benchmarks. For the full current list, see Foundation Models.

Self-hosted models on a DigitalOcean GPU Droplet are the fallback for any open model without a managed caching rate, and for full control over the caching layer. Modern inference engines, vLLM, SGLang, and TensorRT-LLM, all support automatic prefix caching within a replica, matching incoming prompts against previously cached KV state with no user configuration, as DigitalOcean’s Advanced Prompt Caching at Scale engineering post explains in detail. vLLM enables it by default. There is no write premium, no per-token discount to track, and, as the measured run later in this piece confirms, no 1,024-token minimum: vLLM caches in 16-token blocks, and a 155-token prompt hit the cache on its second occurrence. You pay for the GPU by the hour ($3.44 per hour for the single-H200 Droplet used in this piece’s test, per GPU Droplet pricing) whether the cache hits or not, so the benefit shows up as latency and capacity rather than a line-item discount: every cached prefill is GPU time freed for serving other requests.

All DigitalOcean figures below are from DigitalOcean’s own prompt caching how-to and Inference pricing pages, verified as of July, 2026.

Implementing caching on DigitalOcean, provider by provider

All three managed paths use the same OpenAI-compatible endpoint, https://inference.do-ai.run/v1, and the same model access key in the Authorization header. What changes per provider is how you request caching and where the cache counts show up in the usage object. The examples below use the Python standard library so they run with no dependencies, and they follow DigitalOcean’s Prompt Caching how-to exactly. Set your key once:

export DO_MODEL_ACCESS_KEY="your-model-access-key"

A small shared helper reads the caching fields, which differ slightly across providers:

import os
import json
import urllib.request

BASE_URL = "https://inference.do-ai.run/v1"
HEADERS = {
    "Content-Type": "application/json",
    "Authorization": f"Bearer {os.environ['DO_MODEL_ACCESS_KEY']}",
}


def post_chat(body):
    req = urllib.request.Request(
        BASE_URL + "/chat/completions",
        data=json.dumps(body).encode(),
        headers=HEADERS,
    )
    with urllib.request.urlopen(req, timeout=120) as r:
        return json.loads(r.read())


def cache_report(usage):
    """Normalize the cache fields DigitalOcean returns across model families."""
    details = usage.get("prompt_tokens_details") or {}
    return {
        "prompt_tokens": usage.get("prompt_tokens"),
        "cache_created": usage.get("cache_created_input_tokens", 0),
        "cache_read": usage.get("cache_read_input_tokens", 0),
        # open-source models also expose this field:
        "cached_tokens": details.get("cached_tokens", 0),
    }

Anthropic models: explicit cache_control. You mark the end of the stable block with a cache_control breakpoint of type: ephemeral and a ttl of 5m (default) or 1h. Everything up to and including that block is cached. Put the large, stable content (system instructions, tool definitions, reference documents) first, mark it, then append the volatile user turn:

STABLE_SYSTEM_PROMPT = "..."  # your long, stable instructions and reference material

def anthropic_turn(user_message, ttl="1h"):
    body = {
        "model": "claude-haiku-4.5",
        "messages": [
            {
                "role": "system",
                "content": [
                    {
                        "type": "text",
                        "text": STABLE_SYSTEM_PROMPT,
                        "cache_control": {"type": "ephemeral", "ttl": ttl},
                    }
                ],
            },
            {"role": "user", "content": user_message},
        ],
    }
    response = post_chat(body)
    return cache_report(response["usage"])


# First call writes the cache (cache_created > 0); the second reads it (cache_read > 0).
print(anthropic_turn("Summarize the attached policy in three bullets."))
print(anthropic_turn("Now list any compliance risks."))

On the first request the usage object reports cache_created_input_tokens greater than 0 and cache_read_input_tokens at 0; on the repeat within the TTL window it flips, cache_read_input_tokens becomes the cached count and cache_created_input_tokens returns to 0. That read is billed at 0.10x the standard input rate.

OpenAI models: opt in with prompt_cache_retention. No cache_control markers; you set one top-level parameter to in_memory or 24h, and caching applies best-effort to any prompt of 1,024 tokens or more. Keep the stable content at the front of the message list so the matching prefix is as long as possible:

def openai_turn(messages, retention="24h"):
    body = {
        "model": "gpt-5",
        "prompt_cache_retention": retention,  # "in_memory" or "24h"
        "messages": messages,
        "temperature": 0.2,
    }
    response = post_chat(body)
    return cache_report(response["usage"])


history = [
    {"role": "system", "content": STABLE_SYSTEM_PROMPT},  # >= 1,024 tokens to be eligible
    {"role": "user", "content": "Summarize the attached policy in three bullets."},
]
print(openai_turn(history))  # cache_created_input_tokens on first eligible call
history.append({"role": "user", "content": "Now list any compliance risks."})
print(openai_turn(history))  # cache_read_input_tokens on the repeat

The cached-read discount varies by model (50% off on GPT-4o, 75% on GPT-4.1 and o3, 90% on GPT-5), so read the returned cache_read_input_tokens against your model’s rate rather than assuming a flat percentage.

Open-source models: nothing to configure. DeepSeek V3.2 and the other open-source models with a managed caching rate cache automatically. You send an ordinary request, and cached tokens appear in both cache_read_input_tokens and prompt_tokens_details.cached_tokens:

def open_source_turn(messages):
    body = {
        "model": "deepseek-3.2",  # caches automatically, no cache params
        "messages": messages,
        "max_tokens": 512,
    }
    response = post_chat(body)
    return cache_report(response["usage"])


history = [
    {"role": "system", "content": STABLE_SYSTEM_PROMPT},
    {"role": "user", "content": "Summarize the attached policy in three bullets."},
]
print(open_source_turn(history))
history.append({"role": "user", "content": "Now list any compliance risks."})
print(open_source_turn(history))  # cached_tokens > 0 once the prefix matches

The equivalent raw request, straight from DigitalOcean’s documentation, is a plain curl with no caching parameters at all:

curl https://inference.do-ai.run/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $DO_MODEL_ACCESS_KEY" \
  -d '{
    "model": "deepseek-3.2",
    "messages": [
      {"role": "user", "content": "Summarize this: ..."}
    ],
    "max_tokens": 512
  }'

On a hit, the response usage block reports both fields, for example "cache_read_input_tokens": 128 and "prompt_tokens_details": {"cached_tokens": 128} against a prompt_tokens of 214, meaning 128 of the 214 input tokens were served from cache at the discounted rate and the remaining 86 were billed at the standard rate.

For any open model without a managed caching rate (such as the Llama 3.3 70B this piece benchmarks), self-host it on a GPU Droplet, where vLLM applies prefix caching automatically with no parameters and no minimum. That path, and how to measure it directly, is the subject of the firsthand test later in this piece.

Prompt caching and cold start mitigation are different mechanisms solving different problems, and provider marketing sometimes blurs them, so it is worth separating them explicitly.

Prompt caching addresses repeated input content within an already-running model deployment. It has nothing to do with whether the model itself is loaded into GPU memory. Cold starts address the opposite problem: a model deployment that scaled to zero during an idle period must reload before it can serve the first request again. That reload time shows up entirely in time-to-first-token, the delay before the first output token streams back, and does not affect time-per-output-token, the steady-state generation speed once the response is underway.

On DigitalOcean’s Serverless Inference, cold starts can happen after an idle period, and the first request following that idle time can be slower while capacity spins back up. This is the tradeoff for scaling to zero and paying nothing while idle, and DigitalOcean’s own guidance is that steady, latency-sensitive workloads are a better fit for a dedicated deployment with reserved capacity rather than serverless. Source: Where to Host Your Open-Source Model, DigitalOcean Community. For fine-tuned adapters specifically, DigitalOcean’s own analysis found that cold starts spike time-to-first-token by roughly a few hundred milliseconds when only a lightweight adapter needs reloading, and that this can be mitigated with periodic keep-alive requests. Source: Fine-Tuned LLMs on Serverless Architecture, DigitalOcean Community.

What actually reduces cold start latency is model pre-loading, warm pools of ready capacity, or a dedicated deployment that never scales to zero. What actually reduces cost on repeated input content is prompt caching. Neither substitutes for the other, and a workload with both problems needs both solutions.

Caching skips reprocessing of repeated content on an already-running deployment. Cold start mitigation keeps the deployment itself ready to serve. They solve different problems and neither fixes the other.

Here is the part most explainers skip: an actual formula you can rerun with your own provider’s published rates, not just a headline percentage.

Define the write multiplier w as the cost of the first request that establishes a cached prefix, relative to standard input price, and the read multiplier r as the cost of a subsequent request that hits the same cached prefix, also relative to standard price. The number of cache hits needed before caching becomes cheaper than never caching at all, which I will call the break-even point h*, is:

h* = (w − 1) / (1 − r)

Apply Anthropic’s own published numbers. On the 5-minute tier, w = 1.25 and r = 0.10, giving h* = 0.25 / 0.90 ≈ 0.28. Since you cannot have a fractional hit, this rounds up to 1: after a single cache hit, the second request in a two-request sequence, the combined cost of write plus that one read is already lower than paying full price twice. I verified this directly: two full-price requests cost 2.0 times the standard rate, while one write plus one read costs 1.25 plus 0.10, or 1.35, which is cheaper. On the 1-hour tier, w = 2.0, giving h* = 1.0 / 0.90 ≈ 1.11, rounding up to 2 hits needed before the longer, more expensive write pays for itself.

Apply the numbers for a no-write-premium model, which on DigitalOcean covers most OpenAI models (GPT-4o, GPT-4.1, GPT-5, the o-series) and every open-source model. Here w = 1, and the formula gives h* = 0. Every single cache hit is pure savings from the very first one, because there was never an extra cost to recover. This is the practical difference between explicit and automatic caching: Anthropic’s model can lose money on a workload with too few hits, while a zero-premium model structurally cannot. One caveat worth flagging directly: OpenAI’s own pricing documentation states that GPT-5.6 (Sol, Terra, Luna) introduces a cache-write rate of 1.25 times the uncached input rate, a change from earlier OpenAI models. I could not confirm this line item on DigitalOcean’s own Inference Pricing page as of this writing, since that page was last verified July 1, 2026, before GPT-5.6 became available on DigitalOcean’s platform. If you are using GPT-5.6 through DigitalOcean, check the live pricing page for the current cache-write rate before assuming h* = 0.

Each line shows the point where caching becomes cheaper than paying standard price every time. A write premium means the first request costs more, and you have to earn that back before caching pays off at all.

What this formula does and does not tell you

This formula tells you, once you know a provider’s rates, exactly how many hits you need. It does not tell you whether your traffic will actually produce that many hits. That second question depends on the three variables named in this piece’s thesis, and each maps to a specific, measurable failure mode.

Prefix stability below the minimum cacheable length is a hard gate for commercial models, not a matter of degree. If your entire prompt is under 1,024 tokens, Anthropic and OpenAI models (on DigitalOcean and elsewhere) will not cache any of it, no matter how many times you repeat it or how quickly requests arrive. This is worth naming directly against the chatbot workload this piece uses as an example later: a 400-token system prompt is below that threshold on both. That workload cannot cache at all on those models, independent of session depth or timing. The gate does not apply everywhere, though: DigitalOcean’s open-source models have no documented minimum (the docs show caching at 214 tokens), and the measured short-prompt test later in this piece shows a self-hosted engine caches at 155 tokens. The gate is a commercial-model policy, not a property of the mechanism, which matters if a small-prompt workload is pushing you toward an open-source or self-hosted model.

Session depth below h* for your provider’s tier means you never recover the write premium, if one exists. A single one-shot request that writes a cache entry and is never followed by a matching request pays the premium and gets nothing back. This is a straight loss on any provider that charges one.

Interarrival time exceeding the cache lifetime means every request is treated as new regardless of session depth. If your requests are 7 minutes apart and you are on Anthropic’s 5-minute tier, every single request rewrites the cache and none of them read it, guaranteeing you pay the write premium every time and never receive the discount. The fix here is not to abandon caching, it is to use the 1-hour tier if your traffic is sparse but eventually repeats, accepting the higher h* of 2 hits in exchange for a wider window to reach them.

A worked example, with the hit rate measured

To show how the formula scales with real session depth, take a 20-turn agent session on Anthropic’s 5-minute tier, where a stable 3,000-token system prompt and tool definition block is reused on every turn and every turn arrives well within 5 minutes of the last. The pricing multipliers here are Anthropic’s published rates. The hit rate, one miss on the first turn and hits on every turn after, is not an assumption in this piece: the measured 12-turn session in the firsthand test below, run against Llama 3.3 70B on a DigitalOcean GPU Droplet with exactly this traffic shape, produced a cache hit on 11 of 11 subsequent turns with a 98.7 percent token-level hit rate. A stable prefix reused within the cache window hits essentially every time, on every caching implementation this piece examined.

Apply that hit pattern to Anthropic’s pricing. The tokens in the stable block are billed as 1 write (1.25x) plus 19 reads (0.10x each), for a total of 1.25 + 1.9 = 3.15 times the standard rate across 20 requests, versus 20.0 times the standard rate with no caching at all. That is roughly an 84 percent reduction on that portion of the input, well above the 1-hit break-even point, and consistent with the “50 to 90 percent” range reported across the industry sources I reviewed for long agent sessions. Contrast a 3-turn chatbot session with a 4,000-token system prompt, comfortably above the minimum, but only 2 opportunities to hit. At 2 hits: 1.25 + 0.20 = 1.45 times standard across 3 requests, versus 3.0 times standard uncached, a 52 percent reduction on that portion, smaller in absolute terms because there are fewer requests to spread the write cost across, but still past break-even. The chatbot example that fails outright is not this one. It is the 400-token prompt from the earlier gate, which never qualifies to cache on a managed provider in the first place.

The one caveat the measured run adds: the near-perfect hit rate holds only when the prefix is byte-for-byte stable. The same measured session rerun with a timestamp and request ID at the top of the system prompt, everything else identical, dropped from a 98.7 percent token hit rate to 0.7 percent. If your worked example assumes 19 of 20 hits but your prompt template injects anything dynamic ahead of the stable block, your real number is the second one. Measure your own traffic with the harness in the next section before you plan a budget around either figure.

This is the section that matters most in this article, because it replaces assumptions with a run you can reproduce. The test harness below was executed against a live model serving endpoint, and every number in the tables that follow came directly from that test run. The managed-platform DIY test harness is included at the end of this section for anyone with a key. The caching mechanism being measured, exact-prefix KV reuse, is the same one every provider implements, so the hit-rate behavior transfers even where the billing model differs.

The measured environment

• Infrastructure: DigitalOcean GPU Droplet, gpu-h200x1-141gb (1x NVIDIA H200, 141 GB VRAM, 24 vCPUs, 240 GB RAM), $3.44 per hour, NYC2 region, created from the 1-Click Inference Ready image. See GPU Droplet pricing.
• Serving stack: vLLM 0.24.0 (vllm/vllm-openai:v0.24.0 Docker image), which enables automatic prefix caching by default (enable_prefix_caching=True in the engine startup log).
• Model: RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic, FP8 KV cache, 4,096-token context, single GPU.

The exact serving command, so the environment is reproducible:

docker run -d --name vllm-bench --gpus all \
  -v /root/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  vllm/vllm-openai:v0.24.0 \
  RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic \
  --kv-cache-dtype fp8 \
  --max-model-len 4096 \
  --max-num-seqs 256 \
  --gpu-memory-utilization 0.92 \
  --tensor-parallel-size 1 \
  --port 8000

nvidia-smi on the Droplet at test time, confirming the hardware and the loaded engine (output trimmed to the relevant lines):

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 575.57.08              Driver Version: 575.57.08      CUDA Version: 12.9     |
|-----------------------------------------+------------------------+----------------------+
|   0  NVIDIA H200                    On  |   00000000:83:00.0 Off |                    0 |
| N/A   43C    P0            125W /  700W |  134095MiB / 143771MiB |      0%      Default |
+-----------------------------------------+------------------------+----------------------+
|    0   N/A  N/A         1644641      C   VLLM::EngineCore                      13408... |
+-----------------------------------------------------------------------------------------+

And the engine’s own startup log line confirming prefix caching is active with no configuration on our part:

Initializing a V1 LLM engine (v0.24.0) with config:
model='RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic', ...
kv_cache_dtype=fp8, ... enable_prefix_caching=True, enable_chunked_prefill=True, ...

Test design

Four synthetic workloads against the same model, holding everything constant except traffic shape:

• TTFT sweep. Unique prefixes of roughly 512, 1,024, 2,048, and 3,072 tokens, each sent once cold and then three times warm, to measure the latency value of a hit as prefix size grows.
• Agent workload. A stable 2,048-token system prompt followed by 12 sequential turns with growing conversation history, each turn arriving about one second after the last.
• Prefix-buster workload. The identical agent workload, with one change: a request ID and timestamp injected at the top of the system prompt on every turn, the single most common caching mistake in production.
• Short-prompt workload. A 117-token system prompt repeated four times, to test whether the 1,024-token minimum that commercial models enforce exists at the engine level.

What was measured, and how

vLLM does not return per-request cached-token counts in the OpenAI-compatible usage object the way managed providers do, but it exposes engine-level counters on its Prometheus /metrics endpoint: vllm:prefix_cache_queries_total (tokens checked against the cache) and vllm:prefix_cache_hits_total (tokens served from it). One naming note: vLLM’s own metrics documentation lists these as vllm:prefix_cache_queries and vllm:prefix_cache_hits without the _total suffix. Prometheus client libraries conventionally append _total to Counter-type metrics in the actual exposed text output, which is consistent with the curl output below, but if your version differs, list your endpoint’s actual metric names with curl localhost:8000/metrics | grep prefix_cache before relying on the exact string. The harness issues requests serially and diffs those counters around each request, which attributes hit tokens to individual requests exactly. Time-to-first-token is measured client-side with streaming enabled, timestamping the first content chunk.

This is what those counters look like, queried directly on the Droplet:

curl -s localhost:8000/metrics | grep -E "^vllm:prefix_cache_(queries|hits)_total"

vllm:prefix_cache_queries_total{engine="0",model_name="RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic"} 2.99845e+06
vllm:prefix_cache_hits_total{engine="0",model_name="RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic"} 194672.0

The test harness

This is the complete script I wrote to produce every measured number in this article, cache_bench.py, runnable against any vLLM endpoint with no dependencies beyond the Python standard library. Save it, point --base-url at your endpoint, and it runs all four workloads and writes the raw per-request records to a JSON file.

The measurement pattern is worth stealing even if you change the workloads: measured_turn diffs the engine’s prefix-cache counters around each serial request, which attributes cached tokens to individual requests exactly, and make_prefix calibrates prompt sizes against vLLM’s /tokenize endpoint so “a 2,048-token prefix” means what it says.

#!/usr/bin/env python3
"""
Prefix-cache benchmark for a vLLM OpenAI-compatible endpoint.

Measures, per request:
  - TTFT (time to first streamed token)
  - total latency
  - prompt tokens (from the usage object)
  - cached prefix tokens (by diffing vLLM's /metrics prefix-cache counters
    around each request, so requests are issued serially)

Workloads:
  1. ttft_sweep     - cold vs warm TTFT across prefix sizes
  2. agent_session  - stable system prompt + growing history, N turns
  3. prefix_buster  - identical session but a timestamp at the TOP of the
                      system prompt, which breaks exact-prefix matching
  4. short_prompt   - a ~120-token prompt repeated, to observe block-level
                      behavior below typical managed-provider minimums

Run on the droplet (or anywhere that can reach the endpoint):
  python3 cache_bench.py --base-url http://localhost:8000 --out results.json
"""

import argparse
import json
import time
import urllib.request
import uuid

WORD = "inference "  # 1 word ~= 1.2 tokens for this tokenizer; calibrated below


def http_json(url, payload=None, timeout=120):
    data = json.dumps(payload).encode() if payload is not None else None
    req = urllib.request.Request(
        url, data=data, headers={"Content-Type": "application/json"}
    )
    with urllib.request.urlopen(req, timeout=timeout) as r:
        return json.loads(r.read())


def get_cache_counters(base_url):
    with urllib.request.urlopen(base_url + "/metrics", timeout=30) as r:
        text = r.read().decode()
    queries = hits = 0.0
    for line in text.splitlines():
        if line.startswith("vllm:prefix_cache_queries_total"):
            queries = float(line.rsplit(" ", 1)[1])
        elif line.startswith("vllm:prefix_cache_hits_total"):
            hits = float(line.rsplit(" ", 1)[1])
    return queries, hits


def streamed_request(base_url, model, messages, max_tokens=32):
    """Send a streaming chat completion; return TTFT, total time, usage."""
    payload = {
        "model": model,
        "messages": messages,
        "max_tokens": max_tokens,
        "temperature": 0,
        "stream": True,
        "stream_options": {"include_usage": True},
    }
    req = urllib.request.Request(
        base_url + "/v1/chat/completions",
        data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json"},
    )
    start = time.perf_counter()
    ttft = None
    usage = {}
    content_parts = []
    with urllib.request.urlopen(req, timeout=300) as r:
        for raw in r:
            line = raw.decode().strip()
            if not line.startswith("data:"):
                continue
            body = line[5:].strip()
            if body == "[DONE]":
                break
            chunk = json.loads(body)
            if chunk.get("usage"):
                usage = chunk["usage"]
            for choice in chunk.get("choices", []):
                delta = choice.get("delta", {})
                if delta.get("content"):
                    if ttft is None:
                        ttft = time.perf_counter() - start
                    content_parts.append(delta["content"])
    total = time.perf_counter() - start
    return {
        "ttft_s": ttft,
        "total_s": total,
        "prompt_tokens": usage.get("prompt_tokens"),
        "completion_tokens": usage.get("completion_tokens"),
        "content": "".join(content_parts),
    }


def measured_turn(base_url, model, messages, max_tokens=32):
    q0, h0 = get_cache_counters(base_url)
    result = streamed_request(base_url, model, messages, max_tokens)
    q1, h1 = get_cache_counters(base_url)
    result["cache_query_tokens"] = int(q1 - q0)
    result["cache_hit_tokens"] = int(h1 - h0)
    return result


def make_prefix(base_url, model, target_tokens):
    """Build a unique prefix of roughly target_tokens, calibrated via tokenize API."""
    tag = f"run-{uuid.uuid4().hex[:8]} "
    words = int(target_tokens * 0.9)
    text = tag + (WORD * words)
    # calibrate with the /tokenize endpoint
    for _ in range(6):
        n = http_json(
            base_url + "/tokenize", {"model": model, "prompt": text}
        )["count"]
        if abs(n - target_tokens) <= 8:
            break
        delta_words = int((target_tokens - n) * 0.8)
        if delta_words > 0:
            text += WORD * delta_words
        else:
            text = text[: delta_words * len(WORD)]
    return text, n


def ttft_sweep(base_url, model, sizes, warm_repeats=3):
    out = []
    for size in sizes:
        prefix, actual = make_prefix(base_url, model, size)
        messages = [
            {"role": "system", "content": prefix},
            {"role": "user", "content": "Reply with the single word: ready."},
        ]
        cold = measured_turn(base_url, model, messages, max_tokens=8)
        warms = [
            measured_turn(base_url, model, messages, max_tokens=8)
            for _ in range(warm_repeats)
        ]
        out.append(
            {
                "target_prefix_tokens": size,
                "actual_prefix_tokens": actual,
                "cold": cold,
                "warm": warms,
            }
        )
        print(f"[ttft_sweep] {size} tokens done", flush=True)
    return out


def agent_session(base_url, model, prefix_tokens=2048, turns=12, bust_prefix=False):
    prefix, actual = make_prefix(base_url, model, prefix_tokens)
    results = []
    history = []
    for i in range(turns):
        system = prefix
        if bust_prefix:
            # per-request variable content at the TOP: the classic mistake
            system = f"[request-id: {uuid.uuid4()}] [ts: {time.time()}]\n" + prefix
        messages = (
            [{"role": "system", "content": system}]
            + history
            + [{"role": "user", "content": f"Turn {i}: answer in five words or fewer."}]
        )
        r = measured_turn(base_url, model, messages, max_tokens=16)
        r["turn"] = i
        results.append(r)
        history.append({"role": "user", "content": f"Turn {i}: answer in five words or fewer."})
        history.append({"role": "assistant", "content": r["content"] or "ok"})
        time.sleep(1)
    return {"prefix_tokens": actual, "turns": results}


def short_prompt(base_url, model, repeats=4):
    prefix, actual = make_prefix(base_url, model, 120)
    messages = [
        {"role": "system", "content": prefix},
        {"role": "user", "content": "Reply with one word."},
    ]
    return {
        "prefix_tokens": actual,
        "requests": [
            measured_turn(base_url, model, messages, max_tokens=8)
            for _ in range(repeats)
        ],
    }


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--base-url", default="http://localhost:8000")
    ap.add_argument("--out", default="cache_bench_results.json")
    args = ap.parse_args()

    model = http_json(args.base_url + "/v1/models")["data"][0]["id"]
    print(f"Model: {model}", flush=True)

    results = {"model": model, "timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())}

    print("== TTFT sweep ==", flush=True)
    results["ttft_sweep"] = ttft_sweep(args.base_url, model, [512, 1024, 2048, 3072])

    print("== Agent session (stable prefix) ==", flush=True)
    results["agent_session"] = agent_session(args.base_url, model, 2048, 12)

    print("== Prefix-buster session (timestamp first) ==", flush=True)
    results["prefix_buster"] = agent_session(args.base_url, model, 2048, 12, bust_prefix=True)

    print("== Short prompt ==", flush=True)
    results["short_prompt"] = short_prompt(args.base_url, model)

    with open(args.out, "w") as f:
        json.dump(results, f, indent=2)
    print(f"Wrote {args.out}", flush=True)


if __name__ == "__main__":
    main()

The full run, executed on the Droplet:

python3 cache_bench.py --base-url http://localhost:8000 --out cache_bench_results.json

Model: RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic
== TTFT sweep ==
[ttft_sweep] 512 tokens done
[ttft_sweep] 1024 tokens done
[ttft_sweep] 2048 tokens done
[ttft_sweep] 3072 tokens done
== Agent session (stable prefix) ==
== Prefix-buster session (timestamp first) ==
== Short prompt ==
Wrote cache_bench_results.json

Here is how the results look like:

{
  "model": "RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic",
  "timestamp": "2026-07-14T13:45:35Z",
  "ttft_sweep": [
    {
      "target_prefix_tokens": 512,
      "actual_prefix_tokens": 510,
      "cold": {
        "ttft_s": 0.11955002718605101,
        "total_s": 0.14034194918349385,
        "prompt_tokens": 551,
        "completion_tokens": 2,
        "content": "ready",
        "cache_query_tokens": 551,
        "cache_hit_tokens": 16
      },
      "warm": [
        {
          "ttft_s": 0.03656787099316716,
          "total_s": 0.05752997612580657,
          "prompt_tokens": 551,
          "completion_tokens": 2,
          "content": "ready",
          "cache_query_tokens": 551,
          "cache_hit_tokens": 544
        },
        {
          "ttft_s": 0.03680372005328536,
          "total_s": 0.05818751105107367,
          "prompt_tokens": 551,
          "completion_tokens": 2,
          "content": "ready",
          "cache_query_tokens": 551,
          "cache_hit_tokens": 544
        },

Every table below is read directly out of that results file, and the raw per-request records are shown alongside each one.

Measured result 1: a cache hit cut TTFT by 69 to 92 percent, scaling with prefix size

Each row is one unique prefix sent cold (first occurrence, no cache entry) and then three times warm (identical prefix, cache populated). Warm TTFT is the mean of the three repeats, which varied by less than 2 ms.

Cold TTFT scales with prefix size because prefill does. Warm TTFT does not, because a cache hit skips prefill for the matched portion entirely.

The raw records for the 3,072-token case, exactly as the harness logged them:

{"cold": {"ttft_s": 0.4615, "prompt_tokens": 3110, "cache_query_tokens": 3110, "cache_hit_tokens": 16},
 "warm": [
   {"ttft_s": 0.0389, "prompt_tokens": 3110, "cache_query_tokens": 3110, "cache_hit_tokens": 3104},
   {"ttft_s": 0.0384, "prompt_tokens": 3110, "cache_query_tokens": 3110, "cache_hit_tokens": 3104},
   {"ttft_s": 0.0383, "prompt_tokens": 3110, "cache_query_tokens": 3110, "cache_hit_tokens": 3104}]}

Two things are worth noting in this table. First, cold TTFT grows linearly with prompt size, at roughly 134 ms per 1,000 prompt tokens on this hardware and model, which works out to a prefill throughput of about 7,500 tokens per second.

That slope is the thing caching deletes. Second, warm TTFT is flat at 37 to 39 ms regardless of prefix size, because a hit skips prefill for the matched portion entirely. The longer your stable prefix, the more a hit is worth, which is the measured confirmation of why the decision framework in this piece weights prefix size so heavily. The 92 percent reduction at 3,110 tokens lands at the top of the “up to 80 percent” range OpenAI advertises for its own managed caching, measured here on open source infrastructure anyone can rent by the hour.

Cached-token counts arrive in multiples of 16 because vLLM caches KV state in 16-token blocks; the few uncached tokens on each warm request are the final partial block. The blocks holding the chat template header are shared across all requests to the endpoint, which is why even a “cold” request shows 16 cached tokens.

Measured result 2: a 12-turn agent session sustained a 98.7 percent hit rate after turn one

This is the workload the worked example earlier in this article models: a stable 2,080-token system prompt, then 12 turns of accumulating conversation history, one second apart.

One cold turn, then eleven flat, fast ones. Each turn extends the previous turn’s cached prefix, so the hit rate compounds instead of resetting.

The raw records for the first four turns and the last:

{"turn": 0, "prompt_tokens": 2084, "cache_query_tokens": 2084, "cache_hit_tokens": 16,   "ttft_s": 0.3267}
{"turn": 1, "prompt_tokens": 2110, "cache_query_tokens": 2110, "cache_hit_tokens": 2080, "ttft_s": 0.0416}
{"turn": 2, "prompt_tokens": 2136, "cache_query_tokens": 2136, "cache_hit_tokens": 2112, "ttft_s": 0.0425}
{"turn": 3, "prompt_tokens": 2163, "cache_query_tokens": 2163, "cache_hit_tokens": 2128, "ttft_s": 0.0394}
{"turn": 11, "prompt_tokens": 2372, "cache_query_tokens": 2372, "cache_hit_tokens": 2336, "ttft_s": 0.0377}

Across turns 1 through 11, the token-level hit rate was 98.7 percent and mean TTFT was 40 ms, versus 327 ms on the cold first turn. Note what is being cached by turn 3: not just the system prompt but the accumulated conversation history too, because each turn’s prompt is a strict prefix extension of the previous turn’s. This is the append-only property that makes agent loops the best-case caching workload, and it is why the earlier worked example’s “hit on every turn after the first” pattern is a measurement here, not an assumption.

Measured result 3: one timestamp at the top of the prompt collapsed the hit rate from 98.7 to 0.7 percent

The same 12-turn session, rerun with exactly one change: each turn’s system prompt began with [request-id: <uuid>] [ts: <time>] ahead of the stable 2,080-token block.

Same session, same 2,080-token stable block, one difference. Exact-prefix matching means nothing after the first changed byte is ever reached, so the damage is total, not partial.

The raw records for the first four turns, side by side with the stable-prefix records above. The only cached tokens on any turn are the 16-token chat-template block; the 2,080-token stable body behind the timestamp never hits:

{"turn": 0, "prompt_tokens": 2122, "cache_query_tokens": 2122, "cache_hit_tokens": 16, "ttft_s": 0.3150}
{"turn": 1, "prompt_tokens": 2152, "cache_query_tokens": 2152, "cache_hit_tokens": 16, "ttft_s": 0.3137}
{"turn": 2, "prompt_tokens": 2176, "cache_query_tokens": 2176, "cache_hit_tokens": 16, "ttft_s": 0.3133}
{"turn": 3, "prompt_tokens": 2202, "cache_query_tokens": 2202, "cache_hit_tokens": 16, "ttft_s": 0.3048}

Every turn paid full prefill, roughly 8x the warm TTFT, and on a provider with per-token pricing every turn would have paid full input price too. Nothing about the workload’s session depth, interarrival time, or prefix size changed. The cache was defeated purely by prompt structure, and the damage was total, not partial, because exact-prefix matching means nothing after the first changed byte is ever reached. This is the single most common production caching mistake, and this is what it measures as.

Measured result 4: the 1,024-token minimum is a commercial-model policy, not an engine limit

A 117-token system prompt, 155 prompt tokens in total, far below the commercial-model minimum, repeated four times:

Anthropic and OpenAI will not cache this prompt at all, since it is below their 1,024-token floor. A self-hosted engine has no such floor and cached it at vLLM’s 16-token block granularity instead.

The raw records:

{"prompt_tokens": 155, "cache_query_tokens": 155, "cache_hit_tokens": 16,  "ttft_s": 0.0491}
{"prompt_tokens": 155, "cache_query_tokens": 155, "cache_hit_tokens": 144, "ttft_s": 0.0368}
{"prompt_tokens": 155, "cache_query_tokens": 155, "cache_hit_tokens": 144, "ttft_s": 0.0352}
{"prompt_tokens": 155, "cache_query_tokens": 155, "cache_hit_tokens": 144, "ttft_s": 0.0367}

vLLM cached it without complaint, at its 16-token block granularity. The 1,024-token minimum that Anthropic and OpenAI models enforce is a policy choice about when caching is worth the bookkeeping, not a property of the mechanism. Two paths on DigitalOcean escape that floor: its managed open-source models, which the docs show caching at 214 tokens, and a self-hosted engine like the one measured here, caching at 155. So a small-prompt workload that is structurally ineligible on Anthropic or OpenAI models is not out of options; it just needs an open-source or self-hosted model. That changes the earlier hard gate from “cannot cache at all” to “cannot cache on a commercial model.”

What the measured numbers mean for cost on self-hosted infrastructure

On a per-token provider, a cache hit is a line-item discount. On an hourly GPU, the accounting is different: the H200 costs $3.44 per hour whether the cache hits or not, so the saving arrives as reclaimed capacity. At the measured cold prefill rate of about 7,500 tokens per second, every million cached prompt tokens is roughly 134 GPU-seconds of prefill work the card does not repeat, which is GPU time available for decoding other requests instead. For a latency-sensitive workload, the more direct reading is the TTFT table above: the difference between 462 ms and 39 ms to first token is the difference users actually feel, on infrastructure whose caching behavior you fully control.

One scaling caveat that matters as soon as you outgrow one GPU: these hit rates were measured on a single replica, where every request lands on the same cache. Under round-robin load balancing across N replicas, an identical prefix has roughly a 1-in-N chance of landing where its cache entry lives, and the hit rate measured here degrades accordingly. Session-affinity routing, pinning a session’s requests to one replica, is the standard fix, and DigitalOcean’s Advanced Prompt Caching at Scale blog covers that architecture, tiered prefix routing for multi-task deployments, and when a shared cross-replica cache is worth its transfer latency.

Running the same test against Serverless Inference

The DIY test harness below targets DigitalOcean’s managed Serverless Inference endpoint, where caching shows up in the response usage object rather than in engine metrics. The default model here is deepseek-3.2, an open-source model that, per DigitalOcean’s docs, caches automatically with no cache_control or prompt_cache_retention parameter. The field to watch for open-source and Anthropic models is cache_read_input_tokens; open-source models additionally report prompt_tokens_details.cached_tokens. For OpenAI models, set prompt_cache_retention (shown commented out) and expect caching only on prompts of 1,024 tokens or more. Source: DigitalOcean Prompt Caching how-to.

import time
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["DO_MODEL_ACCESS_KEY"],
    base_url="https://inference.do-ai.run/v1",
)

STABLE_PREFIX = "..."  # your stable system prompt and tool definitions go here
MODEL = "deepseek-3.2"  # open-source model; caches automatically, no parameters needed


def cached_tokens(usage):
    """Read cached-token count across the fields DO populates per model family."""
    direct = getattr(usage, "cache_read_input_tokens", 0) or 0
    details = getattr(usage, "prompt_tokens_details", None)
    nested = getattr(details, "cached_tokens", 0) if details else 0
    return max(direct, nested or 0)


def run_turn(messages):
    start = time.perf_counter()
    response = client.chat.completions.create(
        model=MODEL,
        messages=messages,
        stream=False,
        # For OpenAI models, opt in to caching (1,024-token minimum applies):
        # extra_body={"prompt_cache_retention": "24h"},
    )
    elapsed = time.perf_counter() - start
    usage = response.usage
    return {
        "elapsed_seconds": elapsed,
        "prompt_tokens": usage.prompt_tokens,
        "cache_read_input_tokens": getattr(usage, "cache_read_input_tokens", 0),
        "cached_tokens": cached_tokens(usage),
    }


def agent_workload(turns=20, delay_seconds=2):
    history = [{"role": "system", "content": STABLE_PREFIX}]
    results = []
    for i in range(turns):
        history.append({"role": "user", "content": f"Turn {i}: continue the task."})
        result = run_turn(history)
        results.append(result)
        # capture the real assistant response so the prefix extends append-only:
        history.append({"role": "assistant", "content": "..."})
        time.sleep(delay_seconds)
    return results


def long_gap_workload(turns=5, delay_seconds=400):
    # For OpenAI models with prompt_cache_retention, a delay beyond the retention
    # window tests the interarrival failure mode. Open-source caches are unbounded
    # but best-effort, so a miss after a long gap is possible but not guaranteed.
    return agent_workload(turns=turns, delay_seconds=delay_seconds)


if __name__ == "__main__":
    print("Agent workload:")
    for r in agent_workload():
        print(r)
    print("Long-gap workload:")
    for r in long_gap_workload():
        print(r)

In the above code block, swap MODEL for an Anthropic model (and add a cache_control marker) or an OpenAI model (and uncomment prompt_cache_retention) to compare mechanics across different model families on the same endpoint.

Cache when:

• Your stable, repeated content is above the model’s minimum cacheable length, 1,024 tokens for Anthropic and OpenAI models. Open-source models on DigitalOcean and self-hosted engines have no such minimum.
• Your session depth clears the break-even point h* for your model’s pricing, 1 hit on Anthropic’s 5-minute tier, 2 hits on its 1-hour tier, and effectively 0 hits on any model with no cache-write rate (most OpenAI models and all open-source models on DigitalOcean).
• Your requests arrive within the cache’s TTL of each other. For OpenAI models you can set prompt_cache_retention up to 24 hours; open-source caches are unbounded but best-effort.
• You are running agent or coding workloads with fixed tool definitions and large, stable context, the scenario where session depth and prefix size both work in your favor at once. The measured 12-turn agent session above sustained a 98.7 percent token hit rate under exactly these conditions.
• You use an open-source model with a managed caching rate (DeepSeek V3.2, Qwen 3, GLM, and others), where caching is automatic and needs no parameters, or you self-host any open model on a GPU Droplet with vLLM, where prefix caching is on by default with no minimum prefix length.

Do not bother when:

• Your prompt is smaller than 1,024 tokens and you are on an Anthropic or OpenAI model. No amount of session depth or timing fixes this there, though an open-source or self-hosted model has no such floor.
• Your sessions are shallow enough, or your traffic sparse enough, that you will not clear h* before the cache expires, on a model that charges a cache-write premium (Anthropic and GPT-5.6).
• Your prompts change on every request in ways that break exact-prefix matching, defeating the mechanism regardless of size or frequency.

Design your prompts for caching:

Put stable content first and volatile content last. System prompt, then fixed tool definitions, then long static reference material, then slowly-changing conversation history, then the current user message, in that order. Never place a timestamp, a request ID, or any per-request variable ahead of the content you want cached, since exact-prefix matching means anything after the first change is never reached.

This ordering principle is consistent across every provider’s documentation I reviewed and is the single most common implementation mistake reported in production case studies of teams recovering low hit rates. It is also no longer just a reported mistake in this piece: the measured prefix-buster run above put one timestamp line at the top of an otherwise perfectly cacheable session and watched the hit rate fall from 98.7 percent to 0.7 percent.

Three gates, checked in order. A workload only needs to fail one to lose the economics, which is why session depth alone, without checking prefix size and timing, is not enough to decide.

If you run traffic through DigitalOcean’s Inference Router rather than a single fixed model, there is a mechanism interaction worth understanding directly, because it can silently destroy the cache economics described in this entire piece.

The router can select a different model for each request in a session based on what that specific request looks like, which is useful for cost governance across mixed workloads. But prompt caches are scoped to a specific model. If your agent’s first turn routes to one model and its second turn routes to another, the cache written by the first turn is never read by the second, and every turn pays full write cost with no accumulated savings, regardless of how well your prefix and session depth otherwise satisfy the break-even formula above.

DigitalOcean’s router exposes an X-Model-Affinity header specifically to prevent this. Setting it to a session identifier on the first request causes the router to route normally once, then pin every subsequent request with the same identifier to that same model, skipping routing and preserving the cache. According to DigitalOcean’s own documentation, in a 15-turn agentic loop where 90 percent of the input is a cached, stable prefix, this session pinning behavior is associated with 45 to 80 percent savings on input token cost. Source: How to Use Inference Router, DigitalOcean Documentation.

The practical rule: if you route, pin. Session pinning is not a separate optimization from prompt caching, it is the mechanism that keeps caching’s economics intact once more than one model is in play.

Routing without pinning can defeat caching entirely, even on a workload that would otherwise clear every gate in the decision framework above.

1. Does prompt caching change the model’s output?

No. Caching only affects how the input prefix is processed internally. The generated response is computed the same way whether the prefix was read from cache or processed fresh, and providers state the output is unaffected by whether a cache hit occurred.

2. Can I combine prompt caching with batch processing for extra savings?

On providers that offer both, yes, and the discounts have been reported as stacking. Confirm the current combined rate on your provider’s live pricing page, since stacked discounts are exactly the kind of figure that changes between pricing updates.

3. Does a shorter cache TTL always mean lower risk?

No, it means a narrower window to collect hits before the cache expires and you have to write again. A shorter TTL fits tightly-spaced, high-frequency traffic. A longer TTL fits traffic that is sparse but still repeats within an hour, at the cost of a higher write premium and a higher break-even point, as this piece’s formula shows directly.

4. Why does my measured hit rate not match the discount percentage a provider advertises?

Advertised percentages describe the discount on a successful cache read. Your effective savings across an entire workload depend on what fraction of your total requests actually hit, which is a property of your traffic, not of the provider’s pricing table. This is the entire reason this piece leads with a formula and a test harness instead of a percentage.

5. Do open-source models support prompt caching on DigitalOcean’s Serverless Inference?

Yes. Per DigitalOcean’s documentation, prompt caching for open-source models is in public preview, and models such as DeepSeek V3.2, the Qwen 3 series, and GLM cache automatically with no cache_control or prompt_cache_retention parameter. Caches are isolated per account, there is no 1,024-token minimum, and cached tokens are billed at a discounted read rate (roughly 46 to 80 percent off input, depending on the model) with no write premium. Not every open-source model carries a managed caching rate, though; check the Foundation Models list.

6. What if my open-source model does not have a managed caching rate?

Self-host it. vLLM, SGLang, and TensorRT-LLM all support automatic prefix caching within a replica, and vLLM enables it by default. The measured run in this piece confirms it working on a DigitalOcean GPU Droplet with no configuration, no write premium, and no minimum prefix length, including on a 155-token prompt, using Llama 3.3 70B, which is one of the models without a managed serverless caching rate. The tradeoff is that you operate the serving stack and pay for the GPU by the hour whether traffic arrives or not.

7. Do the hit rates measured here hold when I scale past one replica?

Not automatically. These numbers come from a single replica, where every request sees the same cache. Under round-robin load balancing across N replicas, an identical prefix has roughly a 1-in-N chance of landing on the replica holding its cache entry. Session-affinity routing restores most of the benefit, and DigitalOcean’s Advanced Prompt Caching at Scale post covers the multi-replica architectures in detail.

8. How do I check whether my requests are actually hitting the cache?

On DigitalOcean’s Serverless Inference, read the usage object on each response: a nonzero cache_read_input_tokens confirms a hit, and open-source models also report prompt_tokens_details.cached_tokens. On self-hosted vLLM, the per-request usage object does not report cached tokens, but the /metrics endpoint exposes vllm:prefix_cache_queries_total and vllm:prefix_cache_hits_total, and diffing those counters around serial requests attributes hits exactly, which is how the harness in this piece measures them.

Prompt caching is a workload decision. The three variables that decide it, prefix stability, session depth, and interarrival time, are all measurable, and this piece has given you the formula to check the first two against any provider’s published rates, a harness to measure the third against your own real traffic, and one completed run of that harness: on an H200 GPU Droplet serving Llama 3.3 70B, a stable prefix earned a 98.7 percent hit rate and a 92 percent TTFT reduction, and a single timestamp at the top of the prompt erased nearly all of it.

Caching addresses input token cost specifically. It has no effect on output token cost, which needs its own controls and is covered in a companion piece on output pricing. If you route requests across multiple models, session pinning through DigitalOcean’s Inference Router is the mechanism that keeps a cache warm across a routed session, and the two systems should be treated as coupled, not separate, exactly as this piece’s session pinning section describes.

Run the harness on your own traffic shape. Your measured hit rate, not any vendor’s headline percentage and not even the numbers measured here, is the one that should decide whether caching pencils out for you.

DigitalOcean documentation and engineering content

• Use Prompt Caching in Chat Completions and Responses API
• Advanced Prompt Caching at Scale (DigitalOcean Engineering blog)
• Inference Features reference
• Inference Pricing
• Foundation Models (which models support prompt caching)
• Serverless Inference API reference
• How to Use Inference Router
• Multi-Model API Cost Governance with the Inference Router
• DigitalOcean Serverless Inference: A Deep Dive
• Fine-Tuned LLMs on Serverless Architecture
• Where to Host Your Open-Source Model (Under 10B Parameters)
• What’s New on DigitalOcean’s Inference Engine

Provider primary sources

• Anthropic: Prompt caching documentation
• Anthropic: Pricing documentation
• OpenAI: Prompt Caching in the API (announcement)
• OpenAI: Prompt Caching 201 cookbook
• OpenAI: Prompt caching guide
• Google: Context caching, Gemini API documentation

Measured test environment

• vLLM: Automatic Prefix Caching documentation
• vLLM: Production Metrics (/metrics endpoint)
• Model: RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic on Hugging Face