p50 vs p99 Latency: Why Median Benchmarks Mislead AI Agent Workloads

If you are new to latency analysis, skim this once and refer back as needed.

The only formula this article uses is P = 1 − (1 − q)^n, the probability that at least one of n calls hits a tail event when each call has an independent chance q of doing so. It is explained step by step where it first appears, in the agent multiplier section.

• A clean median with a bad tail is not a rounding error. It is a fixed fraction of your traffic having a bad experience, every day, by construction. At 10,000 daily requests, a 1 in 100 tail event means 100 users hit it daily, regardless of how good the median looks. This is a mathematical identity, not a benchmark claim; I verified it by direct calculation and show the formula in the section below rather than asking you to take it on faith.
• For a single request, three different metrics matter for three different workloads. Time to first token for streaming chat UX, inter-token latency for voice and long streamed output, and total completion time for agents and batch jobs. Definitions per Artificial Analysis’s published methodology. Naming “the fastest provider” without naming which of these three you mean is not a complete claim.
• For agent pipelines, the math changes. A chain of sequential calls sums their latencies, and the probability that at least one call lands in the tail rises with chain length. I checked a common claim about how quickly this reaches near-certainty and found the numbers do not support it: at a 1 percent per-call tail rate, a 10-call chain gives 9.6 percent, not near-certainty, verified by direct calculation and shown in full in the section below. The corrected math is still a real, material risk, just not the risk as commonly overstated.
• A model that wins the headline benchmark can lose the task, and I measured this happening rather than simulating it. On DigitalOcean Serverless Inference on July 27, 2026, Llama 4 Maverick won the single-call time-to-first-token comparison against GPT-OSS 120B by 4 times (300 ms median versus 1,202 ms). On the measured 10-call agent chain, the ranking inverted: GPT-OSS 120B finished the task in 20.9 seconds at the median against Maverick’s 26.7 seconds, and Maverick’s worst measured chain took over 20 minutes because one call’s stream stalled mid-generation.
• A repeatable test protocol beats trusting any single published number, including the ones in this article. I ran the full protocol on three models and a bounded follow-up on a fourth, more than 1,500 recorded requests from a GPU droplet in the same cloud, and every number, every caveat, and the exact script are included below so you can rerun it against your own traffic shape. The complete evidence base, harness, raw per-request data, console logs, analysis code, and charts, is public in the serverless-inference-tail-latency-study GitHub repository I created for this research.

State this plainly before anything else. If your median time to first token is a clean 300 milliseconds but your p99 is 4 seconds, and you serve 10,000 requests a day, roughly 100 of those requests each day hit a wait that is more than ten times longer than the number on your dashboard. This is not noise. A percentile is a definition, not a measurement artifact: the p99 is the value that 1 percent of your traffic exceeds, by construction, every single day you run at that rate.

For a single request, this is already a reason to look past the median. For a session with more than one request, the exposure compounds. If each request in a session has an independent 1 percent chance of landing in the tail, a 20-request session has a chance of hitting at least one tail event of:

P(at least one tail event) = 1 − (1 − 0.01)^20 ≈ 18.2%

I verified this by direct calculation. Close to one in five sessions of that length will contain at least one bad request, even though any single request only has a 1 percent chance of being bad. This is the mechanism this entire piece is built on: independent small risks, repeated enough times, stop being small.

Why p50 and p99 diverge on inference specifically

The divergence has a small number of concrete, mechanistic causes, each covered in depth in the companion piece on inference silicon and architecture. Briefly:

Queueing under concurrency means your request’s start time depends on how many other requests are ahead of it, not just on how fast the model runs. Batching dynamics mean your request may wait for a batch window to fill before the GPU starts working on it at all. A long-prompt neighbor sharing your batch or your GPU can extend your prefill time even if your own prompt is short, since prefill is compute-bound and a large concurrent prompt consumes the compute your request needed. Cold starts on scale-to-zero infrastructure add a full model or adapter load time to whichever request happens to be first after an idle period. DigitalOcean’s own analysis of fine-tuned serverless deployments found cold starts spike time-to-first-token specifically, not the steady-state generation rate, and that periodic keep-alive requests are one practical mitigation. Source: Fine-Tuned LLMs on Serverless Architecture, DigitalOcean Community.

The tail is a small area under the curve. At real traffic volumes it is also a fixed, repeating count of real requests, not a rounding error.

When the median is the right number

High-volume, latency-insensitive batch workloads, evaluation suites, offline summarization, synthetic data generation, and content moderation at scale, care about throughput and cost per token, not about any individual request’s wait time, since no user is watching a spinner. DigitalOcean’s own batch inference explicitly isolates this traffic from real-time serving. Its documented limits state that batch scheduling does not degrade real-time inference p99 latency by more than 5 percent, which is the platform’s own commitment that these are different workload classes with different metrics. Source: DigitalOcean Inference Limits. If your workload looks like this, throughput and cost per successful request are the correct metrics, and the rest of this piece does not change your evaluation.

Latency is not one number. It decomposes into three measurements, and each one dominates a different workload.

1. Time to first token is the delay between sending a request and receiving the first output token. This is what a chat interface’s user stares at before anything appears on screen, and it is dominated by prefill, the compute-bound step of processing your entire input prompt before generation starts.

2. Inter-token latency, sometimes called time per output token, is the gap between each subsequent token once generation is underway. This is what determines whether a long streamed response feels smooth or halting, and it matters most for voice interfaces and long-form streamed answers, where the reader or listener is consuming tokens continuously rather than waiting once.

3. Total completion time is TTFT plus the full generation time, and it is the only metric that matters when nobody is watching the stream, which is exactly the case for a batch job or an intermediate call inside an agent pipeline. Source for these definitions: Artificial Analysis Language Model Benchmarking Methodology.

Three different clocks, three different workloads. A provider optimized for one is not automatically strong on the other two.

There is no single fastest provider

Asking which provider has the fastest TTFT is an underspecified question until you also state the concurrency, the prompt length, and which output length you are measuring against, because rankings reorder across these conditions.

This is not a hedge. It is directly checkable against Artificial Analysis’s own published provider comparisons, which explicitly separate speed by input token count and update their default benchmarking workload when they judge it unrepresentative. Their own note on one comparison page states plainly that the “default performance benchmarking workload has updated to 10k input tokens to better reflect production use cases,” which is Artificial Analysis admitting their own prior default was not representative of real traffic.

The spread across providers serving the identical model is large enough to make “fastest provider” a meaningless phrase without a condition attached. On GLM-5.2, Artificial Analysis reports a 963 percent difference between the fastest and slowest provider’s output speed. On gpt-oss-120b, the reported spread is 4,149 percent, with Cerebras at 1,677.5 tokens per second at the top of the range. Sources: Artificial Analysis, GLM-5.2 provider comparison and Artificial Analysis, gpt-oss-120b provider comparison. I want to be precise about what I derived versus what was stated directly: Artificial Analysis states the percentage spread and the fastest provider’s speed directly. Back-solving from that percentage implies a slowest provider near 45 tokens per second for GLM-5.2 and near 39 tokens per second for gpt-oss-120b. Those two implied figures are my own arithmetic from Artificial Analysis’s stated spread, not numbers Artificial Analysis published as a standalone figure, and you should treat them as an estimate rather than a confirmed data point.

The model is identical in both cases. The provider is not, and the gap between the best and worst provider dwarfs any difference you would find between competing models.

The streaming illusion

A provider can win on TTFT and lose on the metric that actually determines whether your user finished reading before they gave up. A fast first token with slow inter-token latency feels responsive for the first instant and then drags, which is a bad profile for a long streamed answer but is close to irrelevant for an agent, since an agent’s orchestration layer is not watching tokens arrive, it is waiting for the complete response before deciding its next action. This is why a single “fastest provider” answer does not exist independent of your workload. The correct question names the metric, the concurrency, and the token lengths, not just a provider name.

This is the section where the math has to be exactly right, so I am going to show my work rather than assert a conclusion.

An agent task rarely completes in one model call. A typical agentic workflow, tool call, reasoning step, tool call, synthesis, runs 5 to 20 sequential model calls to finish one task. Task latency is not any single call’s latency. It is the sum of every call in the chain, because each call waits on the previous one to determine its next input. This reframes the entire question: the metric that predicts your user’s experience is not any individual call’s p50 or p99, it is the distribution of that sum.

The corrected claim on tail probability across a chain

Here is a claim about this effect that I have seen stated, and want to check rather than repeat: that a 10-call chain against a per-call p99 tail rate makes a tail event a near-certainty for the task. I checked this directly, and the numbers do not support it as stated.

Using the same formula as the single-session case above, with q as the per-call probability of a tail event and n as the number of calls in the chain:

P(at least one tail event in the chain) = 1 − (1 − q)^n

At a 1 percent per-call tail rate, the rate implied by a p99 threshold, a 10-call chain gives:

1 − (0.99)^10 ≈ 9.6%

That is a real, material risk, roughly ten times higher than any single call’s own 1 percent rate. It is not “near-certainty” by any reasonable definition of that phrase. I computed the chain length that would actually be needed to make a 1 percent per-call event more than 95 percent likely to occur at least once: approximately 298 calls.

The honest version of this claim: agent chains do not make p99 tail events near-certain at realistic lengths. They make them several times more likely than a single call’s rate suggests, and at a p95 threshold, the risk crosses 50 percent by around 14 calls, which is squarely inside a typical agent’s call count. That is still the point worth making. It is a smaller, more precise point than “near-certainty,” and it is the one the math actually supports.

Longer chains raise the odds of hitting a tail event somewhere in the sequence, but the rate at which they do depends heavily on which percentile you are calling a tail event. Neither line reaches the near-certainty some framings of this argument claim at realistic agent chain lengths.

The demonstration that matters: the benchmark winner loses the task, measured

The more useful and more original point is not about the probability of a single bad call. It is about what happens to the total time across a chain when the model that wins the single-call benchmark is not the model that finishes the task fastest.

I ran the full test protocol described later in this article against real models on DigitalOcean Serverless Inference.

Two of the models measured make the point cleanly. On the benchmark-headline metric, median time to first token at concurrency 1, Llama 4 Maverick beat GPT-OSS 120B by 4 times: 300 milliseconds against 1,202 milliseconds. If you were choosing a model from a TTFT leaderboard, Maverick wins and it is not close. Then both models ran the same measured agent workload: 30 independent chains of 10 sequential calls each, every call’s output feeding the next call’s input.

The ranking inverted. Maverick’s first token arrives fast, but its generation speed during this window was 21 tokens per second against GPT-OSS 120B’s 86, so its total completion time per call was longer, and across ten sequential calls the model that “felt” 4 times faster finished the task 28 percent slower at the median and 84 percent slower at p95. TTFT is the metric a leaderboard shows you. Task completion time is the metric your agent’s user waits on. They picked opposite winners on the same day, on the same platform, under the same test.

Same two models, same platform, same day. The left chart is what a benchmark headline reports. The right chart is what your agent’s user actually waits for. Measured on DigitalOcean Serverless Inference, July 27, 2026.

How to read this chart: each panel asks the same question, “which model is faster?”, using a different clock. The left panel times only the wait for the first token of a single reply, in milliseconds, and the green bar (Maverick) is clearly shorter. The right panel times a complete 10-step agent task, in seconds, and now the green bar is clearly taller.

The worst-measured-chain row deserves its own sentence, because it is the tail argument of this entire piece compressed into one event. One of Maverick’s 300 chain calls delivered its first token in 321 milliseconds, a genuinely good TTFT, and then the stream stalled mid-generation and did not complete for 19.8 minutes. Every headline metric would have scored that request favorably at the moment it started. The chain containing it took 20.4 minutes instead of 27 seconds. That is 1 call in 300, which is a 0.3 percent event rate, and the formula from the section above tells you what that does across chains: at 10 calls per chain, roughly 1 task in 30 contains one. That is not a simulated number. It is the measured reason timeouts and retries exist.

Task completion time compounds with chain length. Lines are resampled from each model’s 75 measured single-call latencies; stars are the directly measured 10-call chain medians. Where the two disagree, trust the stars: real chains carry growing prompt history and live load variation that resampling from a fixed sample cannot capture.

How to read this chart: moving right means your agent makes more sequential calls per task; moving up means the whole task takes longer. Each color is one model, the solid line is the typical (median) task, and the dashed line above it is the unlucky (p99) task. The vertical axis is logarithmic, so each gridline step means roughly multiplying the wait, not adding to it. Two things to take away: the gap between the colors is enormous compared to the gap between any one model’s median and its own tail, so model and serving choice dominate; and the stars, which are real measured 10-call chains rather than projections, confirm the ordering the lines predict.

Tail events do not just slow tasks, they multiply spend

A tail event inside an agent chain has a second cost beyond the delay itself. Orchestration layers commonly implement a timeout, and a request that exceeds it triggers a retry. A retried call means you pay for the tokens of the original attempt and the tokens of the retry, and you absorb the delay of both. Tail latency inside an agent pipeline converts directly into duplicate spend, not just a slower task.

The stalled stream in the measured data above is exactly the request a production timeout exists to catch: a client that gave up at 30 seconds and retried would have paid for the duplicate call and still finished roughly 19 minutes sooner than one that waited. The mechanics of routing retries to a faster fallback model rather than retrying the same slow path.

The short version is that DigitalOcean’s Inference Router can fail over to an alternate model automatically rather than resending an identical request into the same queue. The same routing layer is also the cost lever this measured data points at: the chain results above show that most calls in an agent pipeline do not need the most expensive model on the endpoint, so matching each request to the cheapest model that meets its latency and quality bar, and reserving heavier models for the calls that genuinely need them, is how you keep both the latency budget and the per-token bill under control at once.

Use these seven questions to read any published latency claim critically.

• What percentile is reported?
  A median-only claim tells you about the typical request and nothing about the tail.

• What concurrency was the test run at?
  A number measured at concurrency one describes a best case that queueing under real load will not preserve.

• What prompt and output lengths were used?
  TTFT scales with prompt length because prefill is compute-bound, and total time scales with output length.

• Was the instance warm or cold?
  A cold-start-inclusive benchmark and a warm-instance-only benchmark answer different questions.

• What time window and duration was measured?
  A single burst of requests during off-peak hours can look nothing like sustained production traffic; DigitalOcean’s own consistency testing found that benchmarks run during business hours can look artificially good because other users have kept a model warm.

• Was streaming or non-streaming measurement used?
  TTFT is undefined without streaming.

• Was the measurement taken from a location that includes or excludes real network path?
  A benchmark run from infrastructure adjacent to the provider will understate the latency a real client experiences.

A latency claim missing any one of these seven details is a claim you cannot act on yet. It is not necessarily a false claim, just an incomplete one.

Applying the checklist without the cynicism

A 2026 measurement-bias paper from researchers at Google goes further and identifies a specific, technical failure mode in how many benchmarking clients are built: single-process, asyncio-driven load generators can hit their own client-side queueing bottleneck under high concurrency, which the authors model as an M/G/1 queue, and this client-side bottleneck can inflate the very TTFT and per-token latency numbers the benchmark is trying to measure. Source: Identifying and Mitigating Systemic Measurement Bias in Production LLM Inference Benchmarks, arXiv. The practical implication for you as a reader: even a benchmark run with real intentions can be measuring its own client as much as the server it targets, which is one more reason a documented, reproducible methodology is worth more than a headline number.

DigitalOcean’s own community content has already run this critique on itself. Its analysis of provider consistency for the same model reports a coefficient of variation, standard deviation divided by mean, as low as 21 percent on a well-supported provider and as high as 710 percent for the identical model on a different provider, a 34 times difference in consistency for zero difference in model. Source: Why Serverless Inference Consistency Varies on the Same Model, DigitalOcean Community.

Source: Why Serverless Inference Consistency Varies on the Same Model, DigitalOcean Community. That piece defines a coefficient of variation above 100 percent as the signature of cold starts and queueing variance, and above 300 percent as not production-ready for latency-sensitive work.

Predictable latency, defined properly

“Predictable latency” does not mean a low median. It means low variance around whatever the median is, and the cleanest single number for this is the ratio of your p99 to your p50. A provider with a p50 of 300 milliseconds and a p99 of 350 milliseconds is more predictable than a provider with a p50 of 200 milliseconds and a p99 of 2,000 milliseconds, even though the second provider’s typical case is faster. DigitalOcean’s own published analysis of its serverless behavior for two of its own models found typical and worst-case first-token times within a few hundred milliseconds of each other, with one model moving only from 0.29 to 0.35 seconds between typical and worst case. Source: Metrics that Matter with Serverless Inference, DigitalOcean Community. A p99:p50 ratio close to 1 is what that finding describes in the vocabulary this article uses; the coefficient of variation from the consistency article above is measuring the closely related quantity of variance around the mean rather than the ratio between two named percentiles, and either one is a legitimate way to quantify the same underlying property.

Where an architecture is built specifically to make this ratio close to 1 by design rather than by luck, the deterministic scheduling approach used by Groq’s LPU is the clearest example, and it is covered in depth in the related article on inference silicon.

Tail latency is also not an academic metric to the companies that live with it. Among DigitalOcean’s published customer results, Hippocratic AI, which runs safety-critical healthcare agents, reports 2 times higher production throughput and a 40 percent lower p99 latency across more than 20 million patient interactions on the platform. Note what the metric in that sentence is: p99, not median. Teams whose products break when the tail misbehaves negotiate and measure on the tail, which is worth remembering the next time a benchmark hands you a median and calls it speed.

This section describes the test design, then reports the results of actually running it. I executed this test on July 27, 2026, against DigitalOcean Serverless Inference, from a GPU Droplet (gpu-h200x1-141gb) in DigitalOcean’s NYC2 region, using the exact script printed below for the full protocol runs. Everything the run produced, the harness, the raw per-request JSON for all 1,590 recorded requests, the console logs, the analysis code, and the charts, has been added to this public Github repository: github.com/anishsingh20/serverless-inference-tail-latency-study.

Test design

Run against a single model, held constant, so the only variable under test is traffic shape and concurrency:

• Single-call baseline, three concurrency levels. At least 75 sequential requests per cell at concurrency 1, then repeat the same request pattern at concurrency 5 and concurrency 20, to observe how queueing changes the distribution as load rises. Seventy-five requests is the threshold DigitalOcean’s own consistency testing uses to reliably surface a cold start; fewer requests risk missing the tail entirely. Source for this specific threshold and the peak versus off-peak timing recommendation below: Why Serverless Inference Consistency Varies on the Same Model, DigitalOcean Community.
• Two time windows. Run the full concurrency sweep once during business hours and once overnight or on a weekend, since cold-start behavior is worst when background traffic to a model is lowest and a daytime-only test can look artificially good.
• A chained multi-call test. Construct a 10-call sequential chain that mimics an agent loop, each call’s output feeding the next call’s input, and run at least 30 independent chains through it. Record the total completion time for each full chain, not just each call, and compute the chain-level p50, p95, and p99 directly from those 30 totals, rather than trying to reconstruct them from the per-call numbers.

What to measure and compute

For every cell: median TTFT, p95 TTFT, p99 TTFT, and the p99:p50 ratio. For the chained test specifically: the distribution of total chain completion time, not any individual call’s distribution, since that total is the number that predicts what your user or your downstream system actually waits for.

Two complementary tests. The concurrency sweep shows how queueing degrades a single call under load. The chained test shows what a realistic agent workload actually experiences end to end.

Setup checklist to run this against DigitalOcean’s Serverless Inference

Every step here is the step I actually performed for the results below, documented against DigitalOcean’s own current pages so you can verify each one as you go.

1. Create a model access key. In the DigitalOcean Cloud Console, go to the Inference section and create a model access key. You can scope it to a specific model or leave it unscoped. See Manage Model Access Keys.
2. Confirm your account has a positive prepaid balance. Serverless Inference is prepaid and billed per token, and requests fail if your balance is at zero. See Manage Serverless Inference Prepayment and the Serverless Inference pricing page for current per-model token rates.
3. Pick a model and confirm its exact API model ID. Retrieve the ID with a GET request to https://inference.do-ai.run/v1/models using your model access key, or from the Model Catalog in the Cloud Console, and use whatever string that call returns. For the run below, that call returned the IDs mistral-3-14B, openai-gpt-oss-120b, llama-4-maverick, and llama3.3-70b-instruct, and those exact strings were used. See Retrieve Available Models.
4. Install the one dependency the script needs. The script below uses only the Python standard library, so there is nothing to install beyond Python 3.
5. Export your key as an environment variable and run the script once during business hours and once overnight or on a weekend, saving each run to a different output file. You can copy the harness from the code block below, or clone it together with the raw data and analysis code from this github repository.
6. Compare the resulting JSON against this article’s tables, and note the model, region, and dates you tested. The numbers below are one window on one day; yours are the ones that matter for your workload.

The harness

#!/usr/bin/env python3
"""
Latency benchmark for DigitalOcean Serverless Inference.

Measures, per request:
  - TTFT (time to first streamed token), timestamped client-side
  - total completion time
  - prompt and completion token counts, from the usage object

Workloads:
  1. baseline_sweep  - N requests at concurrency 1, 5, and 20
  2. chained_session - M independent chains of `chain_length` sequential
                        calls, each call's output feeding the next call's
                        input, with total chain time recorded per chain

Concurrency is implemented with a real OS-thread pool (ThreadPoolExecutor),
not a single-process asyncio loop. This is a deliberate choice: a 2026
measurement-bias paper (Chandrasekar and Kramberger, arXiv, cited in this
article) shows that single-process asyncio benchmarking clients can hit
their own client-side queueing bottleneck under concurrency, which
inflates the very TTFT numbers you are trying to measure. Real OS threads
release the GIL during blocking network I/O, which avoids that specific
failure mode at the modest concurrency levels used here.

Run:
  export DO_MODEL_ACCESS_KEY=your_key_here
  python3 do_latency_bench.py --model <model-id> --out results_daytime.json
  python3 do_latency_bench.py --model <model-id> --out results_overnight.json
"""

import argparse
import concurrent.futures
import json
import os
import time
import urllib.request


def percentile(sorted_vals, pct):
    """Linear-interpolation percentile, no external dependency required."""
    if not sorted_vals:
        return None
    k = (len(sorted_vals) - 1) * (pct / 100)
    f = int(k)
    c = min(f + 1, len(sorted_vals) - 1)
    if f == c:
        return sorted_vals[f]
    return sorted_vals[f] * (c - k) + sorted_vals[c] * (k - f)


def summarize(ttfts_ms):
    s = sorted(t for t in ttfts_ms if t is not None)
    if not s:
        return {"n": 0}
    p50 = percentile(s, 50)
    p95 = percentile(s, 95)
    p99 = percentile(s, 99)
    return {
        "n": len(s),
        "p50_ms": round(p50, 1),
        "p95_ms": round(p95, 1),
        "p99_ms": round(p99, 1),
        "p99_to_p50_ratio": round(p99 / p50, 2) if p50 else None,
    }


def streamed_request(base_url, api_key, model, messages, max_tokens=64):
    """Send one streaming chat completion. Return TTFT and total time in ms."""
    payload = {
        "model": model,
        "messages": messages,
        "max_tokens": max_tokens,
        "temperature": 0,
        "stream": True,
        "stream_options": {"include_usage": True},
    }
    req = urllib.request.Request(
        base_url + "/chat/completions",
        data=json.dumps(payload).encode(),
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {api_key}",
        },
    )
    start = time.perf_counter()
    ttft = None
    content_parts = []
    usage = {}
    try:
        with urllib.request.urlopen(req, timeout=120) as r:
            for raw in r:
                line = raw.decode(errors="ignore").strip()
                if not line.startswith("data:"):
                    continue
                body = line[5:].strip()
                if body == "[DONE]":
                    break
                chunk = json.loads(body)
                if chunk.get("usage"):
                    usage = chunk["usage"]
                for choice in chunk.get("choices", []):
                    delta = choice.get("delta", {})
                    if delta.get("content"):
                        if ttft is None:
                            ttft = (time.perf_counter() - start) * 1000
                        content_parts.append(delta["content"])
    except Exception as e:
        return {"error": str(e), "ttft_ms": None, "total_ms": None}
    total = (time.perf_counter() - start) * 1000
    return {
        "ttft_ms": round(ttft, 1) if ttft is not None else None,
        "total_ms": round(total, 1),
        "prompt_tokens": usage.get("prompt_tokens"),
        "completion_tokens": usage.get("completion_tokens"),
        "content": "".join(content_parts),
    }


def baseline_sweep(base_url, api_key, model, n_requests, concurrency_levels, prompt):
    results = {}
    messages = [{"role": "user", "content": prompt}]
    for c in concurrency_levels:
        print(f"[baseline] concurrency={c}, {n_requests} requests", flush=True)
        records = []
        with concurrent.futures.ThreadPoolExecutor(max_workers=c) as pool:
            futures = [
                pool.submit(streamed_request, base_url, api_key, model, messages)
                for _ in range(n_requests)
            ]
            for f in concurrent.futures.as_completed(futures):
                records.append(f.result())
        ttfts = [r.get("ttft_ms") for r in records]
        results[f"concurrency_{c}"] = {
            "summary": summarize(ttfts),
            "records": records,
        }
    return results


def chained_session(base_url, api_key, model, n_chains, chain_length, prompt_prefix):
    chains = []
    for i in range(n_chains):
        history = [{"role": "user", "content": f"{prompt_prefix} Chain {i}, step 0."}]
        start = time.perf_counter()
        calls = []
        for step in range(chain_length):
            r = streamed_request(base_url, api_key, model, history)
            calls.append(r)
            history.append({"role": "assistant", "content": r.get("content") or "ok"})
            history.append(
                {"role": "user", "content": f"Chain {i}, step {step + 1}. Continue."}
            )
        total_chain_ms = (time.perf_counter() - start) * 1000
        chains.append({"chain_index": i, "total_chain_ms": round(total_chain_ms, 1), "calls": calls})
        print(f"[chained] chain {i} done: {total_chain_ms:.0f} ms total", flush=True)
    chain_totals = sorted(c["total_chain_ms"] for c in chains)
    return {
        "chain_summary": {
            "n_chains": len(chain_totals),
            "p50_ms": round(percentile(chain_totals, 50), 1),
            "p95_ms": round(percentile(chain_totals, 95), 1),
            "p99_ms": round(percentile(chain_totals, 99), 1),
        },
        "chains": chains,
    }


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--base-url", default="https://inference.do-ai.run/v1")
    ap.add_argument("--model", required=True, help="Exact model ID from GET /v1/models")
    ap.add_argument("--n-requests", type=int, default=75)
    ap.add_argument("--n-chains", type=int, default=30)
    ap.add_argument("--chain-length", type=int, default=10)
    ap.add_argument("--out", default="latency_bench_results.json")
    args = ap.parse_args()

    api_key = os.environ.get("DO_MODEL_ACCESS_KEY")
    if not api_key:
        raise SystemExit("Set DO_MODEL_ACCESS_KEY before running.")

    results = {
        "model": args.model,
        "timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
    }

    print("== Baseline sweep ==", flush=True)
    results["baseline_sweep"] = baseline_sweep(
        args.base_url,
        api_key,
        args.model,
        args.n_requests,
        concurrency_levels=[1, 5, 20],
        prompt="Reply with a two-sentence summary of why caching matters for LLM inference.",
    )

    print("== Chained session ==", flush=True)
    results["chained_session"] = chained_session(
        args.base_url,
        api_key,
        args.model,
        args.n_chains,
        args.chain_length,
        prompt_prefix="You are debugging a production incident.",
    )

    with open(args.out, "w") as f:
        json.dump(results, f, indent=2)
    print(f"Wrote {args.out}", flush=True)


if __name__ == "__main__":
    main()

Run it exactly as shown in the docstring, once in each time window. The script prints progress as it runs and writes every raw record to the output file.

What was actually run, and under what conditions

Everything below comes from one execution of this protocol, and these are the conditions, stated before the numbers so you can weigh them:

• Client: a single GPU Droplet (gpu-h200x1-141gb, 24 vCPUs) in DigitalOcean’s NYC2 region. The network path is cloud-internal and short, which means these numbers could slightly understate what a client on the public internet would see. But the flip side is that this is also the realistic case for one class of reader: if your application already runs on DigitalOcean next to the inference endpoint, the short path measured here is the path your production traffic actually takes, with no cross-cloud hop or egress in between. The Droplet’s GPU is irrelevant to the test; it was simply the machine available.
• Endpoint and workload: https://inference.do-ai.run/v1/chat/completions, streaming enabled, temperature 0, max_tokens 64, the same short prompt for every baseline request, exactly as the script above sends it.
• Volume: the full protocol per model is 225 baseline requests (75 each at concurrency 1, 5, and 20) plus 30 independent 10-call chains, or 525 requests per model. Three models completed it: Mistral 3 14B, GPT-OSS 120B, and Llama 4 Maverick, for 1,575 recorded requests, with zero failed requests.
• Time window: I performend the test only for one window, 08:39 to 10:53 UTC on Monday, July 27, 2026, which is 4:39 to 6:53 a.m. US Eastern, a low-traffic period. The protocol calls for a second, contrasting window, and this run does not have one; treat every number here as a single-window measurement.
• One measurement caveat: the harness defines TTFT as time to first content token. A small number of GPT-OSS 120B responses (3 of 75 at concurrency 1, 10 of 75 at concurrency 5, 1 of 75 at concurrency 20) spent their entire 64-token budget on internal reasoning tokens and produced no content, so they have a total time but no TTFT, and the TTFT rows for that model are computed on the remaining requests.

The measured results

Time to first token, by concurrency level, 75 requests per cell:

Total completion time for a single call at concurrency 1, with the consistency metrics this piece has been arguing for:

The measured 10-call agent chains, 30 chains per model, total task time per chain:

Excluding Maverick’s one stalled-stream chain, described in the agent multiplier section above, its remaining 29 chains still measure p50 26.6 s, p95 47.2 s, worst 57.7 s: slower and wider than GPT-OSS 120B at every percentile even with the catastrophic event removed.

Each line is 75 real requests. The shape, not the starting point, is the story: Maverick starts fast and spreads wide; GPT-OSS starts slow and stays put.

How to read this chart: Pick any point on a line, and it says “this percentage of requests (vertical axis) got their first token within this much time (horizontal axis).” A line that shoots up almost vertically means a predictable model: nearly every request waits about the same amount. A line that rises fast and then bends into a long, shallow slope to the right means a model with a tail: most requests are quick, but the last few percent wait far longer. That bend is exactly where the p95 and p99 live, and it is invisible if all you publish is the point where the line crosses 50 percent.

In this chart, Mistral 3 14B “wins” for both typical and worst-case latency: it not only has the lowest median time to first token, but it also keeps nearly all requests tightly clustered together—even at the tail. Llama 4 Maverick comes close at the median, but its tail is much wider, meaning a few requests took much longer. GPT-OSS 120B starts slower but maintains a short and consistent tail. Mistral’s combination of fast start and tight spread makes it the clear winner when you care about both typical and worst-case latency.

Medians barely move as concurrency rises from 1 to 20. Tails move more, which is exactly why a concurrency-1 median is the least informative number a benchmark can report.

How to read this chart: each panel is one model, left to right along the bottom is how many requests were in flight at once, and the three lines in each panel are the typical request (p50, solid), the slow request (p95, dashed), and the very slow request (p99, dotted). A flat solid line means the typical experience holds up as load rises. What to watch is the vertical spread between the solid line and the dotted line: when that spread widens as you move right, as it does for GPT-OSS 120B at concurrency 20, queueing is starting to stretch the tail even though the median still looks untouched. This is the mechanism by which a provider’s benchmark number and your production experience quietly part ways.

Which model won and why: Mistral 3 14B, and it is not close. It was the fastest to first token at every percentile and every concurrency level, with a median of roughly 172 to 182 ms whether 1 or 20 requests were in flight, and a p99 that never exceeded 628 ms; even its worst measured percentile was roughly half of GPT-OSS 120B’s best median. Llama 4 Maverick came second with fast medians (214 to 320 ms) but a p99 four to five times higher, around 1.4 to 1.6 seconds. GPT-OSS 120B was last on both counts: the slowest to start (medians of 1.1 to 1.3 seconds) and the only model whose tail visibly degraded under load, with p95 stretching from 2.0 to 3.1 seconds and p99 from 2.1 to 3.6 seconds as concurrency rose from 1 to 20.

The likely reason is mostly size and what happens before the first token: Mistral 3 14B is by far the smallest model here, so the prefill work each request needs before it can emit anything is cheap, and during this window the shared pool absorbed 20 concurrent requests without measurable queueing. GPT-OSS 120B is roughly an order of magnitude larger and, in this deployment, spends part of its budget on internal reasoning before the first content token appears, so its TTFT starts high and is the first to feel queueing pressure. The caveat from the rest of this article still applies, though: winning time to first token is winning the streaming-chat metric. On total completion time and the measured 10-call chain, Mistral 3 14B did also finish first, but GPT-OSS 120B beat Llama 4 Maverick despite losing to it on every panel of this chart.

The fourth model is its own finding

I intended to run the identical full protocol on llama3.3-70b-instruct. I ran 15 sequential requests using the same streamed_request function from the harness above, wrapped in a watchdog that enforces a hard 150-second wall-clock cap per request and writes each result to disk the moment it completes, so a stalled stream can cost at most 150 seconds instead of stalling the whole run.

All 15 completed within the cap. Median TTFT was 2.1 seconds. Median total time for 64 tokens was 43.5 seconds. Two of the 15 requests waited 37 and 68 seconds respectively for their first token, the signature of queueing or a cold path, while the other 13 got their first token within 1.6 to 3.4 seconds. At these speeds, a 10-call agent chain on this model would take over 7 minutes at the median, so the chained test was not run, and this model’s numbers are from 15 requests rather than 525 and should be read with that weight.

This is the sharpest argument in the article and it was not planned: the most recognizable model name in the catalog was, during this specific window, one to two orders of magnitude slower per completed request than three less famous alternatives on the same endpoint, and no headline TTFT figure would have warned you, because its median TTFT was a reasonable-looking 2.1 seconds. The number that told the truth was the one almost nobody publishes: total completion time and its spread.

Interpreting what you find

A fat right tail with an otherwise normal-looking body suggests queueing or batching contention: the typical request is fine, but a subset waits behind other traffic. Maverick’s measured TTFT is this pattern exactly: a 300 millisecond median with a p99 five times higher and a coefficient of variation of 75 percent, against GPT-OSS 120B’s 27 percent. A bimodal distribution, a cluster of fast requests and a separate cluster of slow ones with a gap between them, is the signature of cold starts or a congested path: some requests hit a warm replica and some do not, and there is close to nothing in between. Llama 3.3 70B’s bounded run showed this shape, with 13 requests getting a first token within 3.4 seconds and 2 requests waiting 37 and 68 seconds.

For the fat-tail pattern, the direct mitigation is a dedicated endpoint or reserved capacity, since the underlying cause is competition with other tenants’ traffic. On DigitalOcean specifically, serverless and dedicated inference are part of the same platform, and the move from shared to dedicated capacity is positioned as a capacity decision rather than a migration project, which matters here because the right time to make that decision is after a measurement like this one, not before. For the bimodal pattern, the direct mitigation is a warm pool or scheduled keep-alive traffic that prevents the replica count from dropping to zero in the first place. If you route across multiple models or providers, session pinning is what keeps a chain of calls on the same warm path rather than re-triggering a cold start partway through a task; this is covered directly in one of the related article I wrote on prompt caching and session pinning.

Evaluate on p95 and p99 when your workload is interactive, session-based, or bound by a latency SLA. A single user-facing request or a short session is exactly the case where the tail is what a real person feels.

Evaluate on task completion time when your workload is an agent pipeline of three or more sequential calls. Derive your required per-call p99 from your task-time target and work backward, rather than picking a per-call target first and hoping the total comes out acceptable. If your task budget is 5 seconds across 10 calls, your real per-call budget is close to 500 milliseconds at the tail, not at the median, and the measured results earlier in this piece show why a headline single-call metric is the wrong number to check that budget against: the model that won measured TTFT by 4 times lost the measured 10-call task by 28 percent.

Evaluate on throughput and cost, and set tail latency aside, when your workload is offline batch or an asynchronous pipeline with no user or downstream system waiting on any single request’s completion.

If you route requests across models by cost versus latency, route your latency-critical paths by their measured p99 profile and route your cost-tolerant paths by price, a decision this piece states briefly since the concrete routing mechanics are covered in the Inference Router documentation.

Three workload shapes, three correct metrics. Using the wrong one does not just mislead you, it can point you at the wrong model entirely, as the measured TTFT-versus-task-time inversion above shows directly.

The tail-compounding formula and every worked number derived from it are my own direct calculation, checked with a simple script, and you can rerun the same formula against your own per-call tail rate and chain length. The measured latency numbers in this piece come from one execution of the published test protocol, run by the author against DigitalOcean Serverless Inference on July 27, 2026, from a GPU droplet in the same cloud, using the exact script printed in the harness section; the raw per-request JSON was retained, every table was computed from it, and the complete evidence base is published in the GitHub repository so anyone can recompute every table and chart from the raw records. Those numbers are real, and they are also one time window, one client location, and one day. They describe what that endpoint did during that window, not what it will do during yours, which is the entire reason the script is included. The resampled chain-length curves are explicitly labeled as bootstrap extensions of measured single-call data and are shown alongside directly measured chains; where the two disagree, the measured chains are the ground truth. The Artificial Analysis provider-spread figures and the DigitalOcean consistency and metrics data are real, cited, and linked directly; the two “implied slowest provider” figures in the TTFT-ranking section are my own arithmetic from Artificial Analysis’s stated percentage spread, not a number Artificial Analysis published directly, and I have flagged that distinction at the point it appears. If you run the protocol and your results differ from anything reported here, including on DigitalOcean’s own serverless inference, that correction is worth more than this article, and I would rather you publish it than assume this piece already covered your case.

1. Is a low median latency ever a bad sign?

Not on its own. A low median with a low p99:p50 ratio is the best possible outcome. A low median with a high p99:p50 ratio means the typical case is fast but a meaningful fraction of your traffic is not, which the median alone will never show you.

2. Why does a 1 percent per-call tail rate matter if it sounds so small?

Because “small” is relative to how many times you take the risk. A single request has a 1 percent chance of hitting it. A 20-call agent chain has an 18 percent chance that at least one call in that chain hits it, a number this piece verified directly rather than assumed.

3. Does this mean I should always pick the provider with the tightest tail over the one with the best median?

Not automatically, and it depends entirely on your workload. For a single interactive request in a streaming chat interface, a low TTFT with an acceptable p95 can be the right call. For any workload that chains sequential calls into one task, this piece’s measured results show the TTFT winner losing the 10-call task by 28 percent at the median and 84 percent at p95, which is why the correct evaluation metric depends on your call count and what your user is actually waiting for, not on a fixed preference for one headline number.

4. Do these results mean GPT-OSS 120B is “better” than Llama 4 Maverick?

No, and this piece should not be quoted as saying so. The measurements are one time window on one day, with a fixed 64-token output cap and no evaluation of answer quality, which is usually the deciding factor between models. What the results do show is that the ranking depends on the metric: Maverick genuinely won TTFT and genuinely lost task completion time during this window. The durable takeaway is the method, not the leaderboard: measure the metric your workload actually feels, on your own traffic shape, before you choose.

The benchmark ecosystem reports what is easy to measure and flattering to publish. A clean median is easy to measure and always flattering. Production pain lives in the shape of the distribution for a single request, and for an agent, in the sum across every call in its chain. This article made two changes to how you should read a latency number. Look at the distribution, not the center, for any single interactive request. Look at the total task time, not any individual call’s percentile, for any workload that chains calls together, and derive your per-call requirement backward from your task-level target rather than the reverse.

Please run the test protocol in this article against your own traffic shape, at your own concurrency, on your own chain length. This is the only way to know what your own production experience will look like.

1. GitHub Repository to run the test protocol yourself

• serverless-inference-tail-latency-study on GitHub: the full evidence base for every measured number in this article — the measurement harness, orchestration and bounded-sweep scripts, raw per-request JSON for all 1,590 recorded requests, unedited console logs, the analysis script, and the four charts, under an MIT license.

2. DigitalOcean documentation and engineering content

• Prompt Caching and Cost Break-Even
• Inferentia2, TPU, Groq LPU, vs. GPU for LLM Serving
• Serverless Inference Metrics reference
• Inference Features, observability section
• Inference Limits
• Why Serverless Inference Consistency Varies on the Same Model
• Metrics that Matter with Serverless Inference
• DigitalOcean Inference Mode Comparison for Your Each Use Case
• Fine-Tuned LLMs on Serverless Architecture
• How to Use Inference Router
• DigitalOcean Serverless Inference: A Deep Dive
• Why is p99 Time to First Token (TTFT) High When Everything Else Looks Normal? Debugging Tail Latency in LLM Inference

3. External sources

• Artificial Analysis, Language Model Benchmarking Methodology
• Artificial Analysis, GLM-5.2 (max) provider comparison
• Artificial Analysis, gpt-oss-120b (high) provider comparison
• Ashok Chandrasekar and Jason Kramberger, “Identifying and Mitigating Systemic Measurement Bias in Production LLM Inference Benchmarks,” arXiv, 2026.