Process a Million Documents Overnight: Batch Inference End-to-End

To keep the numbers concrete, here is the project:

• 1,000,000 plain-text documents, averaging about 1,200 tokens each (roughly 900 words)
• For each document: assign one of eight categories and write a 3-4 sentence summary
• Deadline: results ready the next morning
• Model: GPT-5 mini through DigitalOcean’s serverless inference platform, which offers it at $0.25 per million input tokens and $2.00 per million output tokens at real-time rates. Batch requests are billed at up to half those rates, for reasons the billing section covers.

One design decision up front: classification and summarization happen in a single request per document, not two. Two separate calls would double your request count and re-send every document twice. Instead, the prompt asks the model to return one JSON object containing both the category and the summary. That halves the bill and simplifies the pipeline, at the cost of a slightly longer prompt.

With prompt instructions added, each request carries about 1,500 input tokens and produces about 200 output tokens. Across a million documents, that is 1.5 billion input tokens and 200 million output tokens.

Before writing any code, it is worth checking whether the real-time API could even finish in time.

DigitalOcean’s serverless inference rate limits at Tier 3 and Tier 4 are 600 requests per minute and roughly 800K to 2M tokens per minute (see the account tier table on the Inference Limits page). One million requests at 600 per minute takes about 27.8 hours, and that assumes zero retries and a script that never falls behind. The token limit is just as restrictive: 1.7 billion total tokens at 2M tokens per minute is about 14 hours of steady, perfectly timed requests.

You could engineer around this: request a tier increase, tune a rate limiter, checkpoint progress, shard the work. Teams do, and it is usually wasted effort, because the rate limits are not an obstacle here. They are a signal that a latency-optimized API is the wrong execution model for a throughput job.

Batch inference avoids all of this. Batch jobs use a separate quota (by default, you can submit up to 10 billion tokens per model per account) and run on isolated capacity at lower scheduling priority, so a running batch job does not consume your real-time quota or degrade latency for your production applications. You also do not write a rate limiter, a retry loop with backoff, or a checkpoint file. The platform retries transient errors (429, 408, 5xx) up to two times with exponential backoff on its own, and failed requests land in an error file instead of crashing anything (see the batch inference section of the Inference Features page).

The trade-off is speed. A batch job has a 24-hour completion window, and you have no control over when within that window your requests run. If any part of your workload needs an answer in seconds, that part stays on the real-time API. Everything else is a candidate for batch.

Three prerequisites, all one-time setup:

• A DigitalOcean account at Tier 3 or higher. Tier 1 and Tier 2 accounts do not have access to Anthropic models or OpenAI models (except the open-weight gpt-oss models), and batch inference only supports OpenAI and Anthropic commercial models with text prompts (see the Inference Limits page for tier model access and the batch inference limits for supported models).
• A positive prepaid balance. Serverless inference, including batch, is prepaid: usage is deducted from your balance, and if it hits $0 access is suspended. For this job, load enough to cover the estimate in the billing section plus margin.
• A model access key, created from the DigitalOcean control panel. Every API call below authenticates with it against the serverless inference base URL, https://inference.do-ai.run.

One constraint to plan around: each batch job uses a single model, and multi-model batch jobs are not supported (see batch inference limits). This is not the same as DigitalOcean’s Inference Router, which picks a best-fit model per request; that feature applies to real-time inference, not batch jobs. So if you want to send easy documents to a cheaper model and hard ones to a stronger model, that is two separate batch jobs, not one.

Batch inference has three limits that shape how you split a million documents:

• Maximum 50,000 requests per input file
• Maximum 200 MB per input file
• Maximum 10 billion tokens submitted per model per account (the default; you can request an increase)

At first glance the split looks easy: 1,000,000 ÷ 50,000 = 20 files. But each file must also stay under 200 MB, so work through the numbers in three steps.

Step 1: How big is one line? Each line holds one document, the prompt instructions, and a small JSON wrapper. One token is roughly four characters.

Step 2: Which limit fills up first?

The size cap fills up first: only about 31,500 documents fit in a file, well below the 50,000-request cap. In this project, file size decides the split.

Step 3: Choose the split, with room to spare.

Forty files means forty batch jobs, which sounds like a lot but changes almost nothing in the code, because the submission and polling logic is a loop either way.

Last, check the token quota. The job needs 1.5 billion input tokens plus about 200 million for output, 1.7 billion total. That is well under the 10 billion default, so all 40 jobs can be submitted at once.

Each line of a batch input file is one self-contained request. For OpenAI-provider jobs on DigitalOcean, the line follows the OpenAI Batch API shape: a custom_id, a method, a URL, and the request body you would have sent to the real-time endpoint (see the input file format in DigitalOcean’s batch inference guide).

Two details matter more than they look:

First, custom_id is the only key that links a result back to its source document. Results do not come back in input order. Use your real document ID, never an array index that means nothing after the list is re-sorted. Duplicate custom_id values within a file fail validation, so a stable unique ID solves both problems at once.

Second, cap the output length. The summary should be 3-4 sentences, so 500 tokens is plenty. Note the parameter names: GPT-5 models use max_completion_tokens on the chat completions endpoint, not the older max_tokens, and they do not accept a custom temperature. The cap also covers the model’s internal reasoning tokens, so set reasoning_effort to minimal for a simple task like this; it keeps reasoning tokens near zero and the output bill predictable. Without a cap, one overly long completion wastes output tokens, multiplied by however many documents trigger it.

The following script reads document records (ID plus text), builds the request lines, and rolls over to a new file every 25,000 requests:

import json

SYSTEM_PROMPT = (
    "You classify and summarize documents. Respond with a single JSON object: "
    '{"category": "<one of: billing, bug_report, feature_request, account, '
    'security, performance, documentation, other>", '
    '"summary": "<3-4 sentence summary>"} '
    "The category value must be exactly one of the eight listed strings, "
    "lowercase. Never invent another category; if unsure, use \"other\"."
)

CHUNK_SIZE = 25_000

def write_batch_files(documents, prefix="batch_input"):
    """documents yields (doc_id, text) tuples. Returns list of file paths."""
    paths, out, count, part = [], None, 0, 0
    for doc_id, text in documents:
        if count % CHUNK_SIZE == 0:
            if out:
                out.close()
            part += 1
            path = f"{prefix}_{part:03d}.jsonl"
            out = open(path, "w", encoding="utf-8")
            paths.append(path)
        line = {
            "custom_id": doc_id,          # your real document ID
            "method": "POST",
            "url": "/v1/chat/completions",
            "body": {
                "model": "gpt-5-mini",
                "messages": [
                    {"role": "system", "content": SYSTEM_PROMPT},
                    {"role": "user", "content": text},
                ],
                "max_completion_tokens": 500,
                "reasoning_effort": "minimal",
            },
        }
        out.write(json.dumps(line, ensure_ascii=False) + "\n")
        count += 1
    if out:
        out.close()
    return paths

Before uploading anything, validate locally. A broken line or a duplicate custom_id fails the whole file at the validation stage, and finding that out after upload wastes a cycle. A ten-line check that parses every line, verifies the required keys, and confirms custom_id uniqueness saves you that wasted cycle.

Submission is a three-step process per file, and the order matters. The full flow, with official examples in Python, JavaScript, and cURL, is documented in DigitalOcean’s batch inference guide.

Step one: request a file intent. POST /v1/batches/files with a file name ending in .jsonl returns a file_id and a presigned upload URL. The file_id is valid for up to 30 days and reusable across jobs; the upload URL expires in about 15 minutes, so upload promptly. If you miss the window, request a new intent.

Step two: PUT the raw JSONL bytes to the presigned URL. Use Content-Type: application/octet-stream or omit the header entirely. Presigned URLs are signature-sensitive, and a nonstandard content type like application/jsonl can break the signature match.

Step three: create the batch job with the file_id. This step performs a check against object storage and fails if the upload has not finished, so never reorder steps two and three. The create call takes the provider (openai or anthropic), the completion window (only 24h is currently accepted), an endpoint that must match the url on every line for OpenAI jobs (omit it for Anthropic jobs), and a request_id you generate.

That request_id is a safety key that prevents duplicate jobs, and for a 40-job overnight run it matters. If your submission script hits a network error and retries, the same request_id returns the existing job instead of creating a duplicate that would double-bill 25,000 documents. Build it from the file name rather than generating a random UUID each time, so rerunning the whole script is also safe:

import hashlib
import os
import uuid

import requests
from pydo import Client

client = Client(token=os.environ["DIGITALOCEAN_TOKEN"])

def submit_file(path):
    # 1. Reserve a file_id and presigned upload URL.
    intent = client.batches.files.create(file_name=os.path.basename(path))
    file_id, upload_url = intent["file_id"], intent["upload_url"]

    # 2. PUT the raw JSONL bytes. octet-stream keeps the signature valid.
    with open(path, "rb") as fh:
        put = requests.put(
            upload_url,
            data=fh,
            headers={"Content-Type": "application/octet-stream"},
            timeout=300,
        )
    put.raise_for_status()

    # 3. Create the job. request_id derived from the file name makes
    #    the whole script safe to rerun without duplicating jobs.
    request_id = str(uuid.UUID(
        hashlib.md5(f"doc-pipeline-2026-08:{path}".encode()).hexdigest()
    ))
    batch = client.batches.create(
        file_id=file_id,
        provider="openai",
        endpoint="/v1/chat/completions",
        completion_window="24h",
        request_id=request_id,
    )
    return batch["batch_id"]

batch_ids = {}
for path in sorted(paths):           # paths from write_batch_files()
    batch_ids[path] = submit_file(path)
    print(f"submitted {path} -> {batch_ids[path]}")

Persist the batch_ids mapping to disk (a JSON file is fine). It is the only state your pipeline needs to survive a restart.

Each job moves through a fixed set of states: validating (file structure, unique IDs, token counts), queued (waiting for capacity), in_progress, and then one of four final states. completed means every request was processed, even if some individual requests failed; those failures are in the error file, not reflected in the job status. failed means a systemic problem, usually total validation failure. expired means the 24-hour window ran out. cancelled is self-explanatory. Importantly, expired and cancelled are not total losses: everything that completed before the job ended is preserved, downloadable, and billed. Only unprocessed requests are dropped, and you are not charged for them.

Polling means checking the job’s status on a repeating schedule: your script asks the API for the current status, waits, and asks again until the job is done. The batch API does not send you a notification when a job finishes, so this loop is how you find out. There is no reason to check often; once a minute across all jobs is plenty for an overnight run:

import time

def wait_for_jobs(batch_ids, poll_seconds=60):
    pending = set(batch_ids.values())
    terminal = {"completed", "failed", "expired", "cancelled"}
    states = {}
    while pending:
        for bid in list(pending):
            b = client.batches.retrieve(bid)
            status = b["status"]
            counts = b.get("request_counts", {})
            print(f"{bid}  {status:12}  "
                  f"{counts.get('completed', 0)}/{counts.get('total', 0)}")
            if status in terminal:
                states[bid] = status
                pending.discard(bid)
        if pending:
            time.sleep(poll_seconds)
    return states

Run it, go to sleep. The request_counts field gives you per-job progress if you want a dashboard, but for most teams, reading the log in the morning is enough.

Failures happen at three levels, and each has a different fix.

Request-level failures are individual lines that could not be processed: a document that exceeds the model’s context window, a content policy rejection, a badly formatted prompt that slipped through validation. These do not fail the job. Each one is written to an error file with its custom_id and an error code:

{"custom_id": "doc-88213", "error": {"code": "context_length_exceeded", "message": "Request exceeded maximum context length."}}
{"custom_id": "doc-90142", "error": {"code": "content_policy_violation", "message": "Request was blocked by content moderation."}}

The error codes tell you what to do. context_length_exceeded documents need truncation or splitting before resubmission. Content policy rejections need a human look. Anything temporary was already retried twice by the platform before landing here, so simply resubmitting those IDs as one final small batch job is reasonable. If your input files are clean, the error file should be small; in our 200-completion evaluation over the real-time API, zero requests failed at the API level. The entries that do appear are mostly oversized documents you should have caught in validation.

Job-level failures are rarer. If a job expires with work remaining, you can create a continuation job that processes only the requests that failed or were never reached, without re-running (or re-paying for) completed work. The same duplicate protection from request_id covers the submission side: a create call retried after a network error returns the original batch job.

The billing rule is simple: you are charged only for completed requests. Expired, cancelled, or guardrail-blocked requests that never produced output cost nothing.

When a job reaches a terminal state, GET /v1/batches/{batch_id}/results returns presigned URLs for the output file and, if one exists, the error file. Two operational details here: the presigned URLs are short-lived, so download immediately after fetching them rather than storing the URLs for later, and output files are retained for up to 30 days after completion, after which they are permanently deleted. Downloading and archiving results to your own storage should be part of the pipeline, not an afterthought.

Each output line carries the custom_id, the full API response (including per-request token usage), and an error field that is null on success. The join back to your source documents is a dictionary lookup, and summing the usage fields as you go gives you an exact token count to check against your bill:

import json
import requests as http

CATEGORIES = {"billing", "bug_report", "feature_request", "account",
              "security", "performance", "documentation", "other"}

def collect_results(batch_ids, out_path="results.jsonl"):
    total_in = total_out = failures = 0
    with open(out_path, "w", encoding="utf-8") as out:
        for path, bid in batch_ids.items():
            links = client.batches.results.retrieve(bid)
            if not links.get("result_available"):
                print(f"{bid}: results not ready, poll again later")
                continue
            resp = http.get(links["output_file_url"], timeout=300)
            resp.raise_for_status()
            for line in resp.text.splitlines():
                rec = json.loads(line)
                if rec.get("error"):
                    failures += 1
                    continue
                usage = rec["response"]["usage"]
                total_in += usage["prompt_tokens"]
                total_out += usage["completion_tokens"]
                content = rec["response"]["choices"][0]["message"]["content"]
                try:
                    parsed = json.loads(content)
                except json.JSONDecodeError:
                    failures += 1   # model returned non-JSON; queue for retry
                    continue
                if parsed.get("category") not in CATEGORIES:
                    failures += 1   # valid JSON, invalid category value
                    continue
                out.write(json.dumps({
                    "doc_id": rec["custom_id"],
                    "category": parsed["category"],
                    "summary": parsed["summary"],
                }, ensure_ascii=False) + "\n")
    print(f"input tokens: {total_in:,}  output tokens: {total_out:,}  "
          f"failures: {failures:,}")

Note the second failure mode handled here: the request succeeded but the model returned something that is not valid JSON. With a strict system prompt this is rare (we measured zero in 200 evaluation completions), but across a million completions “rare” still happens, and the check costs nothing. Queue those custom_ids with the error-file IDs for the cleanup batch.

The third check, category not in CATEGORIES, exists because we ran this prompt structure against real documents (50 news articles, with an eight-value news-category list in place of the ticket categories) and found a failure the modeled pipeline missed: the model returns perfectly valid JSON with a category that is not on the list. In a 50-document evaluation over the real-time API, GPT-5 nano invented an out-of-list category 4 times in 50 and GPT-5 mini twice, producing labels like “science” and “humanitarian” that no downstream join would recognize. Valid JSON is not valid data; validate the values, not just the structure.

We then tightened the prompt (the “never invent another category” sentence above) and re-ran the same 50 documents. The result is worth knowing before you trust prompt fixes at scale: mini’s violations dropped from 2 to 0, while nano’s stayed at 4 out of 50. Prompt-level fixes are model-dependent; the code-level check is not. If you choose the cheaper model, plan for its measured violation rate (8% in our test) with a retry or normalization pass. Retrying 8% of requests adds about 8% to nano’s bill, which still leaves it roughly 5x cheaper than mini.

Batch tokens on DigitalOcean are billed at up to half the serverless (real-time) rates for OpenAI and Anthropic models (see the batch inference section of the Inference Pricing page). The lower rate is not a promotion; it is scheduling economics. Batch jobs run at lower priority and share off-peak GPU capacity (per the batch inference limits), so work that can wait 24 hours fills hardware that would otherwise sit idle between real-time peaks. A request that must answer in two seconds is more expensive to serve than one that can run at 4 AM, and the pricing reflects that.

Token usage is the only batch charge the pricing page lists; there are no separate fees for file upload, storage, job creation, or polling. At the full batch rate, GPT-5 mini costs $0.125 per million input tokens and $1.00 per million output tokens.

Here is the complete bill for the job as specified. The base rates are GPT-5 mini’s serverless prices ($0.25 input, $2.00 output per million tokens) from the Inference Pricing page, halved for batch; each cost is then tokens multiplied by rate:

That is about $0.0004 per document. The identical workload at real-time rates costs $375.00 for input plus $400.00 for output, or $775.00 total. The $387.50 difference is the price of urgency: what this job would pay for answers in seconds when the actual deadline is tomorrow morning. Most pipelines never ask that question, which is why most pipelines overpay. The useful framing is not “can we afford real-time” but “what is the deadline, really.”

Model choice moves this number more than anything else in the pipeline. The same job on other batch-eligible models, at full batch rates:

For straightforward classification, GPT-5 nano at $77.50 for the whole million is worth evaluating first. The right process is to run representative documents through each candidate at real-time rates, score the outputs side by side, and only then commit the million. We did exactly that for this article: 50 documents (news articles as stand-ins) through both GPT-5 nano and GPT-5 mini with this prompt structure, twice (once before and once after the prompt tightening described earlier), for under a dollar in total. What we measured:

• JSON validity was perfect: 200 completions across the two models and two runs, zero parse failures. The failure mode at this tier is not broken JSON.
• Category discipline was not. Nano returned an out-of-list category on 4 of 50 documents in both runs; mini did twice in the first run and zero times after the prompt fix. Nano’s rate did not improve with the stricter prompt.
• On summary quality, judged document by document, mini was clearly or somewhat better on 14 to 23 of 50 (depending on how strictly you score), the rest were ties, and nano was never better on a single document. The pattern was consistent: mini retains specifics that nano drops, such as names, dollar figures, and case details.

The verdict falls out of the numbers. For classification-dominant work at scale, nano plus a validation-and-retry pass is the rational choice at a fifth the price. When the summaries feed anything a human will read, pay for mini. Either way, an evaluation that cost less than a dollar settled a four-figure decision with measurements instead of instinct; skipping it is how teams end up paying Claude Haiku prices for GPT-5 nano work, or shipping a million thin summaries at any price.

Two billing caveats. Serverless inference is prepaid, so the balance must be loaded before the job runs, and batch pricing is stated as “up to” half the real-time rate, so confirm the effective rate for your model on the pricing page before you commit the workload.

The decision comes down to a few questions you can answer before writing any code.

Batch is the right tool when all of these are true:

• Nobody is waiting on an individual response. Results needed within 24 hours, but not within seconds.
• The workload is large enough that rate limits or cost matter. Below a few thousand requests, the real-time API is simpler and the savings are too small to matter.
• Requests are independent. Each line is self-contained; batch has no mechanism for one request to depend on another’s output. Multi-step chains need orchestration outside the batch API, typically one batch job per step.
• Text in, text out. Multimodal requests and image generation are not supported for batch on DigitalOcean.
• One model per job fits your routing. Mixed-model pipelines mean multiple jobs.

Stay on real-time when latency matters at all, when request volume is small, or when you need features batch does not support (streaming, or provider-specific features like extended thinking).

Self-hosting on dedicated GPUs is the third option, and the math is different rather than automatically better. DigitalOcean’s dedicated inference runs an H100 at $4.41 per hour and an 8x H100 node at $30.32 per hour. Self-hosting starts to win in three situations. First, open-source models: batch inference only supports OpenAI and Anthropic commercial models, so a nightly million-document job on Llama 3.3 or Qwen belongs on dedicated GPUs or serverless per-token open-source pricing instead. Second, steady, heavy use: if the overnight job runs every night and mostly fills the hardware, a dedicated endpoint at a few hundred dollars per night of GPU time can cost less than per-token pricing for large token volumes, and scale-to-zero means you stop paying when the queue is empty. Third, data control: when documents cannot leave infrastructure you control, per-token commercial APIs are off the table regardless of price. The costs self-hosting adds back are engineering time (serving stack, batching logic, monitoring, retries: everything the batch API just did for you) and the risk of idle GPUs. If your volume is uneven or the job is occasional, per-token batch pricing wins on total cost even when the raw GPU math looks close.

The pipeline above is about 150 lines of Python, and most of it is routine data handling rather than machine learning: split the input to respect file limits, use real document IDs as custom_ids, build request IDs from file names so reruns are safe, download results before the URLs and the 30-day retention run out, and check token usage against the bill. The platform handles the parts that are actually hard at this scale: retries, capacity scheduling, and isolation from your production traffic.

The habit worth building is to sort every LLM workload by one question: is anyone waiting for this answer? When a person is waiting, pay real-time rates for real-time behavior. When nobody is, and for backlogs, evaluations, tagging jobs, and reports nobody is, batch should be the default and real-time the exception you justify. Most teams have it backwards: they treat the interactive API as the only API and latency tolerance as something to ignore rather than a design input. On this job, that framing is worth $387.50 out of $775.00, and it required no cleverness, no infrastructure, and no model changes. The work was going to run overnight anyway; it should be priced that way.

DigitalOcean documentation

• Inference Pricing: all model rates used in this article (GPT-5 mini $0.25/$2.00, GPT-5 nano $0.05/$0.40, Claude Haiku 4.5 $1.00/$5.00 per 1M tokens), the “up to 50%” batch discount, dedicated GPU pricing (H100 $4.41/hour, 8x H100 $30.32/hour), prepaid billing, and the rule that only completed requests are charged.
• Inference Limits: account tier rate limits (600 requests and 800K-2M tokens per minute at Tiers 3-4), batch limits (50,000 requests and 200 MB per file, 10 billion tokens per model per account, 24-hour window), tier model access, and batch traffic isolation.
• Inference Features: automatic retries of transient errors, error files, continuation jobs, and duplicate-safe job creation.
• How to Use Batch Inference: the input file format, three-step upload flow, presigned URL lifetime (~15 minutes), file ID validity (up to 30 days), job states, results retrieval, output retention (up to 30 days), and cancellation billing.
• Use Serverless Inference, Manage Model Access Keys, and Manage Serverless Inference Prepayment: setup prerequisites.

Provider and competitor documentation

• OpenAI Batch API guide: 50% discount, 50,000 requests and 200 MB per batch, 24-hour window.
• Anthropic Message Batches: 50% discount, 100,000 requests or 256 MB per batch, 29-day result retention, typical completion under one hour.
• OpenRouter Batch API quickstart: beta status, typically 50% off, inline request format, 24-hour window, 30-day result retention.
• Together AI batch inference: up to 50% off with the full discount limited to selected models, open-weight catalog.
• Fireworks AI Batch API: flat 50% off serverless rates, open-weight and custom models.

Measured data: the failure rates, category-violation counts, token counts, and model-quality comparison come from the authors’ own evaluation runs (two runs of 50 documents each through GPT-5 nano and GPT-5 mini over DigitalOcean’s serverless real-time API, August 2026), not from any documentation.