What Is Batch Inference? (2027)

Last week’s support tickets, a 50,000-item product catalog, and a backlog of contracts to summarize. What do these jobs have in common? None of them needs a response in two seconds. But if you run them through the same real-time endpoints as your live traffic, you pay live-traffic rates for every one.

The costs add up: Gartner forecasts $2.7 trillion in worldwide AI spending in 2026 and reports that enterprises are asking their providers to help them manage costs. Batch inference is built for exactly this gap.

On DigitalOcean, batch requests to OpenAI and Anthropic models cost up to 50% less than real-time rates and run on rate limits separate from your production traffic. Let’s explore how batch jobs work, where they pay off, and where they don’t, so you can decide when to use them for your AI workloads.

What is batch inference?

Batch inference (also called offline inference or asynchronous inference) is running an AI model over a large set of requests as a single asynchronous job, with results returned later instead of immediately. Think of jobs like scoring an evaluation suite, tagging a year of product reviews, or generating synthetic training data. You hand over a file of requests, the system processes them when capacity allows, and you collect the outputs when the job finishes. The trade-off involves giving up an instant response to get a lower cost per request and higher throughput.

Batch inference vs. real-time inference

Real-time inference answers one request at a time while a user or application waits. Batch inference processes thousands of requests in parallel, so nobody waits for the response for any single request.

How batch inference works

A real-time inference system handles one request, returns one response, and repeats. A batch system collects the requests first, then runs them as a group:

Real time: app → request → model → response (repeated for every request)

Batch: app → file of requests → job queue → model → results file + error file

Batch inference follows the same five-step workflow on most platforms, including DigitalOcean. You prepare and submit the job, the platform runs it, and you collect the results:

1. Collect your inputs: Gather the prompts you want to run—one request per line in a JSON Lines (JSONL) format, each with a unique ID so that you can match results back to inputs.

2. Submit them as one job: Upload the file and create the job, setting a completion window, the time the job has to finish. On DigitalOcean, that’s 24 hours.

3. Queue and run the job: The platform validates the file, queues the job, and processes requests as capacity allows within the completion window.

4. Review the output: Outputs are shared in a results file; failed requests go to a separate error file.

5. Download and use the results: Download the files, match each line to its input by ID, and write the outputs to your database or pipeline.

Batch inference vs. continuous batching

Continuous batching (also called dynamic batching) is a scheduling technique inside a model-serving engine: instead of waiting for a group of requests to finish, the engine adds and removes requests from the running batch at every decoding step. Batch inference is more about how you submit work: one asynchronous job, results delivered later.

Note that a provider can use continuous batching for real-time or batch traffic, so one doesn’t replace the other.

Benefits of batch inference

Batch inference pays off when volume is high, and speed isn’t the priority.

The gains show up on your bill, in your rate limits, and in the code you don’t have to write:

• Lower cost: On the DigitalOcean Inference Engine, you pay up to 50% less than real-time rates for batch requests to OpenAI and Anthropic models. You’re billed only for completed requests, so unprocessed work from a failed or expired job never appears on your invoice.

• Higher throughput without rate-limit pain: Batch jobs use separate rate limits from serverless and dedicated inference, so a running job doesn’t consume your real-time quota. Push bulk work through without crowding out the endpoints your users depend on.

• Simpler operations: Skip writing queueing code, because the platform retries transient errors (429, 408, and 5xx) up to two times on your behalf. One failed request doesn’t fail your batch; failures are logged in a separate error file you can review.

• Optimized resource use: Your jobs run at a lower scheduling priority on shared off-peak GPU capacity, trading speed for efficiency. You also submit everything at once instead of keeping thousands of connections open while you wait.

• Affordable evaluation at scale: Run thousands of test cases against a new prompt or model as one batch, and test more variations for the same budget. For multi-turn agent testing, DigitalOcean Simulations (public preview) plays generated conversations against your agent.

Batch Inference use cases

A job is a good fit for a batch when it has a high volume of independent requests, and the results can wait.

Here’s how that works in practice:

Document processing and data extraction

Contracts, research papers, job postings, and financial filings pile up faster than anyone can read them. In a batch job, each line of the input file holds one document and an extraction prompt, and a custom_id ties every result back to its source. Nothing downstream waits on the answer, so batch inference on DigitalOcean can run the whole backlog without tying up your real-time traffic.

Developer Dumky de Wilde used OpenAI’s Batch API to backfill about 60,000 job postings on selectfrom.work, pulling a summary, salary, technologies, and job type from each. The full run costs about $10, and he reports that batches often finish in 10 to 15 minutes—much shorter than the planned 24-hour window.

For the same job inside a data platform, Arthur Dooner, a Senior Specialist Solutions Architect at Databricks, walks through summarizing documents in bulk with the ai_query() function.

Whichever tool you use, check a sample of the extracted fields against the source documents before you load the full output into a database.

Support ticket classification and triage

A support team has no clear view of customer message patterns if their data is raw and unsorted. Batch inference labels each ticket by category, urgency, and sentiment, then returns a .jsonl file you can load into your analytics pipeline or CRM.

DigitalOcean engineers ran this job on 250,000 real consumer complaints, labeling each by product line, scoring sentiment and urgency, extracting entities, and drafting a summary. All 250,000 requests completed across seven batch jobs, with the largest 40,000-request jobs finishing in 1.4 to 2.2 hours. The computed cost was $7.04, against $14.07 for the same tokens at serverless rates.

Taking it one step further, DigitalOcean engineers graded the output against the dataset’s own labels. The cheaper model, GPT-5 Nano, agreed with them 83.6% of the time, and Claude Haiku 4.5 reached 84.9% at roughly 35 times the computed cost. Treat batch labels as a strong first pass, and have humans review the tickets where urgency or category drives a decision.

Content moderation backlogs

Marketplaces, forums, and review sites accumulate backlogs of posts and listings that need a policy check. Batch inference clears that backlog overnight without competing with your live moderation endpoint for rate limits. It handles text only and returns results after the job completes, so it suits audits and re-checks after a policy change.

Social platforms often require heavy moderation, and Anthropic’s content moderation guide helps you size up the work. It shows how to estimate tokens and monthly cost for a platform with a billion posts a month, and how much model choice changes the bill. A backlog check doesn’t need an instant answer, so running it as a batch job can take up to 50% off real-time rates.

Catalog enrichment

An e-commerce platform with 50,000 products needs search-friendly titles, descriptions, and metadata for every listing. Batch inference can generate the English copy in one job and the localized translations in a second, through the same pipeline.

Instacart does this on a far larger scale. Engineer Paul Baranowski wrote up how the Catalog team uses LLMs to fix product data errors and add attributes to listings. Its Maple service sends millions of prompts through a provider’s batch API.

Model evaluation and synthetic data generation

Before you ship a new system prompt or model, you benchmark it against thousands of test cases, and a model often writes those test cases too. Both jobs suit the batch. Each request is independent; the volume is high; nobody is waiting for the answer; and you rerun the same suite whenever a prompt or model changes—so the discount applies every time. DigitalOcean’s docs list evaluation suites such as MMLU, SimpleQA, and SWE-bench, and synthetic data generation among batch jobs.

The UK AI Security Institute’s open-source Inspect framework supports this directly. Pass --batch to send evaluation requests to providers’ batch APIs, including OpenAI and Anthropic. Evals with a small number of sequential generations, such as a QA eval scored by a model, fit batch well, while multi-turn agent tasks can take many hours or days.

Developer James Peralta built an LLM evaluator and then ran the evaluation through OpenAI’s Batch API to cut costs by 50%:

Synthetic data is training or test data that a model generates rather than collects from users. Jason Liu, creator of the open-source Instructor library, shows how to generate synthetic question-and-answer pairs from MS MARCO passages with OpenAI’s Batch API to evaluate RAG retrieval. His batch job took about six minutes and cost $2.72.

A model grading its own output is a weak check. Filter generated data for duplicates and errors before you train or test on it, and use an LLM-as-a-judge pass to screen both the generated data and your eval results at the same scale.

When to use batch inference

Start with one question: Is a person or an agent waiting for this answer? If the output is headed to a database, a report, or a training set instead of someone’s screen, batch inference is worth a look.

Here’s how to decide:

• If your job is a fixed pile of inputs that already exist, such as 50,000 product listings or a year of tickets, submit it as one file and let the 24-hour window do the work. For timely needs, schedule the job the day before you need the output.

• If a backfill keeps tripping rate limits on your live endpoints, move it to batch, where limits are separate, and a running job doesn’t use your real-time quota. Note that you give up control over exactly when the job runs.

• If non-interactive work dominates your inference bill, list every request that has no one waiting on the response, and move those first.

• If you’re preparing to change a prompt or model, run your test suite as a batch before you ship. Since you’ll wait up to a day for results, iterate on a small real-time sample first, then batch the full suite.

• If a person or an agent is waiting for the answer, or you need results in minutes, skip batch. Chat, search, and agent loops belong on Serverless or Dedicated Inference—a 24-hour window is the wrong fit for them.

DigitalOcean Batch Inference

DigitalOcean Batch Inference runs large sets of OpenAI and Anthropic text requests as one asynchronous job on the Inference Engine. It uses the same API and model access key as Serverless Inference, so you can move non-interactive work to batch without changing providers. Jobs have a 24-hour completion window. If a job isn’t done by then, it expires: finished requests are kept and billed, unfinished ones aren’t billed, and a continuation job can run what’s left.

Plan around a few limits: only text prompts are supported, only for OpenAI and Anthropic commercial models, and open-weight and DigitalOcean-hosted models aren’t supported in batch. Input files can be up to 200 MB, with up to 50,000 requests per file.

DigitalOcean Batch Inference key features:

• Job Queue and Insights pages in the Control Panel show job status, progress, and usage for OpenAI and Anthropic jobs in a single list.

• Automatic retries rerun transient errors (429, 408, and 5xx) up to two times with exponential backoff before writing a failed request to a separate error file.

• A continuation job processes only the requests that failed or were never reached, so a partial run doesn’t mean resubmitting the whole file.

• Job creation is idempotent: if a network error causes your client to retry, a repeated request_id returns the original job rather than a duplicate.

DigitalOcean Batch Inference pricing:

• Up to 50% off real-time rates for OpenAI and Anthropic models, with billing for completed requests only. See the Batch Inference pricing details.

Pricing and feature information in this article are based on publicly available documentation as of October 2026 and may vary by region and workload. For the most current pricing and availability, please refer to each provider’s official documentation.

How to use DigitalOcean batch inference

To run a batch job on DigitalOcean, follow these six steps:

1. Create a model access key: In the Control Panel, open the Model Access Keys page and click Create model access key. Select All models, then check Enable batch inferencing for OpenAI and Anthropic text models. You’ll use the key to authenticate every request.

2. Prepare a JSONL file: Put one request per line, each with a unique custom_id. OpenAI and Anthropic use different line formats:

OpenAI:

Anthropic:

3. Reserve an upload slot and upload the file: Send POST /v1/batches/files to get a file_id and a presigned URL on DigitalOcean Spaces, then PUT the file to that URL. The URL expires after ~15 minutes, so upload right away. If it lapses, request a new upload slot because the job creation fails until the file exists in storage.

4. Create the job: Send a POST to /v1/batches with the file_id, provider, a completion_window of 24h, and a request_id that supports safe retries. OpenAI jobs also require an endpoint (/v1/responses or /v1/chat/completions), while Anthropic jobs omit it.

5. Monitor status: Poll GET /v1/batches/{batch_id}, or watch the Job Queue in the Control Panel: open Batch Inference under the Inference section of the left menu, then select the Job Queue tab, which lists every job with its status and progress.

6. Download the results: Send GET /v1/batches/{batch_id}/results. Failed requests come back in a separate error file.

For request formats, file limits, and the full API reference, see the DigitalOcean docs.

Here’s an example that uses PyDo to write the input file, upload it, create the job, poll until the job finishes, and download the results:

       "Classify this review as positive or negative: ..."]

for i, prompt in enumerate(prompts):

    f.write(json.dumps({

        "custom_id": f"req-{i}", "method": "POST",

        "url": "/v1/chat/completions",

        "body": {"model": "gpt-4.1-mini",

                 "messages": [{"role": "user", "content": prompt}],

                 "max_tokens": 256},

    }) + "\n")

requests.put(intent["upload_url"], data=fh,

             headers={"Content-Type": "application/octet-stream"},

             timeout=60).raise_for_status()

file_id=intent["file_id"], provider="openai",

endpoint="/v1/chat/completions", completion_window="24h",

request_id=os.urandom(16).hex())

batch = client.batches.retrieve(batch_id)

if batch.get("status") in ("completed", "failed", "expired", "cancelled"):

    break

time.sleep(60)

out = requests.get(info["output_file_url"], timeout=60)

open("batch_output.jsonl", "wb").write(out.content)

Once results are in, match each output to its input by custom_id and read the error file for failures. Reusing a request_id on retry replays the existing job instead of creating a duplicate, and a continuation job covers failed or unreached requests without resubmitting the whole file.

Batch inference FAQ

What is a cheap way to run batch processing for inference?

Move work that can wait off real-time endpoints and into a batch API. On DigitalOcean Batch Inference, requests to OpenAI and Anthropic models cost up to 50% less than real-time rates, and you pay only for completed requests. The discount depends on the model, so check current pricing for details.

Which inference provider offers a batch API for cheap overnight processing of millions of documents?

DigitalOcean Batch Inference handles this. Split your documents into JSONL files of up to 50,000 requests each and submit each file as a job with a 24-hour completion window. Batch jobs run on rate limits separate from your real-time traffic, so an overnight backfill doesn’t compete with production, and requests cost up to 50% less than real-time rates.

What inference cloud has cheap batch processing?

The DigitalOcean Inference Engine includes Batch Inference, as well as Serverless and Dedicated Inference. Batch requests to OpenAI and Anthropic models cost up to 50% less than real-time rates and run through the same API and model access key as your real-time calls. You can follow each job from queued to complete in the Control Panel: open Batch Inference under the Inference section of the left menu, then select the Job Queue tab, which lists every job with its status and progress.

What is the cheapest way to run DeepSeek for batch processing?

On DigitalOcean, DeepSeek isn’t available in Batch Inference, which covers OpenAI and Anthropic text models only. You can run DeepSeek models on Serverless Inference at real-time rates, and the batch discount applies only to the supported models.

What is continuous batching in LLM inference?

Continuous batching is a scheduling technique within a model-serving engine that adds and removes requests from the running batch at each decoding step, rather than waiting for the entire batch to finish. It keeps the GPU busy when traffic is uneven. It’s different from batch inference, which submits a large set of requests as one asynchronous job, as DigitalOcean Batch Inference does. The two are related, though: both group requests, and a provider can use continuous batching to serve batch jobs or real-time traffic.

Run your overnight work for less with DigitalOcean Batch Inference

DigitalOcean Batch Inference runs large OpenAI and Anthropic text workloads as asynchronous jobs on the Inference Engine. Submit a file, collect results within a 24-hour window, and keep your live traffic’s rate limits untouched. Pay for the intelligence you use, not real-time rates for work that can wait.

Key features include:

• Up to 50% lower cost than real-time inference

• Isolated rate limits, so batch jobs don’t draw on your production quota

• Job tracking from queued to complete in the Control Panel

• OpenAI- and Anthropic-compatible batch formats for easy migration

→ Get started with Batch Inference today

Traversal, an AI SRE startup, makes millions of API calls a month through the DigitalOcean AI Platform with ~99.96% availability.

Results in customer environments may vary depending on configuration, implementation, and usage. Results and/or savings are not guaranteed.

Any references to third-party companies, trademarks, or logos in this document are for informational purposes only and do not imply any affiliation with, sponsorship by, or endorsement of those third parties.