What RAG Actually Costs to Run in Production: A Full Cost Breakdown

The assumptions, stated in full so the model is reproducible. Corpus: 100,000 documents averaging 1,500 tokens each (150M tokens of text), chunked at 512 tokens with 64-token overlap into about 335,000 chunks (171M embedded tokens; overlap causes a 1.14x expansion). Per query: a 32-token question embedding, reranking of 20 retrieved candidates (about 10,400 input tokens, 40 output), and answer generation over the top 5 chunks (about 2,900 input tokens, 350 output). Models: Qwen3 Embedding 0.6B ($0.04 per 1M tokens), DeepSeek V4 Flash for reranking ($0.068 in / $0.168 out per 1M), gpt-oss-120b for answers ($0.10 / $0.70) with Llama 3.3 70B ($0.65 / $0.65) as the alternative. Months are 30 days.

One-time ingestion costs

Ingesting 100,000 documents costs under $10 without enrichment and under $20 with it. If the corpus doubles, these numbers double; they never become the problem.

Monthly costs at three traffic levels

Cost per query, all-in with gpt-oss-120b, is each total divided by that tier’s monthly query volume: about $0.0033 at low traffic ($98 ÷ 30,000 queries, where the fixed database cost dominates), $0.0017 at mid traffic ($496 ÷ 300,000), and $0.0013 at high traffic ($4,000 ÷ 3,000,000). Database plan prices are representative current tiers; confirm against the managed database plans for your region and engine.

Three things fall out of the table, and each one contradicts a common assumption.

First, embeddings and storage are cheap. Query embeddings never reach even 1% of the bill, and the vectors add about a dollar of storage on top of the database plan (the database line in the table is the whole managed cluster, which most applications run anyway). Trying to save money here is optimizing the wrong line item. It also means re-embedding the whole corpus after a chunking change costs about $7. You can afford to experiment.

Second, model choice for generation is a 4x swing. gpt-oss-120b and Llama 3.3 70B run identical prompts in this pipeline, and the difference at 100K queries per day is $1,605 versus $6,338 per month. That makes generation model selection the highest-leverage cost decision in the entire system. Evaluate whether the cheaper model answers your questions well enough before defaulting to the bigger name; in RAG, where the model’s job is reading provided context rather than recalling facts, smaller models close much of the quality gap.

Third, at high traffic the reranker quietly becomes the largest line item. This surprises people because reranking feels like a minor post-processing step, but it reads all 20 candidate chunks per query, roughly four times more tokens than the answer model sees. Token volume, not model price, is what makes it expensive.

How to lower the bill

The reranking bill has straightforward controls. Rerank 10 candidates instead of 20 and the cost halves, usually with little quality loss if retrieval is decent. Rerank conditionally, only when the top vector scores are close together and the ordering is genuinely ambiguous, and you skip the step entirely for the majority of easy queries. Or use the managed knowledge base reranker (BGE Reranker v2 m3 at $0.01 per million tokens, roughly a seventh of the LLM reranker’s effective rate) if you adopt the managed stack.

Generation has levers too. Prompt caching, supported on serverless inference, discounts the repeated system prompt on models that support it, though retrieved chunks differ per query and will not hit the cache. Capping max_tokens and instructing the model to answer concisely directly cuts the most expensive tokens you buy.

There is also a crossover point worth knowing. Serverless pricing scales linearly with traffic forever, but dedicated inference does not: a single NVIDIA H100 runs $4.41 per hour, about $3,220 per month, flat, regardless of query volume. When your monthly serverless generation and reranking spend approaches that number (in this model, somewhere near the 100K-queries-per-day tier), dedicated capacity for an open-weight model starts winning on price and gives you consistent latency at the same time. Below that, serverless wins because you pay nothing for idle.

The token prices in this model are not unusually low. For gpt-oss-120b, Fireworks AI, Groq, and Together AI all published $0.15 per 1M input and $0.60 per 1M output as of July 2026, against DigitalOcean’s $0.10 and $0.70 (verified pricing and compatibility differences are in our comparison of OpenAI-compatible inference APIs). Which is cheaper depends on your traffic shape, and RAG has a distinctive one: this pipeline sends about 8 input tokens for every output token, because the model reads five chunks to write one answer. Input-heavy traffic favors the provider with the cheaper input rate. On this model’s token mix, generation costs $0.000535 per query on DigitalOcean versus $0.000645 at the $0.15/$0.60 providers: $1,605 versus $1,935 per month at 100K queries per day. Routing through an aggregator like OpenRouter adds its 5.5% credit fee on top of whichever provider serves the request.

The larger difference is structural rather than per-token. A RAG pipeline is not just inference: it needs a vector database, and most inference providers do not run one. Building this stack on an inference-only provider means a second vendor for vector storage (a dedicated vector database, or self-hosted pgvector on rented compute), a second bill, separate access control, and a network hop between retrieval and generation that sits on your latency path for every single query. Running Managed PostgreSQL and the inference endpoint in one account removes the cross-vendor hop, keeps the whole pipeline’s spend on one invoice, and means the database backing your vectors has the same backups, failover, and monitoring as the rest of your infrastructure. The per-token differences above are real but small; the operational consolidation is usually the stronger argument.

Everything above is grounded in a concrete pipeline, and the rest of this article builds it. The ingestion half runs once (and again when documents change): read documents, split them into chunks, generate an embedding per chunk, and write chunk text plus vector into PostgreSQL. The serving half runs on every question: embed the question, pull the nearest chunks, rerank them, and pass the survivors to a model that writes the answer.

Documents ──> Chunking ──> Embeddings API ──> Managed PostgreSQL (pgvector)
                                                        │
User question ──> Embeddings API ──> vector search ─────┘
                                          │
                                    top 20 chunks
                                          │
                                    LLM reranker
                                          │
                                     top 5 chunks
                                          │
                            Serverless Inference (answer)

Everything model-related goes through one endpoint, https://inference.do-ai.run/v1, which is OpenAI-compatible: the official OpenAI SDK works after changing the base URL and API key, and you swap models by changing a string. The pipeline uses open-weight models throughout, which is what keeps the cost model above where it is.

To follow along you need a DigitalOcean account with a model access key and a prepaid serverless inference balance, a Managed PostgreSQL cluster, and Python 3.10+ with openai, psycopg2-binary, and tiktoken.

Chunking

Language models and embedding models both work better on focused passages than on whole documents, and vector search retrieves at the granularity you embed. Chunking is where you set that granularity, and it directly drives the 171M-token embedding figure in the cost model.

We use fixed-size chunks of 512 tokens with a 64-token overlap. Fixed-size chunking is not the cleverest strategy, but it is predictable, cheap, and hard to get wrong, which is what you want for the first production version. The overlap exists so a sentence falling on a chunk boundary appears intact in at least one chunk; it is also why 150M tokens of text becomes 171M tokens of embedding input. Overlap is not free, and this is where you pay for it.

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

CHUNK_TOKENS = 512
OVERLAP_TOKENS = 64

def chunk_text(text: str, doc_id: str) -> list[dict]:
    tokens = enc.encode(text)
    stride = CHUNK_TOKENS - OVERLAP_TOKENS
    chunks = []
    for i, start in enumerate(range(0, len(tokens), stride)):
        window = tokens[start : start + CHUNK_TOKENS]
        if len(window) < 32:   # skip trailing fragments
            break
        chunks.append({
            "doc_id": doc_id,
            "chunk_index": i,
            "text": enc.decode(window),
        })
    return chunks

Two practical notes. Keep doc_id and chunk_index from the start; you need them for citations, re-indexing, and deletions. And resist the urge to chunk small: chunks of 100 to 200 tokens retrieve precisely but often lack enough context to answer from, which pushes you into fetching more chunks and paying more per query.

Embeddings

An embedding turns a chunk of text into a vector such that semantically similar texts end up near each other. This is what lets “how do I reset my password” find a chunk titled “credential recovery procedure” even though they share no keywords.

The serverless Embeddings API offers several open models. We use Qwen3 Embedding 0.6B at $0.04 per million input tokens: it handles long inputs comfortably, performs well on multilingual and retrieval benchmarks, and produces 1024-dimensional vectors. If your corpus is English-only and cost matters more than quality, BGE-M3 and E5 Large V2 are available at $0.02 per million tokens, which would cut the already-small $6.86 in half.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://inference.do-ai.run/v1",
    api_key=os.environ["DIGITALOCEAN_INFERENCE_KEY"],
)

def embed_batch(texts: list[str]) -> list[list[float]]:
    resp = client.embeddings.create(
        model="qwen3-embedding-0.6b",
        input=texts,
    )
    return [item.embedding for item in resp.data]

Send chunks in batches (32 to 64 per request balances throughput against request size) and add retry logic with exponential backoff; any ingestion job spanning hundreds of thousands of requests will hit a transient failure eventually.

Optional: enrich chunks with batch inference at half price

Raw chunks lose document-level context. A chunk that says “the limit was raised to 50” retrieves poorly because nothing in it says what limit, or where. A common fix is to prepend a short LLM-generated context line to each chunk before embedding it: which document it came from and what it is about.

This is offline, latency-insensitive work, which makes it a fit for batch inference; how the discount works and how to submit, monitor, and download a job are covered in that piece and in the batch inference guide, so here we stick to the RAG-specific question: which parts of this pipeline belong in a batch job.

The rule is simple: batch anything that runs before users show up. Chunk enrichment qualifies exactly because no one is waiting on it; whether the job finishes in one hour or twenty-four changes nothing about the pipeline. Query-time work (retrieval, reranking, answer generation) never qualifies, because a user is on the other end. Two constraints shape the fit here: batch currently supports only OpenAI and Anthropic text models on the chat endpoints, so this one step uses a commercial model even though the rest of the pipeline is open-weight, and embedding requests themselves cannot go through batch (at $0.04 per million tokens, they do not need to; the LLM enrichment step, not embedding, is where ingestion money actually goes).

Each request in the batch file asks a small model to situate one chunk:

def enrichment_request(chunk: dict, doc_title: str) -> dict:
    return {
        "custom_id": f'{chunk["doc_id"]}-{chunk["chunk_index"]}',
        "method": "POST",
        "url": "/v1/chat/completions",
        "body": {
            "model": "gpt-5-nano",
            "messages": [{
                "role": "user",
                "content": (
                    f"Document title: {doc_title}\n\n"
                    f"Chunk:\n{chunk['text']}\n\n"
                    "Write one sentence situating this chunk within the "
                    "document, for use as a search context prefix. "
                    "Output only the sentence."
                ),
            }],
            "max_tokens": 80,
        },
    }

For 335,000 chunks (roughly 650 input and 60 output tokens per request with GPT-5 nano), the job costs about $18.92 at serverless rates and $9.46 through batch: the 50% discount doing its job on the one ingestion step that involves an LLM. Prepend the returned sentence to each chunk before embedding, and both the embedding and the reranker see the added context.

This step is optional. Ship without it first, look at your retrieval failures, and add it if chunks retrieve poorly for lack of context. Re-embedding afterward costs another $7, cheap enough to treat as an experiment rather than a commitment.

Vector storage in Managed PostgreSQL

You need somewhere to keep 335,000 vectors and their text, and to find the nearest vectors to a query quickly. A dedicated vector database is one option, but if you are already running PostgreSQL, the pgvector extension turns the database you have into the vector store you need, with backups, failover, and access control already handled by the managed platform. DigitalOcean Managed PostgreSQL supports both vector (pgvector) and vectorscale as standard extensions.

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
    id          BIGSERIAL PRIMARY KEY,
    doc_id      TEXT NOT NULL,
    chunk_index INT  NOT NULL,
    text        TEXT NOT NULL,
    embedding   vector(1024) NOT NULL,
    UNIQUE (doc_id, chunk_index)
);

CREATE INDEX chunks_embedding_idx
    ON chunks USING hnsw (embedding vector_cosine_ops);

The HNSW index is what makes search fast. Without it, every query scans all 335,000 vectors; with it, queries traverse a graph and return in single-digit milliseconds at this scale. Build the index after bulk-loading the data, not before: inserting into an existing HNSW index is much slower than building it once over a full table.

The ingestion loop ties the first three stages together:

import psycopg2
from psycopg2.extras import execute_values

conn = psycopg2.connect(os.environ["DATABASE_URL"])

def store_chunks(chunks: list[dict], embeddings: list[list[float]]):
    rows = [
        (c["doc_id"], c["chunk_index"], c["text"], emb)
        for c, emb in zip(chunks, embeddings)
    ]
    with conn.cursor() as cur:
        execute_values(
            cur,
            """INSERT INTO chunks (doc_id, chunk_index, text, embedding)
               VALUES %s
               ON CONFLICT (doc_id, chunk_index)
               DO UPDATE SET text = EXCLUDED.text,
                             embedding = EXCLUDED.embedding""",
            rows,
            template="(%s, %s, %s, %s::vector)",
        )
    conn.commit()

The upsert on (doc_id, chunk_index) makes re-ingestion idempotent: rerun a document and its chunks are replaced, not duplicated. When a document is deleted from the source, delete its rows by doc_id.

On sizing, which is where the database line in the cost table comes from: 335,000 vectors at 1,024 dimensions and 4 bytes per dimension is about 1.4 GB, plus the HNSW graph and chunk text, landing the table around 4 GB on disk. Single-node Managed PostgreSQL starts at $15 per month, fine for development. For production, HNSW search performance depends on the index staying in memory, so pick a plan with RAM comfortably above your index size and add a standby node for high availability ($30 per month and up per node). Additional storage is $0.21 per GiB per month, which for this corpus rounds to about a dollar.

Retrieval

Serving a question starts by embedding it with the same model used at ingestion (not optional; vectors from different models live in different spaces and comparing them is meaningless), then asking PostgreSQL for the nearest chunks:

def retrieve(question: str, k: int = 20) -> list[dict]:
    q_emb = embed_batch([question])[0]
    with conn.cursor() as cur:
        cur.execute(
            """SELECT doc_id, chunk_index, text,
                      1 - (embedding <=> %s::vector) AS score
               FROM chunks
               ORDER BY embedding <=> %s::vector
               LIMIT %s""",
            (q_emb, q_emb, k),
        )
        cols = ["doc_id", "chunk_index", "text", "score"]
        return [dict(zip(cols, row)) for row in cur.fetchall()]

The <=> operator is cosine distance, matching the vector_cosine_ops index. We deliberately over-fetch: 20 chunks is more than the answer model will see, because the next stage exists to separate the truly relevant from the merely similar.

Two upgrades worth knowing about, neither required for version one. Hybrid search combines vector similarity with PostgreSQL’s built-in full-text search, which helps when users search for exact identifiers, error codes, or product names that embeddings blur. And metadata filtering (a WHERE clause on tenant, date, or document type) is often the single biggest retrieval quality win in multi-tenant systems, because the strongest signal about which chunks are relevant is frequently not semantic at all.

Reranking

Vector search is a recall tool. It reliably gets relevant chunks into the top 20, but the ordering within those 20 is loose, and answer quality tracks what is in the top 5 the model actually reads. A reranker looks at the question and each candidate together and reorders them by actual relevance, which embeddings, computed for question and chunk separately, cannot fully judge.

We use a small, fast LLM as the reranker. DeepSeek V4 Flash costs $0.068 per million input tokens and handles listwise ranking well:

import json

RERANK_PROMPT = """You are ranking text passages by relevance to a question.

Question: {question}

Passages:
{passages}

Return a JSON array of the {n} passage numbers most relevant to the
question, most relevant first. Output only the JSON array."""

def rerank(question: str, candidates: list[dict], top_n: int = 5) -> list[dict]:
    passages = "\n\n".join(
        f"[{i}] {c['text']}" for i, c in enumerate(candidates)
    )
    resp = client.chat.completions.create(
        model="deepseek-v4-flash",
        messages=[{
            "role": "user",
            "content": RERANK_PROMPT.format(
                question=question, passages=passages, n=top_n
            ),
        }],
        max_tokens=64,
        temperature=0,
    )
    try:
        order = json.loads(resp.choices[0].message.content)
        return [candidates[i] for i in order[:top_n] if i < len(candidates)]
    except (json.JSONDecodeError, TypeError):
        return candidates[:top_n]   # fall back to vector order

Note the fallback: if the reranker returns malformed JSON, we serve the vector-search order rather than failing the request. Rerankers improve answers; they should never be a point of failure.

This is the step the cost model flagged as the sleeper expense. It reads about 10,400 input tokens per query, four times what the answer model reads, which is trivial at low traffic and the largest line item at high traffic. The levers from the cost section (fewer candidates, conditional reranking, or the managed reranker) all apply here.

Serving answers

The final step assembles the top chunks into a prompt and asks a model to answer strictly from them, using gpt-oss-120b for the reasons the cost model made clear:

SYSTEM_PROMPT = """Answer the user's question using only the provided context.
Cite the source of each claim using the [doc_id] shown with each passage.
If the context does not contain the answer, say so plainly."""

def answer(question: str) -> str:
    candidates = retrieve(question, k=20)
    top = rerank(question, candidates, top_n=5)
    context = "\n\n".join(f"[{c['doc_id']}] {c['text']}" for c in top)
    resp = client.chat.completions.create(
        model="openai-gpt-oss-120b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"},
        ],
        max_tokens=512,
        temperature=0.2,
    )
    return resp.choices[0].message.content

Wrap this in the web framework of your choice; a FastAPI endpoint with streaming enabled (stream=True) is the usual shape, and streaming matters for perceived latency because the user sees the first words while the rest generates.

The prompt structure is doing real work. Instructing the model to answer only from context and to admit when the context is insufficient is your main defense against confident fabrication, and passing doc_id through to the prompt is what makes answers citable. Users forgive “I don’t have that information.” They do not forgive invented policy details attributed to their own documentation.

Everything in this pipeline can also be had as a managed service: DigitalOcean knowledge bases handle chunking, embedding, OpenSearch-backed vector storage, and reranking behind a single retrieve endpoint, billed per token indexed and retrieved plus OpenSearch cluster costs (from $19 per month). The DIY pipeline gives you full control over chunking, hybrid search, filtering, and reranking strategy, and keeps your vectors in a database you can query with plain SQL. The managed path gets you to a working agent faster. Both use the same embedding models at the same prices, so the cost model above transfers.

The argument this article set out to make survives contact with the itemized numbers. RAG’s reputation as an expensive architecture comes from the wrong mental model: teams price the corpus when they should price the queries. Embedding 100,000 documents costs about $7 and storing the vectors about a dollar a month, while reranking and generation are over 99% of the model spend, scaling linearly with traffic and with the number of tokens each query reads (they grow from 38% of the total bill at 1,000 queries per day, where the fixed database cost dominates, to 94% at 100,000).

The three decisions that control your cost are, in order: which model generates answers (a 4x swing), how many chunks each query reads (the reranking multiplier), and when to move hot workloads from serverless to dedicated capacity (around $3,200 per month of token spend). None of these are exotic optimizations; all of them are visible in a cost model you can build in a spreadsheet, and this article gives you the formulas to build yours.

• DigitalOcean Inference pricing
• Generate vector embeddings
• Introducing Unified Batch Inference on DigitalOcean and the batch inference guide
• Best OpenAI-compatible inference APIs: drop-in alternatives for 2026
• Managed PostgreSQL pricing and supported extensions
• Prompt caching on serverless inference
• Knowledge bases and chunking best practices
• pgvector