A Practical Guide to RAG with Haystack and LangChain

Retrieval-Augmented Generation is an AI architecture that combines classic information retrieval with generative large language models (LLMs) in one pipeline. Rather than relying entirely on the static knowledge pre-trained into an LLM (which may be incomplete, out-of-date, or both), RAG dynamically augments the generation process by injecting retrieved information from external sources. The result is a more precise and contextually relevant generation that leverages the best of retrieval and generation technology.

The idea behind a RAG pipeline is conceptually simple. However, making a production-ready RAG pipeline can be complex. There are many tooling decisions and design tradeoffs that must be considered. This includes choosing a framework (Haystack vs. LangChain), selecting and tuning the vector database, document chunking strategies, engineering requirements, and deployment strategies.

The goal of this guide is to walk through, from a technical perspective, how to build production-ready RAG pipelines.

• RAG pipelines augment the responses of LLMs with real-time, retrieved information for more accurate answers.
• Haystack offers production-ready modular pipelines, and LangChain excels in prototyping, MLOps, and integrations with other tools.
• Selecting the right vector database (FAISS, ChromaDB, Pinecone, and more) determines your system’s scalability, speed, and features.
• The embedding model and chunking strategy affect retrieval quality and the quality of the final answer.
• Operationalizing a RAG pipeline in production environments requires specialized approaches, evaluation, monitoring, user feedback, and best deployment strategies.

A RAG pipeline usually has various components that interact with one another. The most essential building blocks are the following:

• Data Ingestion & Preprocessing: Raw knowledge sources in various forms, such as documents, web pages, and PDFs, are loaded into the system and preprocessed (conversion into a standard format, content extraction, normalization, etc. ), along with various text preprocessing techniques.
• Chunking: Each document is split into smaller semantically meaningful chunks. The chunking step is essential for efficient retrieval since it sets the granularity at which information is indexed and retrieved from the database.
• Embedding Generation: Every chunk is embedded into a high-dimensional vector using a language model fine-tuned for embedding.
• Vector Database: The vector database (or index) stores all chunk embeddings while supporting similarity search.
• Retrieval Engine: Upon receiving a user query, it is first embedded into a vector, and the retrieval component fetches the top-k most similar embeddings (chunks) in the vector database.
• Generation Pipeline: An LLM (generative component) finally receives the user’s query and the retrieved context chunks to generate a well-informed answer.

Components flow linearly in the pipeline, and production systems may also introduce branching, looping, or other additional steps (e.g., re-ranking of retrieved results before passing to LLM). The goal is for all components to work together and turn raw data into an accurate, contextually relevant, and up-to-date answer.

For those building RAG into production applications, high-level frameworks are often preferred. The primary feature and selling point of these frameworks is that they provide pre-built components and abstractions for building efficient pipelines. Haystack (by Deepset) and LangChain are by far the most popular and well-known tools in this category. Both aim for the same general use case – ease of building applications around LLMs. However, they have different underlying design principles and relative strengths.

Haystack: Pipeline-Centric Architecture

Haystack (by Deepset) is an open-source framework focused on RAG and other use cases for LLMs. The central feature of Haystack, further emphasized in the most recent release, is its pipeline-centric, modular architecture. Haystack adopts a graph-like approach, where each component (reader, retriever, generator, etc.) represents a node in a directed acyclic graph (DAG) pipeline. Haystack pipelines are composable and customizable – nodes can be added, removed, and replaced with minimal side effects on other components. Below is a table summarizing some of the main benefits of Haystack:

Technical Example (Haystack Pipeline)

Using pseudocode (inspired by Haystack 2.0 syntax), a simple RAG pipeline might be constructed as follows:

# Step 1: Install required packages
#pip install haystack-ai openai python-dotenv

# Step 2: Load environment variables
import os
from dotenv import load_dotenv

load_dotenv()  # Loads variables from a .env file if present

# OPTIONAL: Set the API key directly if not using .env
# os.environ["OPENAI_API_KEY"] = "your-openai-api-key"

# Step 3: Haystack imports
from haystack import Document
from haystack import Pipeline
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.components.embedders import OpenAITextEmbedder, OpenAIDocumentEmbedder
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever

# Step 4: Create the DocumentStore
doc_store = InMemoryDocumentStore()

# Step 5: Add raw documents
documents = [
    Document(content="The Eiffel Tower is located in Paris, France."),
    Document(content="Haystack is a framework for building NLP applications."),
    Document(content="LangChain is often used to build LLM-based agents."),
    Document(content="RAG stands for Retrieval-Augmented Generation.")
]
doc_store.write_documents(documents)

# Step 6: Embed and store document embeddings
embedder = OpenAITextEmbedder(model="text-embedding-ada-002")
for doc in doc_store.filter_documents():
    embedding = embedder.run({"text": doc.content})["embedding"]
    doc.embedding = embedding
doc_store.update_documents(documents)

# Step 7: Build the pipeline
pipeline = Pipeline()

# Add components with API key if needed (optional)
pipeline.add_component("embedder", OpenAITextEmbedder(model="text-embedding-ada-002"))
pipeline.add_component("retriever", InMemoryEmbeddingRetriever(document_store=doc_store))
pipeline.add_component("generator", OpenAIGenerator(model="gpt-3.5-turbo"))

# Connect the components
pipeline.connect("embedder.embedding", "retriever.query_embedding")
pipeline.connect("retriever.documents", "generator.documents")

# Step 8: Run the pipeline with a query
query = "What is RAG in NLP?"
result = pipeline.run({"embedder": {"text": query}})

# Step 9: Print the answer
print("\n Answer:", result["generator"]["replies"][0])

In this code, Haystack connects an embedding component to a dense retriever and then to a generator. You must ensure that OPENAI_API_KEY is set either via .env or by direct assignment in the code.

LangChain: Chain-Based Framework

LangChain is another open-source framework that gained some popularity for quickly prototyping LLM applications. It has a slightly different philosophy. Instead of declaring a static pipeline graph, you often chain together components in code (or with higher-level abstractions like “Chains” or “Agents”).
LangChain is known for its extensive integrations, flexibility, and support for LLM providers, vector stores, and tools(such as a web browser or calculator) integrated through an agent mechanism. The following table outlines some of the key benefits of using Langchain:

Technical Example (LangChain RAG Chain)

LangChain often represents RAG as a RetrievalQA chain. For instance:

# Step 1: Install required packages (run this once)
# pip install -qU langchain_community faiss-cpu openai python-dotenv

# Step 2: Imports
from langchain.vectorstores import FAISS
from langchain.embeddings import OpenAIEmbeddings
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.chains import RetrievalQA
from langchain.llms import OpenAI
from langchain.schema import Document
import os
from dotenv import load_dotenv

# Step 3: Load environment variables (e.g. OpenAI key)
load_dotenv()
# Optionally, you can set manually
# os.environ["OPENAI_API_KEY"] = "your-openai-api-key"

# Step 4: Prepare your documents
raw_documents = [
    "The Eiffel Tower is located in Paris, France.",
    "Haystack is a framework for building NLP applications.",
    "LangChain is often used to build LLM-based agents.",
    "RAG stands for Retrieval-Augmented Generation."
]

# Step 5: Split documents into chunks
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=100
)
chunks = text_splitter.create_documents(raw_documents)

# Step 6: Initialize embeddings
embeddings = OpenAIEmbeddings(model="text-embedding-ada-002")

# Step 7: Create vector store (FAISS is in-memory and fast)
vectorstore = FAISS.from_documents(chunks, embedding=embeddings)

# Step 8: Initialize RetrievalQA pipeline
qa_chain = RetrievalQA.from_chain_type(
    llm=OpenAI(model_name="gpt-3.5-turbo"),  # default temperature is 0.7
    retriever=vectorstore.as_retriever(search_kwargs={"k": 3}),
    chain_type="stuff",  # stuff = concatenate retrieved docs into prompt
    return_source_documents=True  # useful for debugging and transparency
)

# Step 9: Run a query
query = "What is RAG in NLP?"
result = qa_chain(query)

# Step 10: Print result
print("\n Answer:\n", result["result"])

# Optional: Show retrieved docs
print("\n Retrieved Documents:")
for doc in result["source_documents"]:
    print("-", doc.page_content)

This code snippet creates an in-memory FAISS vector index of the texts of all documents. It then calls LangChain’s RetrievalQA chain, which internally embeds the query, searches the vector store, and constructs a prompt for the LLM with the results. chain_type = “stuff” concatenates the retrieved documents (“stuffing” them) into the LLM prompt along with the question. Other chain types are available in LangChain, such as “map-reduce” (summarizing each document, then aggregating) or “refine” (iteratively building an answer), to handle more complex and larger contexts.

When to Choose Which?

We will recommend Haystack if your target is a maintainable, evaluable, production-ready system and you’re largely following a retrieval QA or similar pattern. If you’re trying to prototype something quickly or need complex agent behaviors/integrations, LangChain may help you get there faster. Some teams use Haystack for the core pipeline and add in LangChain agents where necessary, so these aren’t exclusive either.

The vector database is a foundational component of any RAG pipeline. It stores your document embeddings and performs a similarity search. The vector store you choose will impact performance (query latency, speed), but also features like filtering, scaling, and ease of integration. Let’s compare a few popular vector stores and their use cases:

• You can use FAISS for fast local querying on reasonably sized data where you can fit in memory and want full control. (e.g., <10 million vectors)
• Go on with ChromaDB for simple deployment if your dataset is not huge and you want persistence and Python integration. It’s perfect as an in-memory vector store for a single-server app or as an embedded vector store.
• Use Pinecone (or a similar managed service, such as Weaviate Cloud, Azure AI Search, etc. for a production app that may need to scale to billions of embeddings or multiple regions. Pinecone’s additional latency can typically be hidden with caching or batch querying, and its consistency and features (such as metadata-based filtering) are extremely valuable for complex use cases.

Example: Local Development with ChromaDB
During development, you might start with Chroma for convenience. Let’s consider the following pseudo-code using Langchain:

# Install required packages
#pip install langchain langchain-community chromadb pypdf tiktoken python-dotenv

# Additional for document loading
#pip install beautifulsoup4 html2text lxml


# ===== CHROMA/LANGCHAIN RAG PIPELINE PSEUDO-CODE =====

# 1. ENVIRONMENT SETUP
import os
from langchain.vectorstores import Chroma
from langchain.embeddings import OpenAIEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import PyPDFLoader

# 2. CONFIGURATION
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")  # Required environment variable
PERSIST_DIR = "./chroma_db"                  # Storage directory
COLLECTION_NAME = "corporate_docs"           # Vector collection name
EMBEDDING_MODEL = "text-embedding-3-large"   # OpenAI model

# 3. DOCUMENT PROCESSING
# - Load documents from multiple sources
documents = PyPDFLoader("sample_financial_report.pdf").load()  # sample source

# - Semantic-aware chunking
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,      # Character limit per chunk
    chunk_overlap=200,    # Context preservation overlap
    separators=["\n\n", "\n", ". ", " "]  # Natural language boundaries
)
chunks = text_splitter.split_documents(documents)

# 4. EMBEDDING INITIALIZATION
embeddings = OpenAIEmbeddings(
    model=EMBEDDING_MODEL,
    openai_api_key=OPENAI_API_KEY  # Secure credential handling
)

# 5. VECTOR STORE CREATION
vectorstore = Chroma(
    collection_name=COLLECTION_NAME,
    embedding_function=embeddings,
    persist_directory=PERSIST_DIR,

    # Optional production parameters
    client_settings={
        "anonymized_telemetry": False  # Compliance setting
    }
)

# 6. DOCUMENT INJECTION
vectorstore.add_documents(
    documents=chunks,
    ids=[f"doc_{idx}" for idx in range(len(chunks))],  # Unique identifiers
    metadatas=[chunk.metadata for chunk in chunks]  # Preserve context
)

# 7. PERSISTENCE OPERATION
vectorstore.persist()  # Critical for disk storage

# 8. RETRIEVAL VERIFICATION
query = "Q3 financial projections"
results = vectorstore.similarity_search(
    query=query,
    k=5,  # Top 5 results
    filter={"department": "finance"}  # Metadata-based filtering
)

The preceding scripts create a production-ready RAG pipeline built with LangChain and Chroma. It loads PDF documents, splits them semantically, embeds them using OpenAI’s embedding model, stores them in a persistent Chroma vector database, and uses filtered similarity search to retrieve relevant context for queries. This pipeline will be ready for deployment after:

1. Installing required packages
2. Adding documents to the docs/ directory
3. Setting OpenAI API key in .env file
4. Running the script

No matter which vector DB you choose, certain best practices apply:

• Use Approximate Search for Scale: Exact k-NN search is expensive, and doesn’t necessarily scale well to very large datasets. Algorithms like HNSW (Hierarchical Navigable Small World) or IVF (Inverted File Index) can speed the search at the expense of some controllable loss in search accuracy. The exact settings for this are configurable with FAISS and most other services (e.g., Pinecone does approximate search under the hood). In FAISS, you would explicitly choose an index type (IVF1000_HNSW32, etc.) and tune the index parameters (HNSW M(Connections per node), efsearch(Search traversal depth), etc.)
• Dimension and Model Considerations: Vector search can be slower for very high-dimensional embeddings (e.g., 1536 dimensions from ada-002, or 2048+ from the newer models), due to the curse of dimensionality. To address this, you can try PCA or other dimensionality reduction to compress the vectors, or use a smaller embedding model if needed.
• Filtering and Sharding: If your data is organized into categories (by time, by source, etc), you can shard your index or use filters (by maintaining separate indexes for different document types and querying only the relevant one(s), for example). This reduces the search space and query latency. Some databases have built-in support for data sharding or segmentation (Milvus, Weaviate), while others may require manual segmentation.
• Maintain Metadata: Useful metadata (source, document ID, section title, etc) can be stored with each vector in the index, which allows for filtering and better downstream prompting/debugging (e.g., if you know the source, you can preface a response with “According to source X…”).

The vector database is your information store. You can choose the one that scales to your needs and meets deployment constraints, then use the features to best balance speed, accuracy, and development effort.

Choosing the right embeddings is a foundational decision in the architecture of any RAG pipeline. Embeddings influence both retrieval quality and computational efficiency. Optimal embeddings ensure accurate matches between user queries and relevant documents. This directly drives the effectiveness of semantic search applications and context-aware LLM applications.

OpenAI’s text-embedding-ada-002: A Reliable Default

One of the most popular commercial models used in production has been OpenAI’s text-embedding-ada-002.

• The 1536-dimensional vectors used by this model strike a strong balance between representational capacity and computational cost.
• With a generous 8191-token context window, ada-002 can process longer passages, entire paragraphs, or even segments of multi-turn chat without truncation.
• Competitive performance: On the MTEB (Multilingual Text Embedding Benchmarks) leaderboard for embedders, ada-002 was in the top models for English tasks.
• High throughput: The API supports batching. So, thousands of embeddings per second can be served in production.

OpenAI’s latest generations (particularly text-embedding-3-small and text-embedding-3-large, released in late 2024) have further improved quality or lowered prices, but ada-002 remains widely available and a “safe default” for most production scenarios.

Open-Source and Specialized Alternatives

For many organizations, open-source or self-hosted options are preferable for reasons of privacy, fine-tuning, or cost avoidance.

• Instructor Models: Instruct-style (e.g., hkunlp/instructor-xl) models are trained with instructions or prompts to specialize embeddings for different tasks or domains.
• Sentence-BERT (SBERT) Family (e.g., all-MiniLM-L6-v2, all-mpnet-base-v2): SBERT models are lightweight, efficient, and easy to run on standard CPU/GPU hardware. These models are great for general-purpose search, quick prototyping, and in situations where deployment infrastructure needs to be extremely lightweight.
• Multilingual Models (e.g., LaBSE, distiluse-base-multilingual-cased-v2): If you’re building applications for international or multilingual audiences, these models will provide high-quality embeddings for dozens of languages. They are essential for non-English retrieval or cross-lingual search.
• Domain-Specific Models (e.g., PubMedBERT for biomedical, CodeBERT for code): If your RAG pipelines operate within specialized fields (such as medicine, law, engineering, etc. ), then these models will dramatically improve retrieval by understanding the vocabulary and semantic structure specific to those fields.

Recent trends favor dual-embedding strategies: you can use a lower-cost, faster embedder (like OpenAI’s text-embedding-3-small) to retrieve a candidate set for recall, then rerank the results with a more accurate or specialized model. This can provide a good trade-off between speed, cost, and retrieval performance.

Choosing the appropriate chunking strategy is essential for optimizing retrieval accuracy and overall LLM performance within RAG pipelines. Each method presents trade-offs between implementation simplicity, retrieval quality, and resource consumption. The table below outlines the most common chunking strategies and describes their implementation within LangChain and key considerations:

There isn’t a single optimal approach to chunking. Your choice should depend on the structure of your data, your retrieval objectives, and the level of complexity you can manage. Fixed-size chunking offers speed and simplicity, whereas semantic and sentence-level strategies can provide more accurate, contextually relevant results but may involve more overhead in terms of setup and computation. Experimenting with these strategies is important if you want to develop a high-performance, production-ready RAG system.

Beyond the basic setup of “embed -> retrieve -> generate”, additional techniques can be applied to optimize a RAG pipeline. Here are a few techniques RAG practitioners can use to improve relevance and answer quality:

Query Rewriting and Expansion: Query expansion addresses the problem of query-document vocabulary mismatch by creating multiple variations of the user’s query to improve recall by casting a wider net for retrieval.

Typical Workflow:

1. The user submits an initial query.
2. An LLM generates various alternative queries, such as paraphrases, expansions, or decompositions.
3. Each generated query is routed to the vectorstore or retriever system.
4. The results are combined (often as a unique set) to maximize the overall coverage of the response.

HyDE (Hypothetical Document Embeddings): Instead of embedding and searching with the user query directly, first create a hypothetical answer/document for the query using an LLM, then embed that and search.

Process Flow:

1. Generate a hypothetical answer using LLM.
2. Embed the hypothetical answer.
3. Use the embedding for similarity search.
4. Retrieve relevant documents based on answer similarity.

Traditional QA and search evaluation metrics must adapt to align with the RAG context. Our evaluation includes the retrieval step performance and the overall quality of answers. So to start, let’s look at the different elements we may want to evaluate in our RAG pipeline:

RAG-Specific Metrics

Retrieval Metrics:

• Context Recall: Context recall is the proportion of ground-truth relevant documents that were retrieved.
• Context Precision (Context Relevancy): Context precision attempts to measure the fraction of retrieved chunks that were truly on-topic and/or useful. If 5 chunks were retrieved but only 2 were on-topic and relevant, we’d say the precision is 0.4.
• Contextual Relevancy Ranking: If your pipeline ranks results, you may want to make sure that the most relevant ones are ranked highest.

Generation Metrics:

• Answer Relevancy: This is a measure of whether the generated answer is relevant to the original query.
• Faithfulness: This measures how faithful the generated response aligns with the context.
• Factual Correctness: This metric indicates how closely the answer adheres to the reference.

Frameworks like DeepEval and RAGAS provide tooling to compute these metrics. For example, DeepEval can use an LLM to score faithfulness by comparing the answer and context.

Monitoring in Production

Beyond offline evaluation, you want to monitor the pipeline in production for anomalies:

• Retrieval logs: Log which documents were retrieved for a given query (Haystack’s pipelines support returning the doc IDs). By sampling the logs, you may be able to notice if unrelated junk is being retrieved or if certain queries are failing to retrieve.
• Feedback loops: If your application supports user feedback (e.g., “Was this answer helpful?” or a rating system), pass that back to analysis. Repeatedly low-rated answers may correlate with certain query types, or some parts of the pipeline may not be functioning properly.
• Latency and Throughput: Monitor the duration of each stage(embedding, retrieval, LLM generation). A sudden increase in retrieval time could indicate an index issue. Generation lag could mean that the LLM is struggling (e.g., the context passed was too long).
• Content Drift: If your knowledge base is subject to change, or user query patterns evolve, keep track of metrics such as relevance over time. You may need to re-index or periodically fine-tune retrievers to keep up.

Deploying a RAG pipeline in production involves considerations of scalability, security, and operational reliability. The table below provides a quick summary of essential strategies and best practices:

Following these strategies, organizations can achieve scalable, secure, and reliable RAG deployments. Containerization, access controls, and monitoring, when built in from the start, will help your system stay resilient as it scales and matures in production.

What is Retrieval-Augmented Generation (RAG), and why should I care? RAG is a powerful AI architecture that pairs classic information retrieval with large language models to generate answers augmented with up-to-date external knowledge. They are more factually accurate and contextually relevant than LLMs alone.

How do I decide between Haystack and LangChain to build a RAG pipeline? Choose Haystack if you need a production-ready, modular, and evaluable pipeline with comprehensive documentation. You can choose LangChain for prototyping new models quickly, building agent-based workflows, or for more flexible integrations with various models/vector stores.

What is the role of the vector database in a RAG pipeline? The vector database stores your documents’ high-dimensional embeddings and enables efficient similarity search to retrieve the most relevant context to use in informing the LLM’s answer.

What are the best vector databases for RAG pipelines, and how do I choose one?

• FAISS: Open-source, blazing fast for in-memory retrieval; ideal for smaller, local, or high-performance setups. Requires manual persistence and scaling.
• ChromaDB: Open-source, easy to use, supports persistence and metadata. Good for local or small-to-medium production workloads.
• Pinecone/Weaviate/Azure Cognitive Search: Managed cloud services with rich features, metadata filtering, scaling, and easy integration. Suitable for large, distributed, or enterprise workloads.
• Rule of thumb: For local/small data, start with FAISS or Chroma; for large-scale or multi-tenant systems, use Pinecone, Weaviate, or similar.

Engineering a production-ready RAG pipeline requires careful consideration of multiple layers. It extends far beyond the basics of retrieval and generation. Success relies on making the right choices at each step, from selecting the appropriate framework (Haystack or LangChain), embedding models, vector databases, chunking strategies, evaluation, deployment practices, and security measures. To take you further down into the weeds of practical steps and best practices, here are some tutorials relevant for building and operating advanced RAG pipelines:

• Interactive Conversations with PDFs Using Langchain
• Containerized RAG Pipelines on GPU Droplets
• Learn How to Build a RAG Application using GPU Droplets
• Beyond Vectors - Knowledge Graphs & RAG Using GradientAI
• How to Choose the Right Vector Database for Your RAG Architecture

By engineering each layer of the pipeline meticulously and following best practices in scalability, monitoring, and security, you can deliver accurate, reliable, and secure RAG solutions. As the landscape of RAG technologies is rapidly evolving, staying flexible and continuously evaluating your pipeline’s components is essential for maintaining an effective and future-proof solution.

• Haystack vs LangChain
• Building a Production-Ready RAG Application: A Practical Guide
• Building Advanced RAG Applications with Haystack and LangChain: A Comprehensive Guide
• How Query Expansion (HyDE) Boosts Your RAG Accuracy
• Advanced RAG: Query Expansion
• RAGXplain: From Explainable Evaluation to Actionable Guidance of RAG Pipelines
• ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems