Building RAG Systems for Production: From Prototype to Scale

Introduction

Retrieval-Augmented Generation (RAG) has become the standard approach for building LLM applications that ground their responses in specific, up-to-date information. While building a basic RAG prototype takes minutes, productionizing it requires careful attention to chunking, embedding, retrieval, and evaluation.

This guide walks through each component of a production RAG system, with practical recommendations and code examples.

Understanding the RAG Pipeline

A production RAG system consists of several stages:

1. Document processing: Loading and cleaning source documents
2. Chunking: Splitting documents into retrievable units
3. Embedding: Converting chunks into vector representations
4. Storage: Indexing vectors in a vector database
5. Retrieval: Finding relevant chunks for a given query
6. Generation: Using retrieved context to generate an answer
7. Evaluation: Measuring retrieval and generation quality

Each stage presents trade-offs that affect accuracy, latency, and cost.

Chunking Strategies

Chunking is arguably the most impactful decision in a RAG system. The goal is to create chunks that are small enough to be precisely retrievable but large enough to provide meaningful context.

Fixed-Size Chunking

The simplest approach splits text into fixed-size chunks with overlap:

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=50,
    separators=["\n\n", "\n", ". ", " ", ""]
)

chunks = splitter.split_text(document_text)

Pros: Simple, predictable, fast Cons: Can split mid-sentence, loses semantic boundaries

Semantic Chunking

Semantic chunking splits text based on meaning rather than character count. It groups sentences with similar embeddings together:

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings

splitter = SemanticChunker(
    OpenAIEmbeddings(),
    breakpoint_threshold_type="percentile",
    breakpoint_threshold_amount=95
)

chunks = splitter.split_text(document_text)

Pros: Preserves semantic coherence, better retrieval accuracy Cons: Slower, more expensive (requires embedding each sentence)

Document-Aware Chunking

For structured documents (markdown, HTML, code), use structure-aware splitters:

from langchain_text_splitters import MarkdownHeaderTextSplitter

md_splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=[
        ("#", "Header 1"),
        ("##", "Header 2"),
        ("###", "Header 3"),
    ]
)

chunks = md_splitter.split_text(markdown_text)

This preserves document structure, keeping headers and their content together.

Choosing an Embedding Model

The embedding model determines how well your retrieval system matches queries to relevant chunks.

Popular Embedding Models

Model	Dimensions	Strengths	Cost
OpenAI text-embedding-3-small	1536	Good general performance	$0.02/1M tokens
OpenAI text-embedding-3-large	3072	High accuracy	$0.13/1M tokens
Cohere embed-v3	1024	Multilingual, strong	$0.10/1M tokens
BGE-large-en-v1.5	1024	Open-source, self-hostable	Free
Nomic Embed	768	Open-source, long context	Free

Recommendations

• Prototyping: OpenAI text-embedding-3-small offers the best balance of cost and performance
• Production at scale: Self-host BGE or Nomic models to eliminate per-token costs
• Multilingual: Cohere embed-v3 handles multiple languages well
• Maximum accuracy: OpenAI text-embedding-3-large with dimension reduction

Vector Database Selection

Pinecone

Pinecone is a fully managed vector database optimized for production use:

from pinecone import Pinecone, ServerlessSpec

pc = Pinecone(api_key="your-api-key")
index = pc.Index("rag-production")

index.upsert(
    vectors=[{
        "id": "chunk-1",
        "values": embedding,
        "metadata": {"source": "doc1.pdf", "page": 1}
    }]
)

results = index.query(
    vector=query_embedding,
    top_k=5,
    include_metadata=True,
    filter={"source": {"$eq": "doc1.pdf"}}
)

Best for: Teams that want zero infrastructure management and need horizontal scaling.

Weaviate

Weaviate is an open-source vector database with strong hybrid search capabilities:

import weaviate

client = weaviate.connect_to_local()

collection = client.collections.get("Document")

collection.data.insert({
    "content": chunk_text,
    "source": "doc1.pdf"
})

results = collection.query.near_text(
    query="How does RAG work?",
    limit=5
)

Best for: Applications needing hybrid (vector + keyword) search and self-hosting.

pgvector

pgvector extends PostgreSQL with vector search, letting you use your existing database:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE documents (
    id SERIAL PRIMARY KEY,
    content TEXT,
    embedding vector(1536),
    metadata JSONB
);

CREATE INDEX ON documents USING ivfflat (embedding vector_cosine_ops);

INSERT INTO documents (content, embedding, metadata)
VALUES ($1, $2, $3);

SELECT content, metadata, embedding <=> $query_vector AS distance
FROM documents
ORDER BY embedding <=> $query_vector
LIMIT 5;

Best for: Applications already using PostgreSQL, smaller datasets, and teams that want to avoid managing separate infrastructure.

Retrieval Optimization

Hybrid Search

Combining vector and keyword search improves retrieval accuracy significantly:

from langchain_community.retrievers import PineconeHybridSearchRetriever

retriever = PineconeHybridSearchRetriever(
    index=index,
    embeddings=OpenAIEmbeddings(),
    sparse_encoder=sparse_encoder,
    top_k=5,
    alpha=0.5  # Balance between vector (0) and keyword (1) search
)

Re-ranking

Retrieve more candidates with initial search, then re-rank with a cross-encoder:

from langchain_cohere import CohereRerank

compressor = CohereRerank(top_n=3)
compression_retriever = ContextualCompressionRetriever(
    base_compressor=compressor,
    base_retriever=retriever
)

docs = compression_retriever.invoke(query)

Query Transformation

Improve retrieval by transforming the user’s query before searching:

from langchain_openai import ChatOpenAI

def hyde_query(query: str) -> str:
    """Use HyDE - generate a hypothetical answer to improve retrieval."""
    llm = ChatOpenAI(model="gpt-4o")
    prompt = f"Generate a detailed answer to this question: {query}"
    return llm.invoke(prompt).content

# Use the hypothetical answer as the search query
enhanced_query = hyde_query(user_query)
docs = retriever.invoke(enhanced_query)

Evaluation

Retrieval Evaluation

Measure whether your system retrieves the right documents using metrics like context precision and recall:

from ragas import evaluate
from ragas.metrics import context_precision, context_recall, faithfulness

eval_results = evaluate(
    dataset=eval_dataset,
    metrics=[context_precision, context_recall, faithfulness]
)

print(f"Context Precision: {eval_results['context_precision']:.2f}")
print(f"Context Recall: {eval_results['context_recall']:.2f}")
print(f"Faithfulness: {eval_results['faithfulness']:.2f}")

Key Metrics to Track

• Context Precision: Are the retrieved chunks actually relevant?
• Context Recall: Are all necessary chunks being retrieved?
• Faithfulness: Does the generated answer stick to the retrieved context?
• Answer Relevance: Is the answer actually addressing the question?

Production Considerations

1. Cache embeddings: Store embeddings to avoid re-computing them
2. Implement rate limiting: Protect your embedding and LLM APIs
3. Add citations: Always trace answers back to source chunks
4. Monitor costs: Track token usage across embedding and generation
5. Version your index: Allow rollback when chunking or embedding changes

Conclusion

Building a production RAG system requires thoughtful decisions at every stage. Start with simple chunking and a managed vector database, then optimize based on evaluation metrics. The biggest improvements typically come from better chunking, hybrid search, and re-ranking—not from switching to a larger LLM. Measure everything, iterate on weak points, and let evaluation metrics guide your decisions.