Every team building with LLMs hits the same wall at the same point: the model does not know about your data.
The training cutoff means it does not know about recent events. The closed training corpus means it does not know about your internal documents, your product data, or your domain knowledge. And hallucination means asking it to pretend otherwise is unreliable.
RAG (Retrieval-Augmented Generation) is the standard architectural answer. The concept is simple: at query time, find documents relevant to the question and give them to the model as context. The engineering is where most production systems fail — in chunking, retrieval quality, evaluation, and the operational patterns that keep a RAG system working as data changes.
This guide covers the complete production RAG stack, from document ingestion through evaluation, with the specific decisions that separate reliable production systems from demos that break on real queries.
How RAG Works
User query
↓
Embed query → query vector
↓
Search vector store → top-K relevant chunks
↓
Build prompt: [system] + [retrieved chunks] + [user query]
↓
LLM generates response grounded in retrieved context
↓
Return response (with citations)
Every step has decisions that substantially affect quality. The most impactful, in order, are: chunking strategy, retrieval method (vector-only vs hybrid), the retrieval quality threshold, and the generation prompt.
1. Document Ingestion and Pre-Processing
The quality of your RAG system is bounded by the quality of your source documents. Pre-processing matters:
Text extraction. PDFs produced by a PDF printer or scanner have different text extraction profiles. pdfplumber and pymupdf (Python) handle most cases; for scanned PDFs, add an OCR step (Tesseract, AWS Textract, Google Document AI).
Cleaning. Strip headers, footers, page numbers, and boilerplate that does not carry semantic content. Normalise whitespace and encoding. Remove or sanitise HTML markup from web-scraped content.
Metadata. For every document, capture and store: source URL or file path, document title, section or chapter, author, last modified date, and any relevant taxonomy tags. Metadata enables filtered retrieval (only search documents from the last 30 days) and citation in responses.
Update handling. Documents change. Your ingestion pipeline must handle updates — re-embedding and re-indexing changed documents while removing stale embeddings. Implement a document fingerprint (hash of content) to detect changes efficiently.
2. Chunking Strategy
Chunking is the highest-leverage configuration decision in most RAG systems. A chunk is what gets embedded, indexed, and retrieved — if a chunk mixes unrelated content, its embedding is semantically ambiguous and retrieval quality suffers.
Strategies and when to use each
| Strategy | How it works | Best for |
|---|---|---|
| Fixed-size with overlap | Split every N tokens, overlap by M tokens between chunks | General-purpose starting point |
| Sentence-based | Split on sentence boundaries | Conversational text, FAQs |
| Paragraph-based | Split on paragraph breaks | Prose documents, articles |
| Section-aware | Split on headings (H1/H2/H3) | Documentation, manuals, structured reports |
| Semantic | Use an NLP model to detect topic shifts | Complex documents with varied content |
Recommended starting point: Recursive character text splitter with 512 tokens, 50-token overlap. Evaluate on your real query set, then tune.
Chunk size trade-off: Smaller chunks (128–256 tokens) have more precise embeddings but may lack context for the LLM to generate a complete answer. Larger chunks (512–1024 tokens) provide more context but dilute the embedding. Many production systems use a "parent-child" chunking pattern: small chunks for retrieval precision, with each small chunk linked to its larger parent chunk that is passed to the LLM for generation context.
3. Embedding Models
The embedding model converts text into a high-dimensional vector. Every chunk is embedded at indexing time; every query is embedded at retrieval time. The model must be the same for both.
Model comparison
| Model | Provider | Dimensions | Context limit | Best for |
|---|---|---|---|---|
| text-embedding-3-small | OpenAI | 1536 | 8191 tokens | English, cost-efficient |
| text-embedding-3-large | OpenAI | 3072 | 8191 tokens | English, highest quality |
| text-embedding-004 | 768 | 2048 tokens | English, competitive quality | |
| embed-english-v3.0 | Cohere | 1024 | 512 tokens | English, strong for search |
| embed-multilingual-v3.0 | Cohere | 1024 | 512 tokens | Multilingual |
| all-MiniLM-L6-v2 | Hugging Face (local) | 384 | 256 tokens | Self-hosted, low latency |
Do not choose based on benchmark scores alone. Evaluate candidate models on your actual documents and queries. A model that performs well on MTEB may underperform on legal documents or technical code if it was not trained on similar content.
4. Vector Stores and Hybrid Search
Vector store options
| Option | Deployment | Best for |
|---|---|---|
| pgvector | Self-hosted (PostgreSQL) | Teams already on PostgreSQL, lower operational overhead |
| Qdrant | Self-hosted or cloud | High performance, rich filtering, open-source |
| Pinecone | Managed cloud | Simplest operations, serverless option |
| Weaviate | Self-hosted or cloud | Multi-tenancy, hybrid search built-in |
| Chroma | Self-hosted | Prototyping, local development |
Hybrid search is worth implementing from the start
Pure vector search misses exact term matches. If a user asks about "GDPR Article 17" or a specific product model number, keyword search finds it reliably while semantic search may miss it because the embedding space does not differentiate between similar-sounding article numbers.
Hybrid search combines both:
from qdrant_client import QdrantClient
from qdrant_client.models import SparseVector, NamedSparseVector
# Dense vector search (semantic)
dense_results = client.search(
collection_name="documents",
query_vector=query_embedding,
limit=20
)
# Sparse vector search (keyword / BM25)
sparse_results = client.search(
collection_name="documents",
query_vector=NamedSparseVector(
name="bm25",
vector=SparseVector(indices=query_token_ids, values=query_weights)
),
limit=20
)
# Merge with Reciprocal Rank Fusion
results = reciprocal_rank_fusion([dense_results, sparse_results], k=5)
5. The Generation Layer
Retrieved chunks are only as useful as the prompt that delivers them to the model.
Prompt structure
System: You are an assistant that answers questions about [domain].
Answer ONLY using the provided context.
If the context does not contain the answer, say "I don't have that information."
Always cite the source document for each claim using [Source: {source}].
Context:
[CHUNK 1] Source: {source_1}
{chunk_1_content}
[CHUNK 2] Source: {source_2}
{chunk_2_content}
[CHUNK 3] Source: {source_3}
{chunk_3_content}
User: {query}
Key principles:
- Explicit instruction to stay within context prevents hallucination against irrelevant retrieved content
- The "I don't know" instruction prevents confident fabrication when retrieval fails
- Citations let users verify claims and build trust in the system
- Pass 3–5 high-quality chunks rather than 10+ lower-quality ones — precision beats recall in the context window
Retrieval quality threshold
Set a minimum similarity score below which retrieved chunks are not passed to the LLM:
results = vector_store.search(query_embedding, top_k=5)
filtered = [r for r in results if r.score >= 0.75]
if not filtered:
return "I don't have relevant information on this topic."
Passing low-relevance chunks to the LLM is worse than passing none — the model may generate a hallucinated response that references the irrelevant content.
6. Evaluation: The Overlooked Step
Most RAG prototypes skip systematic evaluation. This is why they fail in production.
RAGAS metrics
| Metric | What it measures | Target |
|---|---|---|
| Context Precision | Are the retrieved chunks relevant to the query? | > 0.7 |
| Context Recall | Does the retrieved set contain the answer? | > 0.8 |
| Faithfulness | Is the response supported by the retrieved context? | > 0.9 |
| Answer Relevance | Does the response actually answer the question? | > 0.8 |
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision
results = evaluate(
dataset=eval_dataset,
metrics=[faithfulness, answer_relevancy, context_precision],
)
Run evaluation on a held-out set of 100–200 representative queries with known correct answers. Treat the evaluation set as a test suite — run it automatically on every configuration change.
7. Production Architecture
A production RAG system needs:
| Component | Purpose |
|---|---|
| Ingestion pipeline | Document processing, chunking, embedding, indexing |
| Update mechanism | Detect changed documents and re-index |
| Vector store | Embedding storage and search |
| Retrieval service | Query embedding, hybrid search, score filtering |
| Generation service | LLM API calls with grounded prompts |
| Evaluation pipeline | Automated RAGAS metrics on each deploy |
| Observability | Query logs, latency tracking, user feedback |
| Cache layer | Cache embeddings for repeated queries, reduce API cost |
Documents → Ingestion Pipeline → Vector Store
↕
User Query → Query Embedding → Retrieval Service → Generation Service → Response
↑
LLM API (Claude / GPT-4 / etc.)
Common RAG Failure Modes
| Failure | Cause | Fix |
|---|---|---|
| Irrelevant chunks retrieved | Poor chunking, wrong embedding model, no hybrid search | Re-chunk, evaluate embeddings, add BM25 |
| Hallucination despite good retrieval | Prompt does not constrain the model to context | Add explicit grounding instruction and faithfulness check |
| Missing the answer despite relevant documents | Chunk boundary cuts the answer | Increase chunk overlap or use parent-child chunking |
| Slow retrieval at scale | Vector store not indexed appropriately | Add HNSW index, evaluate query latency under load |
| Stale answers | Ingestion pipeline does not process updates | Implement document fingerprint and incremental re-indexing |
Building a production RAG system is a significant engineering undertaking. If you are evaluating whether RAG is the right pattern for your use case or need help designing the architecture, see our Enterprise Software Development capabilities or get in touch.
For the broader context of building AI-powered systems that scale reliably, our guide on Scalable AI Platforms covers the infrastructure patterns that apply across RAG, model serving, and AI application development.