Related: The Best AI Tools for Small Business in 2026 (and the Starter Stack That Actually Works)
A RAG demo that works on ten documents often falls apart in production. At scale you meet slow lookups, wrong chunks, confident hallucinations, and an API bill that climbs every week. The gap between a notebook prototype and a system people can trust is an architecture problem, not a model problem. This guide is a reference architecture for retrieval augmented generation that holds up in production, with the numbers that matter at each stage.
Key takeaways
- Split the system into two planes: an offline indexing pipeline and an online query pipeline. They scale and fail in different ways.
- Chunking is the highest leverage decision. Start near 512 to 1024 tokens with 10 to 15 percent overlap.
- Use hybrid search (keyword plus vector), then a reranker. Retrieve about 20, rerank to 5, send 3 to 5 to the model.
- Measure quality with a real eval set. Track faithfulness, answer relevance, context precision, and context recall.
- Most hallucinations are retrieval failures, not model failures. Fix retrieval first.
The reference architecture
Retrieval augmented generation has two halves that are easy to blur but should be designed apart. The indexing path runs offline and prepares your knowledge. The query path runs online when a user asks something. Treating them as one pipeline is where many builds go wrong, because they have different speed needs and different failure modes.
Two planes, not one
The offline plane parses documents, splits them into chunks, turns chunks into embeddings, and writes them to a vector store. It can be slow and batched. The online plane takes a question, retrieves the best chunks, reranks them, builds a prompt, and calls the model. It must be fast and cheap per request. Keep them separate so you can re-index your knowledge without touching live traffic, and scale query throughput without re-running ingestion.
Ingestion and document processing
Good answers start with clean input. This stage parses PDFs, HTML, and tables, strips boilerplate, removes duplicates, and attaches metadata such as source, title, date, and section. Metadata is not decoration. It powers filtered retrieval later, like limiting a search to one product or one date range.
The quiet failure here is stale data. If your documents change and your index does not, the system answers confidently from old facts. Decide your refresh cadence up front and treat re-indexing as a scheduled job, not an afterthought.
Chunking: the highest leverage decision
A chunk is the unit you retrieve. Too large and you bury the answer in noise. Too small and you lose the context that makes it make sense. This single choice moves answer quality more than almost anything else.
- Fixed size is simple and predictable but cuts sentences in half.
- Recursive splits on natural boundaries (paragraphs, then sentences) and is a strong default.
- Semantic chunking starts a new chunk when the meaning shifts, measured by a drop in similarity between neighbouring sentences. It is the most accurate and the most expensive to build.
A safe starting point for prose is 512 to 1024 tokens with 10 to 15 percent overlap. For precise fact lookup, go smaller, around 100 to 256 tokens. A useful pattern is small to big, also called parent document retrieval: search small chunks for precision, then feed the larger parent section to the model for context.
Embeddings and the vector store
Embeddings turn text into vectors so similar meaning sits close together. Pick the embedding model for domain fit, not just leaderboard rank. A model trained on general web text may do poorly on legal or medical language. Higher dimensions (for example 3072 versus 1536) can lift recall but cost more in storage and lookup time.
One rule saves a lot of pain: the embedding model is a lock-in decision. Change it later and you must re-embed the entire corpus, because old and new vectors are not comparable. Choose carefully, then store the vectors in a database built for this, such as pgvector, Qdrant, Pinecone, or Weaviate, with an approximate index like HNSW for fast search at scale.
Retrieval: hybrid search done right
Pure vector search is great at concepts and weak at exact terms. Ask for an error code, a product SKU, or a rare name and dense vectors often miss it. Keyword search (BM25) catches those exact matches. The fix is to run both and combine the results. This is hybrid search, and in production it beats either method alone.
To merge the two ranked lists, Reciprocal Rank Fusion (RRF) is a robust default. It scores each result by its rank in each list, so you do not have to normalise different score scales.
# Hybrid retrieval with Reciprocal Rank Fusion (RRF)
dense = vector_search(query, k=20) # concepts and meaning
sparse = bm25_search(query, k=20) # exact terms and IDs
def rrf(rank, k=60):
return 1 / (k + rank)
scores = {}
for rank, doc in enumerate(dense):
scores[doc.id] = scores.get(doc.id, 0) + rrf(rank)
for rank, doc in enumerate(sparse):
scores[doc.id] = scores.get(doc.id, 0) + rrf(rank)
candidates = sorted(scores, key=scores.get, reverse=True)[:20]For hard cases you can rewrite the query first, or generate a hypothetical answer and search with that (a technique called HyDE). Add these only when plain hybrid search falls short, since they cost extra latency and money.
Reranking
Retrieval is fast but rough. A reranker is a second, more careful pass that reads the query and each candidate together and reorders them by true relevance. A bi-encoder scores documents independently and is fast. A cross-encoder reads the query and document jointly and is more accurate. For the final ranking, the cross-encoder usually wins.
The production rule is simple: retrieve about 20, rerank down to 5, and send 3 to 5 to the model. Reranking 100 or more candidates rarely pays off. Expect a lightweight reranker to add roughly 80 to 120 milliseconds for 20 documents. If that latency hurts, a model like ColBERT is a middle ground. One published pipeline measured context precision climb from about 0.61 with dense search alone, to 0.71 with hybrid, to 0.79 with hybrid plus reranking.
Prompt assembly and generation
Now you build the prompt. Order matters, because models pay less attention to the middle of a long context, a pattern often called lost in the middle. Put the strongest chunks first and last. Inject citations so the answer can point back to sources. Most important, instruct the model to answer only from the provided context and to say it does not know when the context is thin. That single instruction prevents a large share of made up answers.
You will also weigh RAG against simply pasting a lot of text into a long context window. Retrieval grounds the answer in the right facts and keeps cost down. Long context can help refine, but it is slower, pricier, and dilutes attention. For a deeper comparison of teaching a model your data, see our guide on RAG versus fine tuning.
Guardrails and hallucination control
Hallucinations come from two places. The first is retrieval failure: the right chunk was never fetched, so the model falls back on memory. The second is generation noise. Knowing which one you have tells you what to fix.
- Confidence gating: if the top retrieval score is low, refuse rather than guess.
- Enforced citations: require the answer to cite the chunks it used.
- Chain of verification: a second model checks each claim against the retrieved context before the answer ships. In production pilots this has cut hallucination rates by around 40 percent.
Evaluation and observability
You cannot improve what you do not measure. Before launch, build a golden set of about 100 real questions with the chunks that should answer them. Then score retrieval and generation separately, because they fail separately.
| Metric | What it tells you |
|---|---|
| Recall@k, MRR, NDCG | Did retrieval fetch the right chunks, and rank them high |
| Faithfulness | Are the answer claims supported by the retrieved context |
| Answer relevance | Does the answer actually address the question |
| Context precision and recall | Signal to noise of the context, and whether anything needed was missed |
Tools like RAGAS, DeepEval, and Promptfoo automate these scores. Faithfulness is often computed as verified claims divided by total claims, and a score above 0.85 is becoming the baseline for high stakes or regulated use. Treat single study benchmarks as directional, not as universal truth.
Cost control and caching
RAG cost hides in four buckets: the embedding API, vector storage per gigabyte, the reranker per request, and generation tokens. The biggest savings come from caching. Cache embeddings so you never re-embed unchanged text. Cache results for repeated or near identical queries. Cache rerank outputs where inputs repeat. Most overspending comes from over retrieving, re-embedding documents that did not change, and stuffing oversized context into every call.
Where teams go wrong
- Skipping the reranker, then blaming the model for weak answers.
- Picking a chunk size by feel instead of testing it on a golden set.
- Shipping with no evaluation, so quality drifts silently.
- Changing the embedding model without re-indexing.
- Letting an agent loop forever. Cap retrieval steps at about 3 and set a budget per tool.
FAQ
What is the difference between a RAG prototype and a production RAG system?
A prototype retrieves from a few documents with a single vector lookup. A production system adds an offline indexing pipeline, hybrid search, reranking, evaluation with measurable thresholds, guardrails against hallucination, caching, and monitoring, because at scale weak retrieval turns into latency, wrong answers, and runaway cost.
How do you stop a RAG system from hallucinating?
Improve retrieval first with hybrid search and reranking, then add a confidence gate that refuses on weak matches, enforce citations, and use chain of verification where a second model checks each claim against the retrieved context.
How do you know a RAG system is good enough for production?
Build a golden set of about 100 questions with known correct chunks, then score retrieval (Recall@k, MRR) and generation (faithfulness, answer relevance, context precision and recall) with a tool like RAGAS or DeepEval. A faithfulness score above 0.85 is a reasonable bar for high stakes use.
Working with Apex Logic
We design and ship production RAG systems and AI chatbots that stay grounded, fast, and affordable to run. If you want a system that answers from your own data and you can actually trust, see our AI solutions or tell us about your knowledge base for a fixed, fair quote. For more reading, see how to add an AI chatbot to your website.
References
RAGAS (2023 to 2026) - automated RAG evaluation and the faithfulness metric.
Hybrid search and reranking in production RAG - measured precision gains from hybrid plus reranking.
RAG best practices (2026) - chunking, retrieval, and reranking defaults.
Apex Logic project data (2024 to 2026) - production RAG and chatbot builds.
Comments