Designing an Enterprise RAG System: Hybrid Retrieval to Grounded Answers

An operations copilot that answers from a plant's SOPs, maintenance manuals, and incident logs is only useful if every answer is correct and traceable. A bare LLM hallucinates; a naive vector search returns plausible-but-wrong chunks. This article covers the architecture and implementation of a retrieval-augmented generation (RAG) system built for accuracy and auditability — where each answer cites the source paragraph it came from, and quality is measured before anything ships.

Architecture overview

RAG has two pipelines that people often conflate: an offline ingestion pipeline that turns documents into a searchable index, and an online query pipeline that turns a question into a grounded answer. Treating them as separate systems with their own tests is the first design decision.

Chunking is a retrieval decision, not a formatting one

The unit you index is the unit you retrieve. Splitting on a fixed token count shreds tables and procedures mid-step. We chunk structure-aware: respect headings and list boundaries, keep each procedure step whole, and attach metadata (document, section path, revision, equipment tag) to every chunk. A small overlap preserves context across boundaries, and a parent-child scheme lets us retrieve a tight chunk but feed its surrounding section to the model.

Structure-aware chunking with metadata

def chunk_document(doc):
    chunks = []
    for section in split_by_headings(doc):          # respect H1/H2/H3
        for window in pack_by_tokens(section.blocks, # keep steps/rows intact
                                     target=450, overlap=60,
                                     never_split=("table", "list_item")):
            chunks.append(Chunk(
                text=window.text,
                meta={"doc_id": doc.id, "title": doc.title,
                      "section": section.path, "rev": doc.revision,
                      "equipment": doc.tags.get("equipment")}))
    return chunks

Hybrid retrieval beats either method alone

Dense (embedding) retrieval captures meaning but misses exact identifiers — part numbers, error codes, valve tags — that operators actually search for. Sparse (BM25) retrieval nails those literals but misses paraphrase. We run both and fuse the ranked lists with Reciprocal Rank Fusion, which needs no score calibration between the two systems.

Hybrid retrieval with reciprocal rank fusion

def hybrid_search(q, k=40):
    dense = vector_store.search(embed(q), k=k)        # pgvector cosine
    sparse = bm25.search(q, k=k)                      # exact terms / codes
    scores = {}
    for rank, hit in enumerate(dense):  scores[hit.id]  = scores.get(hit.id, 0) + 1/(60+rank)
    for rank, hit in enumerate(sparse): scores[hit.id]  = scores.get(hit.id, 0) + 1/(60+rank)
    fused = sorted(scores, key=scores.get, reverse=True)
    return [store[i] for i in fused[:k]]

Rerank, then ground

Fusion gives a good top-40; a cross-encoder reranker (e.g. bge-reranker) then scores each candidate jointly with the query and keeps the top 5–8. This single step is usually the biggest precision win in the whole pipeline, because it judges actual relevance rather than embedding proximity. Only the reranked, citation-tagged context reaches the model.

Grounded prompt assembly with citation anchors

SYSTEM = ("Answer ONLY from the context. Cite every claim as [n]. "
          "If the context does not contain the answer, say you don't know.")

def build_prompt(q, contexts):
    blocks = [f"[{i+1}] ({c.meta['section']}) {c.text}" for i, c in enumerate(contexts)]
    return [{"role": "system", "content": SYSTEM},
            {"role": "user", "content": f"Context:\n" + "\n\n".join(blocks)
                                         + f"\n\nQuestion: {q}"}]

Evaluation gates every change

RAG quality is multi-dimensional, so we evaluate it that way using a curated question set with known answers and gold passages. Each pipeline change runs against four metrics before it can ship:

• Context recall — did retrieval surface the gold passage at all? (a retrieval problem)
• Context precision — how much retrieved context was actually relevant? (rerank quality)
• Faithfulness — is every claim supported by the cited context? (hallucination rate)
• Answer relevance — does the answer address the question asked?

These run in CI (frameworks like RAGAS or a custom harness). A change that lifts answer relevance but drops faithfulness is rejected — exactly the trade-off you want a gate to catch. This is where RAG meets LLMOps: the evaluation harness is the release gate.

Serving and operations

The LLM runs behind a gateway (self-hosted via vLLM for data-residency, or a managed API) with semantic caching for repeated questions and streaming responses for perceived latency. Ingestion is incremental and idempotent: documents are versioned, and re-indexing on a revision only re-embeds changed chunks. Every query, its retrieved context, and the operator's thumbs-up/down are logged — that trace store is both the audit trail and the next evaluation set.

What this delivers

The payoff is a copilot that returns accurate, cited answers at the point of need and makes decades of institutional knowledge searchable — with a measurable quality bar that every change must clear. The same architecture underpins contract analysis, document intelligence, and any domain where being right and being able to prove it both matter.