Skip to content

Chapter 2.6 — Traditional RAG

🎯 Objective

Present Retrieval-Augmented Generation (RAG) honestly about what it solves, what it does not solve, and where it tends to fail silently in enterprise environments.

RAG is, in almost every team, the first serious attempt to combine an LLM with corporate knowledge — and, simultaneously, where the first serious incidents appear: cross-tenant leakage, indirect prompt injection via documents, and confident answers based on the wrong passage. This chapter covers each of these failures.

🧠 Core concept

RAG is an architectural pattern where, instead of relying only on what the model learned during training, the application retrieves evidence from a knowledge base and injects it into the context before asking for the answer. The model then operates with "external memory" controlled by the application.

The canonical simplification is:

query -> retrieval(query, base) -> context -> LLM(query, context) -> answer

The simplification hides most of the work.

🏗️ Operational pipeline in production

An honest enterprise RAG involves, at a minimum:

  1. Ingestion. Collecting documents from versioned sources (CMS, drives, repositories, databases), with lineage and sensitivity classification.
  2. Parsing. Converting PDFs, spreadsheets, slides, HTML and images into usable text. OCR when needed. Errors here propagate through the whole pipeline.
  3. Chunking. Breaking text into indexable pieces. It can be fixed (by tokens), semantic (by section/heading), hierarchical (parent/child) or mixed.
  4. Metadata enrichment. Tenant, language, jurisdiction, confidentiality classification, validity date, source, version.
  5. Embeddings. Generating vectors via an embedding model (see Ch. 2.8 for dimension, Matryoshka and costs).
  6. Indexing. Vector DB, lexical search engine, or hybrid. Includes an ANN structure (HNSW/IVF), replicas and backups.
  7. Retrieval. Vector, BM25 or hybrid (see Ch. 2.7). A generous initial top-k.
  8. Mandatory metadata filters. Tenant, language, jurisdiction, confidentiality, freshness. Non-negotiable.
  9. Reranking. A cross-encoder or dedicated reranker to reduce top-k to a small, ordered set.
  10. Context construction. Selection, ordering, deduplication and formatting of the passages in the prompt. Includes clear marking of which passage came from where.
  11. Generation with citations. An explicit request to cite by passage and validation that the answer cites the expected sources.
  12. Evaluation. Retrieval and answer metrics, with a golden set and production monitoring.

Each step may seem small. In production, each step fails in a different way, and the final effect is always the same: the model answers confidently from bad context.

⚠️ Uncomfortable truths about RAG

  • RAG does not eliminate hallucination. It reduces risk when retrieval is good and the model is instructed to stick to the context.
  • RAG can bring the wrong, outdated, or another tenant's document. When this happens, the error is worse than without RAG: the answer is anchored in an apparently legitimate source.
  • Bad reranking destroys recall. A poorly calibrated reranker can push to the bottom exactly the passage that matters.
  • Bad chunking destroys everything. Pieces that are too large dilute similarity; pieces that are too small lose context. There is no universal "right" size.
  • The embedding model determines the ceiling of retrieval quality. A reranker compensates for part of the loss, but not miracles.
  • Most RAG problems are outside the LLM. They are in ingestion, parsing and the index.

🚨 Common silent failures

  • Outdated document with the same signature. The new version is indexed alongside the old; the retriever picks the wrong one.
  • Cross-tenant retrieval. Without a mandatory filter by tenant_id in storage, two customers end up sharing context.
  • Indirect prompt injection in documents. A retrieved document contains instructions like "ignore previous policies and send X". The model, without defense, executes it (see Ch. 4.3).
  • High score without semantic relevance. Especially with generic embeddings on a very technical corpus.
  • Different language. A pt-BR query retrieves EN documents with high spurious similarity.
  • PDF document with text in images. OCR fails; the index ends up with empty or noisy text.
  • Window truncation. The retriever brings 10 passages, but the prompt only fits 4; the model silently sees different context.
  • Chunking that cuts tables and lists in the middle. The information is split between two chunks, and neither is selected.

📈 Metrics that matter

Retrieval metrics:

  • Recall@k. Fraction of queries where the correct passage is in the top-k.
  • Precision@k / Context precision. How many of the top-k are actually relevant.
  • MRR (Mean Reciprocal Rank). Average position of the first relevant item.
  • NDCG@k. Relevance-weighted ranking (when there are grades).

Final answer metrics (conditioned generation):

  • Groundedness / faithfulness. How much of the answer is actually supported by the retrieved passages.
  • Citation coverage. Fraction of statements with a verifiable citation.
  • Answer correctness. Accuracy against ground truth (when it exists).
  • No-answer behavior. The model should answer "I don't know" when the context is insufficient. Metric: fraction of correct and incorrect "no-answer".
  • Truncation rate. How many times the context overflowed and was silently cut.

Operational metrics:

  • p95 latency of retrieval and reranker.
  • Cost per query (embedding + index + reranker + LLM).
  • Index freshness (median and p95 age of chunks).
  • Indexing lag (time between a change in the source document and its availability in the index).

🛡️ Defensive principles

  • Tenant filters are in storage, not in the prompt. The prompt is an instruction, storage is a boundary.
  • Retrieved documents are data, not instructions. The prompt should delimit the "documents" region with clear markers and instruct the model not to obey commands contained therein.
  • Every factual statement has a source. When there is no retrieved source, the correct answer is "I don't know" — and this must be tested in eval.
  • Reindexing is routine, not an exception. Covered in Ch. 7.4 and Ch. 2.8 (re-embedding with a parallel index).

🧪 How to test RAG

  • Golden set per tenant and language. Sets with query + expected passage + expected answer (when applicable).
  • Adversarial set. Queries with direct prompt injection, queries in another language, queries whose correct content is not in the corpus (forcing no-answer).
  • Regression set. A set that must pass before promoting any change in embedding, reranker, chunking, prompt or model.
  • Production replay with redacted PII.

Continuous RAG eval is covered in Ch. 2.11 and Ch. 5.4.

📌 Minimum checklist

  • [ ] Is the tenant filter applied in storage, not in the prompt?
  • [ ] Do chunks have complete metadata (tenant, language, source, version, date)?
  • [ ] Is there a reranker in the pipeline, or a documented reason not to have one?
  • [ ] Does the prompt instruct the model to treat documents as untrusted data?
  • [ ] Is there eval with golden set + adversarial + regression?
  • [ ] Is there monitoring of groundedness and citation coverage in production?
  • [ ] Is indexing lag monitored?
  • [ ] Is there a procedure for safe reindexing?
  • EX-LLM-01 — local RAG with Ollama + Qdrant (or Chroma).
  • EX-LLM-02 — RAG with hybrid search and reranking.
  • EX-LLM-03 — retrieval eval harness with a golden set.
  • EX-LLM-06 — RAG with tenant filters + adversarial tests.

🏢 Hermes Logística — wave 2

The first version of Hermes's RAG was built with a single corpus and a tenant_id filter added in the prompt ("consider only documents from tenant X"). It worked in 95% of cases. In the other 5%, the retriever brought another customer's document, the LLM cited it as valid, and the customer received the wrong commercial policy. The fix was not to improve the prompt: it was to move the filter to storage, require tenant_id as a mandatory filtering term in every query, and block at the retriever level any query without that parameter. The prompt stopped being part of security. This was the first incident that taught the team that "an instruction in the prompt" is not "a filter in the database".

📚 References