Skip to content

Chapter 2.7 — Hybrid search, reranking and metadata filtering

🎯 Objective

Show why pure vector search is rarely enough in corporate production and how hybrid search + reranking + metadata filtering, together, form the defensible retrieval pipeline that appears in almost every mature RAG.

Vector search captures semantic similarity ("contract with an early termination clause" is close to "how to cancel the contract"), but it tends to fail when the query depends on literal strings: product names, error codes, SKUs, internal acronyms, version numbers.

BM25 and lexical variants solve exactly that case: high precision in term matching.

Hybrid search combines the two and fuses the ranking. The most common strategies:

  • Linear combination. score = α · score_vec + (1−α) · score_bm25, with α calibrated on a golden set.
  • Reciprocal Rank Fusion (RRF). Combines rankings, not scores: insensitive to different scales, it is the sensible default in most cases.
  • Late fusion with a reranker. Separate searches, union of the top-k from each, and a cross-encoder reranker decides.
When hybrid helps When hybrid is wasted
Corpus with a lot of technical jargon, acronyms, codes Uniform free text in a single language
Queries that mix natural language with literal terms Purely semantic queries
Regulated domains where the exact term matters Small, homogeneous corpus
Multilingual with a shared glossary Domains where BM25 alone already saturates

🧠 Reranking

Reranking reorders the initial top-k using a model that is more expensive and more precise than the retriever. The dominant pattern is the cross-encoder: the model reads the query and the document together, instead of comparing independent embeddings.

Typical production pipeline:

query -> retrieval top 50–100 -> metadata filter (mandatory) ->
reranker top 5–10 -> context builder -> LLM

Operational notes:

  • A weak reranker is worse than no reranker. Measure and compare with and without the reranker on a golden set before promoting it.
  • Latency and cost grow. The reranker runs over all candidates, not just the winners; a top-50 with a cross-encoder reranker is typically the largest latency consumer in RAG.
  • A reranker compensates for part of the embedding's weakness, but it does not solve missing filters or bad chunking. It is the last capsule in the stack, not the first.

🧠 Metadata filtering

Mandatory filters in enterprise RAG:

  • tenant_id.
  • language.
  • jurisdiction / region.
  • confidentiality_level.
  • product / version.
  • valid_from / valid_until (freshness).

Principles:

  • Filters are applied in storage, before the reranker, before the prompt. When the vector DB or search engine receives the query, it already knows there is a set of non-negotiable filters.
  • A filter failure must be explicit. A query without tenant_id should be rejected, not treated as "all tenants".
  • Filters do not replace isolation. For very sensitive tenants, separate indexes or distinct collections are preferable to filters over a shared index.

🚨 Classic failure modes

  • Hybrid search without calibration: a high lexical weight degrades semantic queries and vice versa.
  • A reranker trained on another domain: it prioritizes irrelevant patterns.
  • Metadata applied after the retriever: the top-k comes contaminated by other tenants and the filter only cleans the residue.
  • Filters based on text fields without normalization: pt-BR and pt_br become different tenants.

📌 Checklist

  • [ ] Are BM25 + vector combined via RRF or calibrated linear fusion?
  • [ ] Was the reranker evaluated against "no reranker" on a golden set?
  • [ ] Do tenant/language/jurisdiction filters run in storage, not in the prompt?
  • [ ] Is a query without mandatory filters explicitly rejected?
  • [ ] Are the reranker's latency and cost monitored at p95?

📚 References