Sub-150ms RAG Pipelines: Hybrid Vector Search & Re-Ranking at Scale
A deep dive into our production RAG stack. We analyze the latency and precision tradeoffs between dense cosine embeddings and BM25 sparse keyword indices, and how cross-encoder reranking eliminates data hallucination.
- Dense vectors alone miss exact keyword matches like SKUs and transaction IDs
- Reciprocal Rank Fusion (RRF) delivers 23% higher precision on technical corpora
- Semantic caching in Redis resolves 44% of repetitive corporate queries in < 12ms
- Cross-encoder reranking eliminates irrelevant context before prompt assembly
Standard vector databases with dense embeddings frequently fail in enterprise domains where exact identifiers, legal citation numbers, model SKUs, or customer IDs must be matched precisely. Dense vector embeddings map semantic intent rather than exact character tokens.
To overcome this limitation, our architecture deploys a hybrid retrieval pipeline. We run dense vector similarity (via Qdrant or pgvector) concurrently with BM25 sparse lexical search. The resulting candidate documents are merged using Reciprocal Rank Fusion (RRF).
Before sending context to the generator LLM, candidate chunks pass through a cross-encoder reranker (such as Cohere Rerank or BGE-Reranker-Large). This reduces prompt token bloat by 70% and drives P95 retrieval latency down to under 150 milliseconds.
def reciprocal_rank_fusion(dense_ranks: dict, sparse_ranks: dict, k: int = 60):
scores = defaultdict(float)
for doc, rank in dense_ranks.items():
scores[doc] += 1.0 / (k + rank)
for doc, rank in sparse_ranks.items():
scores[doc] += 1.0 / (k + rank)
return sorted(scores.items(), key=lambda x: x[1], reverse=True)OXELVO Engineering Group
The Systems Architecture & Applied AI team at OXELVO. We engineer resilient distributed backends, deterministic multi-agent state machines, and high-performance commercial software.
Discuss this architecture with OXELVO Engineering
Interested in implementing deterministic agents, high-throughput RAG, or multi-tenant database isolation in your business?
Start Technical Consultation