Eliminating Hallucinations in Production RAG Systems: Vector Retrieval, Hybrid Search, and Token Guardrails

Why Naive Vector Search Hallucinates

Retrieval-Augmented Generation (RAG) is celebrated as the solution to LLM hallucinations. The theory is simple: instead of relying on the model's internal memory, retrieve relevant company documents and inject them into the prompt.

Yet in production, naive RAG systems hallucinate constantly. Why?

1. The Semantic Trap: Dense embedding models (e.g. OpenAI text-embedding-3, Cohere) evaluate *meaning*, not *exact tokens*. When a customer searches for "Warranty terms for Model B-240", vector search frequently retrieves "Warranty terms for Model B-340" because the semantic distance is tiny.
2. Forced Synthesis on Empty Retrieval: When a customer asks about a topic not present in your knowledge base, vector algorithms still return the "nearest" chunks, even if their similarity score is abysmal. The LLM tries its best to fabricate a plausible answer.
3. Context Poisoning: Irrelevant chunks clutter the attention window, causing the model to lose track of key facts.

Here is how we eliminate hallucinations in enterprise production environments.

Pillar 01: Hybrid Search (RRF) Architecture

Never rely on vector embeddings alone. Enterprise search requires Hybrid Search: the mathematical combination of dense semantic retrieval with sparse full-text lexical search (BM25 / PostgreSQL tsvector).

PostgreSQL / Supabase Hybrid Search Function

-- Hybrid search combining pgvector cosine similarity with full-text search
CREATE OR REPLACE FUNCTION match_documents_hybrid(
  query_text TEXT,
  query_embedding VECTOR(1536),
  match_count INT,
  rrf_k INT DEFAULT 60
)
RETURNS TABLE (
  id UUID,
  content TEXT,
  metadata JSONB,
  combined_score FLOAT
)
LANGUAGE sql
AS $$
WITH semantic_search AS (
  SELECT id, content, metadata,
         ROW_NUMBER() OVER (ORDER BY embedding <=> query_embedding) AS rank
  FROM knowledge_chunks
  ORDER BY embedding <=> query_embedding
  LIMIT match_count * 2
),
keyword_search AS (
  SELECT id, content, metadata,
         ROW_NUMBER() OVER (ORDER BY ts_rank_cd(to_tsvector('english', content), plainto_tsquery('english', query_text)) DESC) AS rank
  FROM knowledge_chunks
  WHERE to_tsvector('english', content) @@ plainto_tsquery('english', query_text)
  LIMIT match_count * 2
)
SELECT 
  COALESCE(s.id, k.id) AS id,
  COALESCE(s.content, k.content) AS content,
  COALESCE(s.metadata, k.metadata) AS metadata,
  (COALESCE(1.0 / (rrf_k + s.rank), 0.0) + COALESCE(1.0 / (rrf_k + k.rank), 0.0)) AS combined_score
FROM semantic_search s
FULL OUTER JOIN keyword_search k ON s.id = k.id
ORDER BY combined_score DESC
LIMIT match_count;
$$;

Pillar 02: Cosine Distance Thresholding

If the retrieved chunks do not meet a strict similarity threshold (e.g. cosine distance < 0.22 / similarity > 0.78), do not invoke the synthesis model.

Return an explicit, graceful fallback:

*"I apologize, but I do not have verified documentation regarding that specific question in our knowledge base. Would you like me to connect you with an engineering specialist?"*

This single rule eliminates over 80% of hallucinated responses.

Pillar 03: Metadata Filtering & Partitioning

Never dump all company files into a flat vector index. Tag every chunk with rich metadata:

• tenant_id: Enforces company isolation.
• doc_version: Ensures deprecated 2023 pricing is never retrieved over 2026 active pricing.
• access_level: Prevents internal HR files from being retrieved by public customer chat.

Pillar 04: The Dual-Prompt Evaluator Pattern

Before streaming the final response to the user, pass the generated answer and the source chunks through an unprivileged evaluator model:

// lib/rag/evaluator.ts
export async function verifyFaithfulness(
  question: string,
  answer: string,
  sourceChunks: string[]
): Promise<boolean> {
  const prompt = `
You are a strict compliance verification auditor.
Evaluate whether the following ANSWER is 100% supported by the EVIDENCE provided.
If the answer makes ANY claim not directly present in the evidence, output FALSE.
Otherwise output TRUE.

QUESTION: ${question}
EVIDENCE: ${sourceChunks.join("
---CHUNK---
")}
ANSWER: ${answer}

RESULT (TRUE/FALSE):`;

  const result = await callLightweightModel(prompt);
  return result.trim().toUpperCase() === "TRUE";
}

Case Evidence: Standout4Growth RAG

We deployed this grounded architecture for the Standout4Growth RAG Assistant live on standout4growth.com. Combining personal brand principles with precise corporate knowledge retrieval, the system maintains its coaching persona while achieving a 0% verified hallucination rate across thousands of production sessions.

The HarLyn Standard

We build AI systems that businesses can legally and operationally trust.