RAG Without the Hype: Retrieval That Actually Helps

Retrieval-augmented generation has a branding problem. The name makes it sound like an architecture you adopt. It's really one idea: before you ask the model a question, go fetch the relevant text and paste it into the prompt. Everything else is plumbing around that sentence.

You reach for RAG when the model needs to answer using knowledge it wasn't trained on and that changes too often, or is too large, to bake into a prompt. Your company's internal docs. A user's own documents. A product catalog that updates hourly. The model supplies the language ability; retrieval supplies the facts.

When you don't need it

Plenty of teams build a vector pipeline they didn't need. If your entire knowledge base fits comfortably in the context window, just put it in the prompt. Modern long-context models handle a lot of text in one call, and prompt caching means the repeated portion is cheap after the first request. A 40-page policy document doesn't need a retrieval system; it needs to be pasted in and cached.

RAG earns its place when the corpus is too big to fit, or when only a small, query-dependent slice is relevant each time, or when you need citations back to source documents. If none of those is true, skip it and save yourself an index to maintain.

The actual pipeline

There are two phases. Indexing happens ahead of time. Retrieval happens per query.

Indexing: split your documents into chunks, turn each chunk into a vector with an embedding model, and store the vectors alongside the original text.

# one-time (and on document updates)
for doc in corpus:
    for chunk in split(doc, target_tokens=500, overlap=50):
        vec = embed(chunk.text)              # embedding model call
        db.insert(vector=vec, text=chunk.text, meta=chunk.source)

Retrieval: embed the user's question with the same model, find the nearest chunks, and hand them to the generation model as context.

def answer(question):
    q = embed(question)
    hits = db.search(q, top_k=6)
    context = "\n\n".join(f"[{h.meta}] {h.text}" for h in hits)
    return llm(
        system="Answer only from the context. Cite the source in [brackets]. "
               "If the answer isn't there, say so.",
        user=f"Context:\n{context}\n\nQuestion: {question}")

That is a complete RAG system. You can build it in an afternoon. The reason RAG has a reputation for being hard is that the naive version works in the demo and disappoints in production, and closing that gap is where the real work lives.

You don't need a dedicated vector database

If you're on Postgres, pgvector handles retrieval for millions of chunks without a new piece of infrastructure. Keeping your vectors next to your relational data means you can filter by tenant, permissions, or recency in the same query, which you will need to do.

-- pgvector, with a metadata filter you'll actually use
SELECT text, source
FROM chunks
WHERE tenant_id = $1 AND updated_at > now() - interval '90 days'
ORDER BY embedding <=> $2   -- cosine distance to the query vector
LIMIT 6;

Dedicated vector stores buy you scale and some convenience at very large volumes. For most applications, that's premature. Start with what your database already offers.

Chunking is where quality is won or lost

The single biggest lever on RAG quality is how you split documents, and it gets almost no attention. Chunk too small and each piece lacks the context to be meaningful; a retrieved sentence about "the 30-day window" is useless if the paragraph naming what window is in a different chunk. Chunk too large and you dilute the relevant signal and waste context budget.

Split on natural boundaries, not fixed character counts. Markdown headings, function definitions, and paragraph breaks carry meaning; splitting at exactly 1000 characters severs sentences mid-thought. Add a small overlap so a fact near a boundary appears in both neighbors. And keep the source metadata on every chunk so you can cite it and, when a document changes, re-index just that document rather than the whole corpus.

Pure vector search isn't enough

Embeddings capture semantic similarity, which is exactly wrong for some queries. Ask for an exact error code, a product SKU, or a specific person's name, and vector search happily returns things that are about the same topic while missing the literal match. This is the most common reason a RAG system feels dumb.

The fix is hybrid retrieval: run a keyword or full-text search alongside the vector search and combine the results. Postgres gives you full-text search natively, so you can do both in one system.

# retrieve by both, then merge
vec_hits = vector_search(q, top_k=20)
kw_hits  = fulltext_search(question, top_k=20)
candidates = reciprocal_rank_fusion(vec_hits, kw_hits)   # simple, robust
top = candidates[:6]

For a further quality bump, add a reranking step: pull 20 candidates cheaply, then use a cross-encoder reranker to score each against the query and keep the best 6. Rerankers are far more accurate at judging relevance than raw vector distance because they look at the query and document together. They add latency and cost, so add one only after you've confirmed retrieval is your bottleneck.

Debug retrieval and generation separately

When a RAG answer is wrong, there are two suspects and you must isolate them. Either the right chunk was never retrieved, or it was retrieved and the model ignored or misread it. Log the retrieved chunks for every query. If the answer to the question wasn't in what you fetched, no prompt engineering will save you; the problem is retrieval, and you fix it with chunking, hybrid search, or reranking. If the right chunk was there and the model still got it wrong, the problem is your prompt or the model, and retrieval is a red herring. Teams waste days tuning the wrong half because they never looked at what was actually retrieved.

The failure modes to expect

Retrieval returns plausible-but-irrelevant chunks and the model dutifully answers from them, confidently and wrongly. Guard against this by instructing the model to say when the context doesn't contain the answer, and by keeping top_k modest so you're not stuffing marginal chunks into the prompt.

Stale indexes are a quiet killer. If your source documents change and you don't re-embed, the system answers from old data with total confidence. Tie re-indexing to your content updates, not a weekly cron you'll forget.

Embedding model mismatch will silently ruin everything: you must embed queries and documents with the same model, and if you ever upgrade the embedding model you have to re-index the entire corpus. Vectors from different models are not comparable.

None of this requires a framework. LangChain or LlamaIndex can accelerate a prototype, but they also hide the two things you most need to see: what got retrieved and what got sent to the model. Build the plain version first so you understand your own pipeline, then adopt tooling if it genuinely saves you work. RAG is not a product you install. It's a pattern you tune to your data, and the tuning is the job.