RAG in Production vs RAG in Tutorials

The tutorial version of RAG takes about 30 minutes to build.

Load some docs. Chunk them. Embed them. Store in a vector DB. Query at runtime. Return context to the LLM. Done.

I've built a few of these tutorials. They're clean. They work. And they're basically useless as a reference for what you'll actually face in production.

Here's the gap.

Chunk size is not a knob you set once

Every tutorial picks a chunk size, usually 512 or 1000 tokens, and moves on. In production, chunk size is one of the most important decisions you'll make, and the right answer changes based on your content type.

Technical documentation chunks differently from legal contracts. A code snippet needs to stay together with its surrounding context. A paragraph from a legal doc might be self-contained.

What I do now: I run a small set of representative queries against different chunk sizes and measure retrieval relevance manually. Takes an afternoon. Saves weeks of debugging why your retrieval feels off in ways you can't quantify.

And no, there's no universal answer. Anyone telling you "always use 512 tokens" hasn't shipped to production yet.

The embedding model matters more than people think

Tutorials default to whatever the standard embedding model is at the time. It works well enough to demo. It might not work well enough for your domain.

If you're building RAG for a specific domain — medical, legal, financial, code — a domain-specific embedding model or a fine-tuned one will often significantly outperform a general-purpose model on retrieval.

I've seen cases where switching the embedding model improved retrieval accuracy by 15-20% on domain-specific queries. That's a big number. Test before you commit.

Metadata filtering is not optional

Tutorials query the vector DB and return the top-k results. That's it.

In production, you need metadata filtering or your retrieval becomes a mess at scale. Example: you have 50,000 documents from different departments. A user asks a question clearly about HR policy. If you retrieve purely by semantic similarity, you'll get a mix of HR docs, finance docs, and maybe some IT policy docs that happen to use similar language.

Add metadata at ingestion time: department, document type, date. Filter at query time. Now you're retrieving from a meaningful subset.

results = vectorstore.similarity_search(
    query=user_query,
    k=10,
    filter={
        "department": user_context.department,
        "doc_type": "policy",
        "last_updated_after": "2024-01-01"
    }
)

The LLM ignores your context sometimes

This one is frustrating.

You've retrieved good context. You've passed it to the LLM. The LLM answers from its training data anyway, ignoring the context you provided.

It happens more than you'd expect, especially with longer contexts where your relevant chunk is buried in the middle. A few things that help:

• Put the most relevant context first, not last. LLMs pay more attention to the beginning and end of context windows.
• Be explicit in your system prompt: "Answer only using the provided context. If the context doesn't contain the answer, say so."
• Add a citation requirement. "Reference the specific section you're drawing from." This forces the model to actually engage with the context.

Re-ranking is worth it

Most tutorials stop at retrieval. You get back the top-k most similar documents and send them all to the LLM.

A re-ranker is a second model that takes your retrieval results and re-orders them by relevance to the specific query. It's more expensive computationally, but it meaningfully improves the quality of what gets into context.

I've added a re-ranking step to almost everything I build now. The tradeoff is about 100-200ms of extra latency. Usually worth it.

Eval is the thing you can't skip

Tutorials measure nothing. They just show you output that looks good.

In production, you need to measure. At minimum: retrieval precision, answer faithfulness, and answer correctness.

Build your eval set before you start tuning. Otherwise you're optimizing for vibes.

The gap is coordination, not technology

None of this is secret knowledge. The gap between tutorial RAG and production RAG isn't technical mystery. It's that tutorials optimize for clarity of explanation. Production optimizes for reliability at scale with real data and real users doing unexpected things.

Build for the unexpected things. That's where production actually lives.