A retrieval-augmented generation (RAG) demo takes an afternoon. A RAG system you can trust in front of customers takes weeks — and almost all of that time goes into three things the demo never shows you.
Retrieval is the whole game
The “G” in RAG gets the attention, but the “R” decides whether the answer is right. If retrieval surfaces the wrong chunk, no amount of prompt engineering saves you. In practice this means:
- Chunking that respects meaning, not arbitrary character counts.
- Hybrid search — combining keyword and semantic retrieval — because pure vector search misses exact matches like SKUs and error codes.
- Reranking the top candidates before they ever reach the model.
We spend more time on retrieval quality than on prompting, and it’s not close.
If you can’t measure it, you can’t ship it
The single biggest difference between a hobby project and a production system is evaluation. Before launch we build an eval set of real questions with known-good answers, then score every change against it. It turns “this feels better” into “recall went from 71% to 89%.”
Guardrails are a feature, not a nicety
Production RAG needs to know when it doesn’t know. A confidence threshold, a grounding check that the answer is actually supported by the retrieved sources, and a graceful hand-off to a human are what keep the system from confidently making things up.
The unglamorous parts that matter most
Freshness (how quickly new content becomes retrievable), cost per query, latency budgets, and observability so you can debug a bad answer after the fact. None of it is exciting. All of it is why the system still works six months later.
RAG is a fantastic pattern — we ship it constantly. Just budget your time where the value is: retrieval, evaluation, and guardrails, not the prompt.