Building RAG systems that survive real production traffic

Start with the question the business actually asks

Most RAG projects begin with a model choice. The better starting point is the question a real user will ask and the document set that can answer it. If the source data is incomplete, outdated, or poorly chunked, no model will save the experience.

Map the top workflows first: support lookups, internal knowledge search, product policy answers, or operations checks. Each workflow needs a different retrieval shape.

Retrieval quality beats prompt cleverness

Grounded answers depend on what gets retrieved. Chunk by meaning, not only by token count. Keep metadata such as product, date, owner, and source type so filters can reduce noise before ranking.

• Prefer smaller, coherent chunks over large mixed passages.
• Store source URLs and document versions with every chunk.
• Re-index when content changes; stale embeddings create confident wrong answers.
• Log the retrieved passages for every response during early rollout.

A minimal production path

A production RAG path is usually retrieve → rank → generate → cite → evaluate. Keep each step explicit so failures are visible.

async function answer(query: string) {
  const hits = await retrieve(query, { topK: 8 });
  const ranked = rerank(query, hits).slice(0, 4);
  if (ranked.length === 0) {
    return { answer: "No grounded source found.", citations: [] };
  }
  const answer = await generate(query, ranked);
  return { answer, citations: ranked.map((hit) => hit.source) };
}

Evaluate before you scale

Create a small golden set of questions with expected sources. Measure retrieval hit rate, citation accuracy, and refusal quality when evidence is missing. Ship only after those checks are stable.

When RAG works, it feels quiet: the answer is useful, the source is clear, and the system fails safely when it should.