Retrieval-Augmented Generation (RAG): A Complete Guide from Basics to Production

Large language models run into two recurring problems. They hallucinate — when they don't actually know an answer, they'll often produce something plausible-sounding rather than correct. And their knowledge is frozen at whatever point their training data ended, so they have no way of knowing what's sitting in a private company database.

Retrieval-Augmented Generation (RAG) addresses both issues at once, essentially by turning a closed-book exam into an open-book one. Rather than asking the model to answer purely from what it learned during training, RAG searches a set of private documents first, pulls out the relevant facts, and hands those to the model to write the actual answer.

1. A working analogy: closed-book versus open-book

Picture a skilled lawyer answering questions about a company's internal policy. Without RAG, that lawyer is working from what they memorized in law school five years ago — if the policy changed yesterday, they either guess or get it wrong. With RAG, an assistant searches the company handbook the moment a question comes in, pulls the exact paragraph that answers it, and hands it to the lawyer with instructions to answer using only that text. The response ends up grounded in something real rather than recalled from memory.

2. How basic RAG works

A standard RAG system runs in three stages.

Ingestion and storage. Before the system can answer anything, the underlying knowledge base — PDFs, Notion pages, SQL tables, whatever it might be — has to be indexed. Long documents get split into smaller chunks, each chunk is run through an embedding model that converts it into a numerical vector representing its meaning, and those vectors are stored in a vector database such as Pinecone, Chroma, or Qdrant.

Retrieval. When a user asks something like "What is our remote work policy?", that question gets converted into the same kind of vector, and the database calculates similarity — cosine similarity is a common choice — to find the chunks whose meaning is closest to the question.

Generation. The retrieved chunks are combined with the original question into a single augmented prompt, which the model reads and uses to write a response grounded directly in the source material rather than in its training data.

3. Basic RAG versus production RAG

A basic RAG setup can be built in a handful of lines of code, but getting it to hold up in production means addressing several edge cases that a simple prototype glosses over.

4. A simple Python prototype

sentence-transformers makes it possible to see the core retrieval math directly, without a full RAG framework wrapped around it:

from sentence_transformers import SentenceTransformer, util

class=class="text-emerald-600 dark:text-emerald-400 font-semibold">"text-slate-400 dark:text-slate-500 italic"># 1. Load a lightweight embedding model
model = SentenceTransformer(class="text-emerald-600 dark:text-emerald-400 font-semibold">'all-MiniLM-L6-v2')

class=class="text-emerald-600 dark:text-emerald-400 font-semibold">"text-slate-400 dark:text-slate-500 italic"># 2. Mock knowledge base (private company docs)
documents = [
    class="text-emerald-600 dark:text-emerald-400 font-semibold">"Employees receive 20 days of paid time off (PTO) annually.",
    class="text-emerald-600 dark:text-emerald-400 font-semibold">"Health insurance benefits kick in on the first day of employment.",
    class="text-emerald-600 dark:text-emerald-400 font-semibold">"Engineers can expense up to $500 per year for home office equipment."
]

class=class="text-emerald-600 dark:text-emerald-400 font-semibold">"text-slate-400 dark:text-slate-500 italic"># 3. Embed the documents into vectors
doc_embeddings = model.encode(documents, convert_to_tensor=True)

class=class="text-emerald-600 dark:text-emerald-400 font-semibold">"text-slate-400 dark:text-slate-500 italic"># 4. User question
query = class="text-emerald-600 dark:text-emerald-400 font-semibold">"How much money can I spend on a home desk setup?"
query_embedding = model.encode(query, convert_to_tensor=True)

class=class="text-emerald-600 dark:text-emerald-400 font-semibold">"text-slate-400 dark:text-slate-500 italic"># 5. Compute similarity scores (retrieval)
scores = util.cos_sim(query_embedding, doc_embeddings)[0]
best_match_idx = scores.argmax().item()

class=class="text-emerald-600 dark:text-emerald-400 font-semibold">"text-slate-400 dark:text-slate-500 italic"># 6. Output retrieved context for the LLM
retrieved_fact = documents[best_match_idx]
print(fclass="text-emerald-600 dark:text-emerald-400 font-semibold">"User Query: {query}")
print(fclass="text-emerald-600 dark:text-emerald-400 font-semibold">"Retrieved Fact: '{retrieved_fact}'")
print(fclass="text-emerald-600 dark:text-emerald-400 font-semibold">"Similarity Score: {scores[best_match_idx]:.4f}")

class=class="text-emerald-600 dark:text-emerald-400 font-semibold">"text-slate-400 dark:text-slate-500 italic"># --- Final step (passed to LLM) ---
class=class="text-emerald-600 dark:text-emerald-400 font-semibold">"text-slate-400 dark:text-slate-500 italic"># Prompt = fclass="text-emerald-600 dark:text-emerald-400 font-semibold">"Context: {retrieved_fact}\nQuestion: {query}\nAnswer:"

User Query: How much money can I spend on a home desk setup?
Retrieved Fact: 'Engineers can expense up to $500 per year for home office equipment.'
Similarity Score: 0.6841

The core idea

RAG closes the gap between a model's static training-time knowledge and whatever data actually needs to be current. Pairing fast search over a document set with the model's ability to write natural language means the resulting assistant can cite where an answer came from, reflect information that changed after training ended, and stay closer to what its source documents actually say.