What is RAG?

The freshness problem

Models don’t know your catalog, yesterday’s policy, or private tickets. Training has a cutoff. Your docs change daily.

Three fixes:

1. Fine-tune — bake knowledge into weights. Good for style, bad for changing facts (needs retraining).
2. RAG — keep docs outside, retrieve at query time. Update index, next query uses it.
3. Long context — paste everything in prompt. Simple for tiny docs, expensive for large ones.

For private/changing data, start with RAG.

How RAG works

Offline: load docs → chunk → embed → store in vector DB.

Online:

1. Embed question (same model)
2. Search vectors (cosine similarity)
3. Take top-K chunks
4. Prompt = question + context + “answer only from context”
5. Generate grounded answer

Why it won

• Separation: update knowledge without touching model
• Cheap updates: re-embed changed chunks
• Bounded cost: send 5×200 tokens, not whole wiki
• Transparency: show sources, cite, audit

When NOT to use basic RAG

• Need behavior change (tone/format) → fine-tune
• Tiny static docs → long context is simpler
• Near-duplicate docs where one word matters → add filters, rerank, keyword search
• Live state (price, inventory) → call source of truth directly

Minimal RAG (conceptual)

# 1. index once
db.add(docs, embeddings=embed(docs))
# 2. per query
q_vec = embed(question)
ctx = db.search(q_vec, k=4)
answer = llm(f"Q:{question}\nContext:{ctx}\nAnswer from context only, else say 'not found'.")

Production adds: parsing, smart chunking, metadata, query rewrite, evals. That’s the next chapter.