Getting Started with RAG: Grounding LLMs in Your Knowledge Base

Why RAG

LLMs have two chronic problems: knowledge cutoffs and no access to your private data. RAG (Retrieval-Augmented Generation) is straightforward — retrieve relevant passages first, then let the model answer grounded in them. Compared with fine-tuning it is cheaper, traceable and updatable.

The five-step pipeline

1. Load and chunk: convert PDFs, web pages and tables to text, then split into a few-hundred-word chunks. Chunking strategy directly determines retrieval quality — preserve headings and table structure.
2. Embed: turn chunks into vectors with an embedding model. Sentence Transformers is the most common open-source library; pick a multilingual model for mixed Chinese-English content.
3. Store in a vector database: open-source options like Qdrant, or managed services like Pinecone. For small demos, FAISS or sqlite-vec are plenty.
4. Retrieve and rerank: recall candidates by vector similarity, then rerank with a cross-encoder and put only the best passages into context.
5. Generate with citations: hand retrieved passages plus the question to the LLM and require sources. OpenAI Cookbook has plenty of copy-pasteable examples.

Don't skip evaluation

"Feels okay" is not evaluation. RAGAS provides faithfulness, answer relevance and context recall metrics — rerun them after every chunking change and let data decide. For depth, read the RAG survey paper and Pinecone's RAG introduction.

Common pitfalls

• Chunks too large or too small: too large dilutes similarity, too small loses context — 200–500 words is a common starting point.
• Vector-only retrieval: hybrid keyword+vector search is usually noticeably more robust.
• Skipping query rewriting: when users ask colloquially, rewrite the query into a retrieval-friendly form first.
• No fallback: when nothing relevant is retrieved, say "I don't know" instead of hallucinating.