Large language models are powerful, but they have a fundamental limitation: they only know what was in their training data. When a user asks about your company’s internal docs, today’s pricing, or last week’s support tickets, the model either guesses or refuses. Retrieval-Augmented Generation (RAG) solves this by giving the model access to external knowledge at inference time.
The core idea is simple: instead of relying on the model’s frozen training knowledge, you search a database of your own documents, pull the most relevant chunks, and hand them to the model as context before it answers. The model then grounds its response in those retrieved facts rather than hallucinating.
How RAG Works
A RAG system has two phases. Ingestion happens once: you split your documents into chunks, convert each chunk into a vector embedding using a model like OpenAI’s text-embedding-3-small, and store those vectors in a vector database like Pinecone or Weaviate. Retrieval happens at query time: you embed the user’s question, find the closest document chunks by vector similarity, and pass them to the LLM as context.
Choosing a Vector Database
For managed deployments, Pinecone and Weaviate are the most popular choices — they handle scaling, replication, and indexing for you. For self-hosted setups, Qdrant and Chroma are lighter and run well in a single container. The choice mostly comes down to whether you want to operate infrastructure and how much data you are storing. For most teams starting out, a managed service removes a category of problems you do not need to own.
Common RAG Pitfalls
- Poor chunking. Split documents too small and you lose context; too large and retrieval becomes noisy. Aim for 500–1000 tokens with overlap.
- Bad embeddings. A cheap embedding model produces poor retrieval. This is not the place to save money.
- No reranking. Vector similarity returns approximate matches. A cross-encoder reranker dramatically improves precision.
- Ignoring metadata. Store source, date, and document type as metadata so you can filter before retrieval.
RAG is the single highest-ROI pattern in applied AI today. If you have documents your team constantly searches, RAG turns your LLM from a generalist into a domain expert overnight.