How RAG Works: From Documents to Better LLM Answers
Large Language Models do not automatically know an organization’s private or up-to-date information. For example, a company may want an AI assistant to answer questions using internal PDFs, Word documents, Excel files, manuals, or policies that were not included in the model’s training data.
Retrieval-Augmented Generation, or RAG, addresses this limitation by retrieving relevant information from a knowledge base before asking the LLM to generate an answer.
What the Article Covers
- Why LLMs need access to external information
- How documents are divided into smaller sections called chunks
- Why a RAG system retrieves relevant chunks instead of entire documents
- How embeddings represent the meaning of text as numerical vectors
- How vectors are stored in a vector database
- How a user’s question is matched with relevant document chunks
- How retrieved content is provided to the LLM as context
A simple RAG pipeline can be represented like this:
Collect Documents
↓
Divide Documents into Chunks
↓
Convert Chunks into Embeddings
↓
Store Vectors in a Vector Database
↓
Convert the User Question into an Embedding
↓
Find Similar Document Chunks
↓
Send Retrieved Chunks to the LLM
↓
Generate an Answer
Instead of sending a complete document to the model, the system retrieves the sections most closely related to the user’s question. These chunks are added to the prompt, giving the LLM useful context for generating its answer.
A more advanced RAG system must also determine which retrieved information should actually be included. This introduces additional operations such as ranking, context selection, and prompt construction.