Building Your First Retrieval-Augmented Generation (RAG) Pipeline: A Hands-on Guide

Large Language Models (LLMs) have transformed how we interact with technology. However, out-of-the-box LLMs suffer from two major limitations: knowledge cut-offs and hallucinations. When asked about proprietary enterprise data, recent events, or specific internal documentation, an LLM will either confidently deliver incorrect answers or state that it does not possess the information.

To bridge this gap, engineers leverage Retrieval-Augmented Generation (RAG). Instead of retraining or fine-tuning expensive models, RAG retrieves relevant documents from an external data source and injects them into the prompt as context.

In this step-by-step LangChain tutorial, you will learn how to build a RAG pipeline from scratch. We will cover document ingestion, text chunking, embedding generation, semantic search, and context-aware response generation using LangChain, OpenAI, and ChromaDB.

---

Why RAG Matters for Enterprise Data

In enterprise environments, data is dynamic, proprietary, and highly confidential. Training a custom LLM from scratch is economically unviable for most businesses, and fine-tuning often fails to guarantee factual accuracy.

A retrieval augmented generation architecture solves these issues by acting as an "open-book" exam for the LLM. The pipeline follows a clean, three-step paradigm:

1. Retrieve: When a user asks a question, the system queries a vector database to find documents semantically related to the query.
2. Augment: The retrieved documents are appended to the user’s original prompt alongside system instructions.
3. Generate: The LLM reads the context-rich prompt and generates a factually accurate, grounded response.

By decoupling knowledge storage (the vector database) from reasoning (the LLM), you gain complete control over data security, access permissions, and real-time document updates.

---

Architectural Blueprint of a RAG Pipeline

Before diving into the code, let's look at the logical architecture of a production-ready RAG pipeline:

[Document Ingestion] ──> [Text Chunking] ──> [Embedding Model] ──> [Vector Store (ChromaDB)]
                                                                           │
                                                                           ▼
[User Query] ──────────────────────────> [Query Embedding] ──> [Semantic Search (Top-K)]
                                                                           │
                                                                           ▼
[Synthesized Response] <── [LLM Generation] <── [Augmented Prompt (Query + Context)]

This workflow ensures that only the most relevant text segments are processed by the LLM, keeping latency low and API token costs optimized.

---

Prerequisites and Environment Setup

To follow this tutorial, you will need:

• Python 3.9 or higher installed on your machine.
• An OpenAI API key with access to GPT-4o or GPT-3.5-turbo.
• Basic familiarity with Python and command-line interfaces.

Let's start by installing the required Python packages. Open your terminal and run the following command:

pip install langchain langchain-openai langchain-community chromadb tiktoken pypdf

Next, set up your OpenAI API key as an environment variable in your Python script or system environment:

import os
import getpass

if "OPENAI_API_KEY" not in os.environ:
    os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API Key: ")

---

Step-by-Step Implementation

In this vector database tutorial section, we will build a complete RAG system using a sample text document representing internal company policy.

Step 1: Document Loading and Chunking

LLMs have limited context windows. If you pass an entire 100-page employee handbook into the prompt, you will encounter high costs, slow response times, and degraded accuracy. To prevent this, we divide our documents into smaller, coherent fragments called chunks.

We will use the RecursiveCharacterTextSplitter, which splits documents by looking at common separators like double newlines, single newlines, and spaces, preserving semantic structure where possible.

from langchain_text_splitters import RecursiveCharacterTextSplitter

# Sample internal company knowledge
enterprise_data = """
Deepskilling Employee Handbook 2024.
1. Remote Work Policy: Employees are allowed to work remotely up to 3 days a week. Core collaborative hours are from 10:00 AM to 3:00 PM EST.
2. Learning & Development Budget: Every engineer receives an annual stipend of $2,500 to spend on courses, certifications, and technical conferences.
3. Equipment Allocation: New hires receive an M3 MacBook Pro, a 27-inch external monitor, and a $300 home-office setup allowance.
4. Health and Wellness: Premium health insurance is covered 100% for employees and 50% for dependents, starting from day one.
"""

# Initialize the text splitter
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=150,
    chunk_overlap=30,
    length_function=len,
)

# Split the source document
chunks = text_splitter.create_documents([enterprise_data])

print(f"Created {len(chunks)} text chunks.")
for i, chunk in enumerate(chunks):
    print(f"Chunk {i+1}: {chunk.page_content}\n")

Why Chunk Overlap Matters: The chunk_overlap parameter ensures that context is not lost at the boundary of a split. A small overlap keeps sentences from being cut in half, maintaining continuity for downstream semantic searching.

---

Step 2: Vector Embeddings and Database Population

To perform semantic search, text must be converted into vector embeddings—high-dimensional numerical representations that capture the meaning of the words. If two sentences are conceptually similar, their vector representations will sit close together in vector space.

We will use OpenAI’s text-embedding-3-small model to generate embeddings and load them into ChromaDB, an open-source, lightweight vector database designed for AI application workflows.

from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma

# Initialize OpenAI Embeddings model
embeddings_model = OpenAIEmbeddings(model="text-embedding-3-small")

# Initialize and populate ChromaDB vector store
vector_store = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings_model,
    persist_directory="./chroma_db"
)

print("Vector database populated and saved locally.")

---

Step 3: Setting Up the Semantic Search Retriever

With our vector database populated, we need a mechanism to query it. In LangChain, this is handled by a Retriever. The retriever uses cosine similarity to find the top $K$ document chunks most relevant to a user's question.

# Convert the vector store into a LangChain retriever
retriever = vector_store.as_retriever(
    search_type="similarity",
    search_kwargs={"k": 2}  # Retrieve the top 2 most relevant chunks
)

# Test the retriever
query = "How much can I spend on learning certifications?"
retrieved_docs = retriever.invoke(query)

print(f"Query: {query}\n")
for doc in retrieved_docs:
    print(f"Retrieved Context: {doc.page_content}")
    print("-" * 40)

---

Step 4: Constructing the End-to-End RAG Chain

Now, we will assemble the final RAG pipeline. We will use LangChain Expression Language (LCEL) to chain the components together: the retriever, a prompt template, our LLM (gpt-4o-mini), and an output parser.

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough

# Initialize the LLM
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

# Define the system and user instructions inside a ChatPromptTemplate
rag_prompt = ChatPromptTemplate.from_template("""
You are an expert HR assistant for Deepskilling. Use only the following pieces of retrieved context to answer the question. 
If you do not know the answer based on the context, state that you do not have enough information. Do not make up answers.

Context:
{context}

Question: 
{question}

Answer:
""")

# Helper function to format retrieved documents
def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)

# Construct the LCEL RAG Chain
rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | rag_prompt
    | llm
    | StrOutputParser()
)

# Execute the chain
user_question = "What computer hardware do I get as a new employee, and is there a budget for home setups?"
response = rag_chain.invoke(user_question)

print(f"Question: {user_question}\n")
print(f"Response:\n{response}")

Output Evaluation

When executed, the system successfully searches the vector database, extracts the document chunk containing "Equipment Allocation," feeds it to the LLM, and produces a highly accurate answer, such as:

As a new employee, you will receive an M3 MacBook Pro and a 27-inch external monitor. Additionally, there is a $300 home-office setup allowance.

---

Production Best Practices for Engineering Teams

While the steps above build a working pipeline, moving a retrieval augmented generation system to production requires careful planning. Here are three critical engineering optimizations:

1. Document Pre-processing and Clean Up

Raw files like PDFs, Word files, and Confluence pages contain structural noise (headers, footers, HTML tags, page numbers). Before running embeddings, write robust preprocessing scripts to strip out repetitive boilerplates. Clean data directly correlates with retrieval quality.

2. Hybrid Search and Reranking

Simple vector (dense) retrieval sometimes misses exact keyword matches (e.g., product SKUs or acronyms). Implement hybrid search—which combines BM25 keyword matching with dense vector embeddings—and apply a reranking model (like Cohere Rerank) to sort the top retrieval results before passing them to the LLM.

3. Continuous Evaluation

RAG systems can degrade over time as documents change. Use frameworks like Ragas or TruLens to evaluate key metrics:

• Faithfulness: Is the LLM response derived only from the context? (Reduces hallucinations)
• Answer Relevance: Does the response address the user's question directly?
• Context Recall: Did the retriever retrieve all the relevant information needed to answer the query?

---

Conclusion

In this tutorial, you learned how to build a RAG pipeline using LangChain, OpenAI, and ChromaDB. By systematically loading documentation, creating semantic embeddings, storing them in a vector database, and injecting relevant context directly into the model's prompts, you have built an AI application grounded in truth.

As your enterprise data scales, mastering these retrieval architectures will distinguish your AI implementations from generic wrapper applications.

Ready to take your AI engineering and cloud architecture skills to the next level? Explore comprehensive hands-on labs and capstone projects at Deepskilling to master production-grade AI deployment, vector database administration, and LLMOps.