Building a Privacy-First RAG Pipeline with LangChain and Local LLMs

A standard Retrieval-Augmented Generation pipeline typically depends on external APIs at multiple stages, creating serious data exposure risks. This tutorial walks through building a fully offline "Chat with your PDF" application that never sends data across the network, using LangChain, Ollama, ChromaDB, and a locally served Llama 3 model.

How to Build a Privacy-First RAG Pipeline with Local LLMs

1. Install Python 3.10+, Ollama, and pull the Llama 3 8B Instruct and nomic-embed-text models locally.
2. Ingest PDF documents using PyMuPDFLoader and split them into overlapping chunks with RecursiveCharacterTextSplitter.
3. Generate embeddings locally via Ollama's nomic-embed-text or HuggingFace Sentence Transformers.
4. Store the resulting vectors in ChromaDB (file-based) or pgvector (Postgres-backed) with local persistence.
5. Build a retrieval chain using LangChain LCEL that fetches the top-k relevant chunks per query.
6. Connect the retriever to the locally served Llama 3 model for context-grounded answer generation.
7. Optimize retrieval quality with contextual compression, chunk-size tuning, and metadata filtering.
8. Deploy a Streamlit chat UI for uploading PDFs and querying documents entirely on localhost.

Table of Contents

• Why Privacy-First RAG Matters
• Architecture Overview
• The Stack: Choosing Your Local Tools
• Code Implementation: Building the Pipeline
• Optimization: Improving Retrieval Quality and Performance
• Adding a UI Layer with Streamlit
• Implementation Checklist
• Where to Go from Here

Why Privacy-First RAG Matters

The Problem with Cloud-Based RAG

A standard Retrieval-Augmented Generation pipeline typically depends on external APIs at multiple stages. Document chunks get sent to OpenAI's embedding endpoint, vectors land in a hosted store like Pinecone, and every user query plus its retrieved context travels to GPT-4 for generation. Each of those network calls represents a point where sensitive data leaves the local environment. For organizations handling legal contracts, medical records, financial disclosures, or proprietary research, this architecture creates serious exposure. GDPR mandates data minimization and restricts cross-border transfers. HIPAA requires safeguards around protected health information. Even outside regulated industries, corporate IP leaking through third-party inference APIs is a risk most security teams will not accept. Building a local RAG pipeline with LangChain and a local LLM removes these exposure points: no API keys, no cloud dependencies, no data leaving the local machine (assuming Ollama is configured to bind only to localhost, which is its default behavior).

Building a local RAG pipeline with LangChain and a local LLM removes these exposure points: no API keys, no cloud dependencies, no data leaving the local machine.

What You'll Build

This tutorial produces a fully offline "Chat with your PDF" application running entirely on localhost. The architecture spans five stages: document ingestion, local embedding, vector storage, similarity-based retrieval, and answer generation through a locally served Llama 3 model. The final product includes a Streamlit-based UI for uploading PDFs and conducting private document chat sessions.

This tutorial uses llama3:8b-instruct (the Ollama tag for the Llama 3 8B Instruct model). If your machine has 8GB of VRAM or less, pull the quantized variant llama3:8b-instruct-q4_0 instead.

Prerequisites: Python 3.10 or later, approximately 16GB of RAM (8GB minimum when using smaller quantized models), and basic familiarity with LangChain concepts such as chains, retrievers, and document loaders.

For a foundational overview of RAG, see our complete guide to Retrieval-Augmented Generation.

Architecture Overview

The Five-Stage Pipeline

The pipeline follows a linear flow through five discrete stages:

1. Document Ingestion parses PDF files and splits them into overlapping text chunks suitable for embedding.
2. Embedding converts each chunk into a dense vector representation using a local model.
3. Vector Store indexes and persists these vectors on disk through ChromaDB (or pgvector).
4. A Retrieval step runs similarity search against the vector store, returning the most relevant chunks for a given query.
5. Generation feeds those chunks to a locally running Llama 3 model, which synthesizes an answer grounded in the retrieved context.

Why Each Component Stays Local

Every layer operates without network calls, assuming Ollama binds only to localhost (the default). PDF parsing happens through a local Python library. A model running inside Ollama or loaded directly via HuggingFace Transformers generates the embeddings on the local machine. ChromaDB writes to a local directory; pgvector runs inside a local or containerized Postgres instance. The LLM itself sits behind Ollama on localhost, and no query or document content ever crosses a network boundary. Contrast this with the typical cloud-dependent stack: OpenAI's text-embedding-ada-002 for embeddings, Pinecone for vector storage, and GPT-4 for generation. That stack sends data across the network at three separate stages. The local stack sends data across zero.

The Stack: Choosing Your Local Tools

LangChain as the Orchestration Layer

LangChain provides the abstractions that tie these components together: document loaders for ingestion, text splitters for chunking, vector store integrations for persistence, retrievers for similarity search, and chains (or the newer LangChain Expression Language) for wiring retrieval into generation. This tutorial targets langchain 0.3.x and langchain-community for integrations with Ollama and ChromaDB.

Ollama for Local LLM Inference (Llama 3)

Ollama enables one-command model serving on macOS, Linux, and Windows. It handles model downloading, quantization selection, and inference serving behind a local HTTP API. Llama 3 8B is the target model here. It fits within 8GB of VRAM when pulled as a 4-bit quantized variant (llama3:8b-instruct-q4_0); the default weights require approximately 16GB. Llama 3 ships under the Meta Llama 3 Community License, which permits commercial use subject to conditions including a 700M MAU threshold; review the full license at meta.com/llama/license before deployment. Installation is straightforward: run the install script, then pull the model.

If you're evaluating serving options, see our comparison of Ollama and vLLM.

Embeddings: Nomic Embed or Sentence Transformers

The simplest path is nomic-embed-text served through Ollama, keeping a single runtime for both embeddings and generation. The alternative is sentence-transformers/all-MiniLM-L6-v2 loaded via HuggingFace, which has a lighter memory footprint and works well when VRAM is constrained.

Vector Store: ChromaDB vs pgvector

ChromaDB is the default for this tutorial. pgvector setup is covered as a drop-in alternative.

pip install langchain langchain-community chromadb pymupdf ollama streamlit langchain-chroma langchain-ollama langchain-text-splitters

Code Implementation: Building the Pipeline

Step 1: Project Setup and Dependencies

Create a project directory and virtual environment:

mkdir local-rag && cd local-rag
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

Create a requirements.txt:

langchain>=0.3.0
langchain-community>=0.3.0
langchain-chroma>=0.2.0
langchain-ollama>=0.2.0
langchain-text-splitters>=0.3.0
chromadb>=0.5.0
pymupdf>=1.24.0
ollama>=0.3.0
streamlit>=1.38.0

Install everything:

pip install -r requirements.txt

Then install Ollama and pull the required models. To verify the installer before executing, download it first with curl -fsSL https://ollama.com/install.sh -o install.sh, inspect it, then run sh install.sh.

curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3:8b-instruct
ollama pull nomic-embed-text

All scripts in this tutorial must be run from the local-rag/ project directory with the virtual environment active, since modules import each other using bare module names.

Step 2: Document Ingestion and Chunking

PyMuPDFLoader handles complex PDF layouts more reliably than PyPDF in the author's experience, preserving text from multi-column pages and embedded tables. Chunk size directly controls retrieval precision: too large and the retrieved context contains noise, too small and the LLM lacks sufficient context. RecursiveCharacterTextSplitter with a chunk size of 1,000 characters and 200 characters of overlap provides a solid starting point.

# ingest.py
from langchain_community.document_loaders import PyMuPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter


def load_and_chunk(pdf_path: str):
    loader = PyMuPDFLoader(pdf_path)
    documents = loader.load()

    splitter = RecursiveCharacterTextSplitter(
        chunk_size=1000,
        chunk_overlap=200,
        length_function=len,
    )
    chunks = splitter.split_documents(documents)

    if not chunks:
        raise ValueError(f"No text extracted from {pdf_path}")

    print(f"Loaded {len(documents)} pages, split into {len(chunks)} chunks")
    print(f"Sample chunk:
{chunks[0].page_content[:200]}...")
    return chunks


if __name__ == "__main__":
    chunks = load_and_chunk("sample.pdf")

Each chunk carries metadata including the source filename and page number, which proves useful for filtering later.

Step 3: Generating Local Embeddings

Ensure Ollama is running (ollama serve in a separate terminal, or the system service started by the installer) before executing any Python code that calls the Ollama API.

Using Ollama's built-in embedding support keeps the runtime unified. The code instantiates the embedding model lazily to avoid import-time failures when Ollama is not running:

# embeddings.py
from langchain_ollama import OllamaEmbeddings

_embedding_model = None


def get_embedding_model() -> OllamaEmbeddings:
    global _embedding_model
    if _embedding_model is None:
        _embedding_model = OllamaEmbeddings(model="nomic-embed-text")
    return _embedding_model


# Module-level alias for backward compatibility, resolved lazily on access.
class _LazyEmbedding:
    def __getattr__(self, name):
        return getattr(get_embedding_model(), name)


embedding_model = _LazyEmbedding()


if __name__ == "__main__":
    # Verify with a test string
    model = get_embedding_model()
    test_vector = model.embed_query("This is a test sentence.")
    print(f"Embedding dimension: {len(test_vector)}")  # nomic-embed-text produces 768-dimensional vectors

For a lighter alternative using HuggingFace:

from langchain_community.embeddings import HuggingFaceEmbeddings

embedding_model = HuggingFaceEmbeddings(
    model_name="sentence-transformers/all-MiniLM-L6-v2",
    model_kwargs={"device": "cpu"},
    cache_folder="./models",
)

Step 4: Storing Vectors in ChromaDB

ChromaDB persists vectors to a local directory, so you do not need to re-index between application restarts.

# store.py
import os
from langchain_chroma import Chroma
from embeddings import embedding_model
from ingest import load_and_chunk

PERSIST_DIR = os.environ.get("CHROMA_PERSIST_DIR", "./chroma_db")


def create_vector_store(pdf_path: str):
    chunks = load_and_chunk(pdf_path)

    vector_store = Chroma.from_documents(
        documents=chunks,
        embedding=embedding_model,
        persist_directory=PERSIST_DIR,
    )

    print(f"Stored {len(chunks)} vectors in ChromaDB")
    return vector_store


def load_vector_store():
    return Chroma(
        persist_directory=PERSIST_DIR,
        embedding_function=embedding_model,
    )


if __name__ == "__main__":
    store = create_vector_store("sample.pdf")

    # Quick retrieval test
    results = store.similarity_search("What is the main topic?", k=2)
    for doc in results:
        print(doc.page_content[:150])

Step 5: Building the Retrieval Chain

This is where the components converge. The retriever pulls the top k=4 chunks from ChromaDB. A prompt template places those chunks as context alongside the user's question. The Ollama-served Llama 3 model generates the answer, and a StrOutputParser extracts the text.

# chain.py
import os
from langchain_ollama import ChatOllama
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
from store import load_vector_store

PERSIST_DIR = os.environ.get("CHROMA_PERSIST_DIR", "./chroma_db")


def build_chain():
    if not os.path.exists(PERSIST_DIR):
        raise FileNotFoundError(
            f"Vector store not found at '{PERSIST_DIR}'. "
            "Run store.py to ingest a document first."
        )

    vector_store = load_vector_store()
    retriever = vector_store.as_retriever(search_kwargs={"k": 4})

    llm = ChatOllama(model="llama3:8b-instruct", timeout=120)

    prompt = ChatPromptTemplate.from_template("""
Answer the question based only on the following context. If the context
does not contain enough information, say so clearly.

Context:
{context}

Question: {question}

Answer:
""")

    def format_docs(doc_list):
        return "

".join(doc.page_content for doc in doc_list)

    return (
        {"context": retriever | format_docs, "question": RunnablePassthrough()}
        | prompt
        | llm
        | StrOutputParser()
    )


if __name__ == "__main__":
    chain = build_chain()
    try:
        response = chain.invoke("What are the key findings in this document?")
        print(response)
    except Exception as e:
        print(f"Chain invocation failed: {e}")

This LCEL (LangChain Expression Language) chain is composable and supports streaming, which the Streamlit UI will use.

Step 6: Optional pgvector as an Alternative Store

For teams with existing Postgres infrastructure, pgvector provides a production-grade alternative. Start a Postgres container with the pgvector extension enabled:

docker run -d --name pgvector \
  -e POSTGRES_PASSWORD=postgres \
  -p 5432:5432 \
  pgvector/pgvector:0.7.0-pg16

WARNING: Replace postgres with a strong password; never use default credentials in production or shared environments.

The pgvector path requires psycopg2-binary, which is not included in the base requirements.txt. Install it separately:

pip install "psycopg2-binary>=2.9.0"

Then swap ChromaDB for PGVector in the store code:

import os

CONNECTION_STRING = (
    "postgresql+psycopg2://{user}:{password}@{host}:{port}/{db}".format(
        user=os.environ["PGVECTOR_USER"],
        password=os.environ["PGVECTOR_PASSWORD"],
        host=os.environ.get("PGVECTOR_HOST", "localhost"),
        port=os.environ.get("PGVECTOR_PORT", "5432"),
        db=os.environ.get("PGVECTOR_DB", "postgres"),
    )
)

vector_store = PGVector.from_documents(
    documents=chunks,
    embedding=embedding_model,
    connection_string=CONNECTION_STRING,
    collection_name="local_rag",
)

Set the required environment variables before running:

export PGVECTOR_USER="your_db_user"
export PGVECTOR_PASSWORD="your_db_password"

The rest of the pipeline remains identical. The retriever interface is the same regardless of which vector store backs it.

Optimization: Improving Retrieval Quality and Performance

Tuning Chunk Size and Overlap

Smaller chunks (around 500 characters) yield more precise retrieval hits but provide less surrounding context per chunk. Larger chunks (up to 1,500 characters) include more context but introduce noise that degrades answer accuracy. A 10-20% overlap ratio prevents information from being split across chunk boundaries. The optimal setting depends on the document type: dense legal text benefits from smaller chunks, while narrative reports work better with larger ones. Experimentation is unavoidable. A practical starting point: run 10 representative queries at each candidate chunk size and compare answer relevance across the results.

Maximizing the Context Window

Llama 3 8B supports an 8,192-token context window. With k=4 chunks at roughly 1,000 characters each (approximately 200-300 tokens per chunk for English prose; this ratio varies by content type and tokenizer), the retrieved context consumes around 800-1,200 tokens. The prompt template and generation headroom account for the rest. This leaves substantial room, but retrieving too many chunks degrades answer quality by diluting relevant content. Beyond roughly k=6-8 at 1,000-character chunks, dilution typically outweighs the benefit of additional coverage.

A ContextualCompressionRetriever can re-rank and trim irrelevant content before it reaches the LLM:

from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import EmbeddingsFilter
from store import load_vector_store
from embeddings import embedding_model

vector_store = load_vector_store()
base_retriever = vector_store.as_retriever(search_kwargs={"k": 8})

compressor = EmbeddingsFilter(
    embeddings=embedding_model,
    similarity_threshold=0.75,
)

compression_retriever = ContextualCompressionRetriever(
    base_compressor=compressor,
    base_retriever=base_retriever,
)

This retrieves a broader initial set (k=8), then filters down to only those chunks exceeding a 0.75 similarity threshold, producing a tighter and more relevant context window.

Metadata Filtering

During ingestion, PyMuPDFLoader automatically attaches page numbers and source filenames as metadata. You can use this metadata to filter retrieval to specific documents or sections, which is especially useful in multi-document scenarios.

Performance Tips for Consumer Hardware

For machines with only 8GB VRAM, use a quantized model variant: ollama pull llama3:8b-instruct-q4_0. 4-bit quantization typically cuts VRAM usage from ~16 GB to ~5 GB, with a small but measurable increase in perplexity. Batch embedding (passing multiple chunks in a single call) is faster than embedding one at a time. ChromaDB's persistence directory means the vector store survives restarts without re-indexing, which avoids multi-minute re-embedding runs during development iteration.

4-bit quantization typically cuts VRAM usage from ~16 GB to ~5 GB, with a small but measurable increase in perplexity.

Adding a UI Layer with Streamlit

Building a Minimal Chat Interface

Streamlit provides the fastest path to a working UI without requiring any frontend framework. The key components are st.file_uploader for PDF upload, st.chat_input for user queries, and st.chat_message for displaying the conversation.

# app.py
import streamlit as st
from langchain_ollama import OllamaEmbeddings, ChatOllama
from langchain_chroma import Chroma
from langchain_community.document_loaders import PyMuPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
import tempfile
import os
import shutil
import threading

st.set_page_config(page_title="Private Document Chat", layout="wide")
st.title("🔒 Chat with Your PDF — Fully Local")

MAX_UPLOAD_BYTES = 50 * 1024 * 1024  # 50 MB
MAX_QUERY_LEN = 2000

_chroma_lock = threading.Lock()

if "messages" not in st.session_state:
    st.session_state.messages = []
if "chain" not in st.session_state:
    st.session_state.chain = None
if "current_file" not in st.session_state:
    st.session_state.current_file = None


@st.cache_resource(show_spinner="Loading models…")
def load_models():
    embedding_model = OllamaEmbeddings(model="nomic-embed-text")
    llm = ChatOllama(model="llama3:8b-instruct", timeout=120)
    return embedding_model, llm


embedding_model, llm = load_models()

uploaded_file = st.sidebar.file_uploader("Upload a PDF", type="pdf")

if uploaded_file and (
    st.session_state.chain is None
    or st.session_state.current_file != uploaded_file.name
):
    file_bytes = uploaded_file.read()
    if len(file_bytes) > MAX_UPLOAD_BYTES:
        st.error(
            f"File exceeds {MAX_UPLOAD_BYTES // (1024 * 1024)} MB limit. "
            "Please upload a smaller document."
        )
        st.stop()

    with _chroma_lock:
        if os.path.exists("./chroma_ui_db"):
            shutil.rmtree("./chroma_ui_db")
    st.session_state.messages = []
    st.session_state.current_file = uploaded_file.name

    tmp_path = None
    try:
        with tempfile.NamedTemporaryFile(delete=False, suffix=".pdf") as tmp:
            tmp.write(file_bytes)
            tmp_path = tmp.name

        with st.spinner("Ingesting and embedding document..."):
            loader = PyMuPDFLoader(tmp_path)
            docs = loader.load()
            splitter = RecursiveCharacterTextSplitter(
                chunk_size=1000, chunk_overlap=200
            )
            chunks = splitter.split_documents(docs)

            vector_store = Chroma.from_documents(
                documents=chunks,
                embedding=embedding_model,
                persist_directory="./chroma_ui_db",
            )
            retriever = vector_store.as_retriever(search_kwargs={"k": 4})

            prompt = ChatPromptTemplate.from_template("""
Answer the question based only on the following context.
If the context does not contain enough information, say so.

Context: {context}
Question: {question}
Answer:
""")

            def format_docs(doc_list):
                return "

".join(d.page_content for d in doc_list)

            st.session_state.chain = (
                {
                    "context": retriever | format_docs,
                    "question": RunnablePassthrough(),
                }
                | prompt
                | llm
                | StrOutputParser()
            )
        st.sidebar.success(
            f"Ingested {len(chunks)} chunks from {len(docs)} pages."
        )
    except Exception as e:
        st.error(f"Failed to ingest document: {e}")
    finally:
        if tmp_path and os.path.exists(tmp_path):
            os.unlink(tmp_path)

for msg in st.session_state.messages:
    with st.chat_message(msg["role"]):
        st.markdown(msg["content"])

if query := st.chat_input("Ask a question about your document"):
    query = query[:MAX_QUERY_LEN]
    st.session_state.messages.append({"role": "user", "content": query})
    with st.chat_message("user"):
        st.markdown(query)

    if st.session_state.chain:
        with st.chat_message("assistant"):
            with st.spinner("Thinking..."):
                response = st.session_state.chain.invoke(query)
            st.markdown(response)
        st.session_state.messages.append(
            {"role": "assistant", "content": response}
        )
    else:
        st.warning("Please upload a PDF first.")

Running and Testing

Launch the application with:

streamlit run app.py

The UI presents a sidebar with a file upload widget and a main panel with a chat interface. After uploading a PDF, the ingestion spinner runs while chunks are embedded and stored. Once complete, queries entered in the chat input field return answers grounded entirely in the uploaded document's content. The conversation history persists in session state across interactions within the same browser session. Test with any multi-page PDF to verify end-to-end functionality.

Implementation Checklist

Use this as a quick-reference summary to track your progress through the build.

Where to Go from Here

Several directions open up from this foundation. Add multi-document support with metadata-based filtering. Wire in conversation memory using RunnableWithMessageHistory (the 0.3.x-compatible approach; ConversationBufferMemory is deprecated). Layer in authentication for multi-user deployments. For scaling beyond a single machine, swap ChromaDB for pgvector to use Postgres durability and concurrency (noting that pgvector write throughput is bounded by Postgres MVCC overhead; evaluate HNSW vs. IVFFlat index types for your workload), and consider replacing Ollama with vLLM when moving to GPU-server or team deployments where throughput matters.

For deeper context on RAG patterns and architecture decisions, revisit our complete guide to Retrieval-Augmented Generation.