Run retrieval and generation fully on localhost with Llama-3, Ollama embeddings, and Qdrant in one private Python workflow. Full breakdown.
Why keep retrieval and generation on localhost
Private RAG means both the search step and the answer step stay on your machine. Documents never leave for a remote embedding API, and prompts never leave for a hosted model endpoint. That matters when you work with internal notes, customer drafts, or any text you would not paste into a third-party chat box. A local stack also gives you a fixed cost model and predictable latency once the models are loaded, without depending on external rate limits or outages.
The tradeoff is hardware and setup. You need enough RAM and disk for Llama-3, an embedding model via Ollama, and a vector store. In return you get a single Python workflow you can version, audit, and run offline after the initial downloads.
The three pieces and how they fit
Llama-3 handles generation: given a question and retrieved context, it produces the final answer. Ollama serves embeddings for your chunks and for the query so both live in the same vector space. Qdrant stores those vectors and runs similarity search when a question arrives. Your Python process is the glue: chunk documents, embed them, upsert into Qdrant, then at query time embed the question, pull the top matches, build a prompt, and call Llama-3.
- Ingest: load text, split into chunks with overlap so sentences are not cut mid-idea, embed each chunk, store vectors plus the original text as payload in Qdrant.
- Query: embed the user question, search Qdrant for nearest neighbors, assemble a prompt that includes only those passages, generate with Llama-3.
- Guardrails: cap how many chunks you inject, instruct the model to answer only from the provided context, and surface source snippets so answers stay checkable.
Practical workflow choices that affect quality
Chunk size is the main dial. Chunks that are too large dilute relevance and waste context window; chunks that are too small lose surrounding meaning. Prefer splitting on paragraphs or headings when the source allows it, then fall back to fixed-size windows with modest overlap. Store metadata such as filename and section title in Qdrant payloads so you can filter collections later and cite sources in the UI or logs.
At generation time, keep the system prompt short and strict: use only the retrieved passages, say when evidence is missing, and do not invent citations. Rank or re-score results if you have a second pass available, but even a simple top-k from Qdrant is enough for a solid private baseline. Log query, retrieved IDs, and answer so you can debug wrong retrieval versus weak generation.
Running it as one private Python path
Structure the project as a few clear modules: config for model names and Qdrant URL, ingest that walks a documents folder, query that takes a string and returns answer plus sources. Point Ollama and Qdrant at localhost. Use environment variables for collection name and top-k so you can tune without editing code. Start Qdrant, confirm Ollama can serve both the embedding model and Llama-3, then run ingest once and query repeatedly.
When something fails, separate the failure modes. Empty or irrelevant retrieval usually means bad chunking, wrong collection, or a query that does not match how documents are written. Fluent but wrong answers usually mean the model is ignoring context or the prompt is too loose. Fix retrieval first, then tighten generation. That loop is the whole point of a local RAG stack you fully control.