Grok 4.5 RAG Pipeline: Chunk, Rerank, Cite
By Dillip Chowdary • July 21, 2026 • Grok 4.5 engineering series
Chunk for questions, not for storage
Prefer 300–800 token chunks with 10–15% overlap for technical docs. Tables and code need special splitters so blocks stay intact.
Store source URL, heading path, and commit hash with every chunk. Citations without provenance are theater.
Refresh embeddings on doc change; stale vectors are a silent accuracy regression.
Hybrid retrieval beats pure vectors
Combine BM25/keyword with dense vectors. Product names, error codes, and CVE IDs often fail pure embedding search.
Rerank the top 50–100 candidates to a final 4–8 chunks before calling Grok 4.5. Quality jumps more than raw recall.
Apply tenant and ACL filters before scoring. Never retrieve then hope the model ignores other customers’ data.
Advertisement
Tech Pulse Daily
Get tomorrow's pulse first
Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.
Prompt pattern for grounded answers
Instruct the model to answer only from provided sources and to say “not in sources” when missing. Then enforce with evals.
Require inline citation markers that map to chunk IDs your UI can expand. Unlinked footnotes are weak SEO and weak trust.
Strip or paraphrase less; engineers want exact config keys and error strings when present in sources.
Eval suite that actually catches hallucinations
Include “unsupported claim” tests: answers that invent APIs should fail automatically via claim-vs-source checks.
Track citation coverage and answer faithfulness separately from stylistic preference.
Run nightly on a fixed query set; block deploys when faithfulness drops beyond a threshold.
Advertisement