RAG over a codebase fails in retrieval, not generation: structural code chunking, hybrid BM25 search, recall@k eval, cross encoder re ranking, and citations.
My weekend RAG assistant demoed like magic and lied to my teammate by Monday. I asked it "where do we validate coupon codes?" and it handed me the exact function; I was already imagining deleting half my grep habits. Then a teammate asked a real question about our payment retry logic, and the thing pointed him, with total confidence, at a Firestore security rule that had nothing to do with payments. Not wrong in a funny way. Wrong in a way that would have cost him an afternoon if he'd trusted it.
That gap — between the cherry-picked demo and the first honest question — is the whole story of building retrieval-augmented generation over source code. Every RAG tutorial spends 90% of its air on the generation step, the part a large language model does well almost by accident. The parts that actually decide whether the system is useful — chunking and retrieval — get one paragraph and a RecursiveCharacterTextSplitter with chunk_size=1000. That default is where most home-grown code assistants quietly die. Here is what it took to drag mine from "demoed great" to "I'd actually trust it" over a living Flutter and Firebase repo, and the specific mistakes I made on the way.
Before the pipeline, it's worth naming why code is a hostile corpus for naive retrieval. Prose has redundancy: the same idea gets restated three ways in a chapter, so even a sloppy chunk usually lands somewhere near the meaning. Code doesn't. A single function is the only place a behavior lives, and its meaning is scattered across an identifier here, a call there, and an import at the top of the file. Miss the exact chunk and there is no paraphrase nearby to bail you out.
Codebases have three more properties that break the document-RAG playbook. They change every single day, so a corpus you indexed "once" is stale by the next merge. They're full of near-duplicate boilerplate — a hundred build() methods, a dozen serializers — that all embed to roughly the same vague point in vector space, so semantic similarity can't tell them apart. And they lean hard on exact literal tokens like error codes, environment variables, and feature flags, which is precisely the case dense embeddings are structurally bad at. If you've only ever done RAG over PDFs or a docs site, none of that intuition transfers cleanly. Treat a code retrieval system as its own beast.
The starter pipeline is four boxes: split files into chunks, embed each chunk, store the vectors, and at query time embed the question, pull the top-k nearest chunks, stuff them into a prompt. It's maybe 60 lines. It runs. On a small, tidy question it feels like the future.
The reason it demos well and answers wrong is that retrieval is being graded by nobody. The generator is fluent, so whatever garbage you hand it comes back as a confident, well-structured paragraph. Fluency masks retrieval failure completely. My teammate's payment question retrieved a security rule because both mentioned write and request.auth, and a 1000-character blind split had shredded the actual retry function across two chunks so neither one looked like a strong match on its own.
The mental shift that fixed everything: stop thinking of RAG as a generation problem and start treating it as a search problem with a language model bolted on the end. If the right chunk isn't in your top results, no model on earth saves you. It will just lie more eloquently. Everything below is about getting the right chunk into the top results — better chunking, hybrid retrieval, honest evaluation, and re-ranking, in that order of leverage.
The default splitters were built for prose. They cut every N characters, maybe trying to respect paragraph breaks. Code has no paragraphs. A function is a semantic unit whether it's 8 lines or 80, and a fixed window will happily slice through the middle of one, leaving you with a chunk that starts mid-if and a signature stranded three chunks away from its body.
I ran a quick audit: of my first 4,000 chunks, roughly a third started or ended mid-statement. Those chunks are close to useless. They embed to a vague average of whatever tokens they happen to contain and rarely win a retrieval, even when they hold the exact answer.
The fix is to chunk along the structure of the code, not the character count. Parse the file into a syntax tree and split on real boundaries: functions, methods, classes, top-level declarations. Tree-sitter has grammars for basically every language and is fast enough to run over a whole repo in seconds, which makes it the natural backbone for a code-aware chunker.
from tree_sitter_languages import get_parserdef chunk_code(source: str, language: str) -> list[dict]: parser = get_parser(language) tree = parser.parse(source.encode()) chunks = [] # Grab top-level functions, classes, and methods as whole units. node_types = {"function_declaration", "method_declaration", "class_declaration"} for node in tree.root_node.children: if node.type in node_types: text = source[node.start_byte:node.end_byte] chunks.append({ "text": text, "start_line": node.start_point[0] + 1, "end_line": node.end_point[0] + 1, "type": node.type, }) return chunksThree rules earned their keep once I switched to structural chunking:
retry() means nothing alone; payments/retry_service.dart :: RetryService.retry() means a lot, both to the embedding model and to the human reading the citation later.##/### boundaries so each chunk is one coherent idea. This one change did more for documentation retrieval than any embedding upgrade I tried.Structural chunking alone took my retrieval hit rate from embarrassing to respectable. It's the single highest-leverage change in the whole pipeline and it's the part the tutorials skip.
Dense embeddings are great at semantics and bad at specifics. Ask "how do we handle auth" and they shine. Search for FirebaseAuthException or the exact env var STRIPE_WEBHOOK_SECRET and they get fuzzy, because the embedding blurs a rare literal token into the general neighborhood of "auth stuff" instead of matching it exactly.
Code is full of exact tokens that must match exactly: identifiers, error codes, config keys, flag names, function signatures. Those are precisely where a decades-old keyword index beats a shiny vector model. BM25 — classic lexical, term-frequency search — nails the rare-literal case that embeddings fumble, because it rewards documents that contain the exact term rather than something semantically adjacent.
So run both and fuse the results. This is what people mean by hybrid search: a dense vector retriever for meaning, a sparse BM25 retriever for exact tokens, and a merge step that combines their rankings. The cheapest robust fusion is Reciprocal Rank Fusion (RRF): score each document by summing 1 / (k + rank) across every ranked list it appears in. No score normalization headaches, no tuning of a dense-vs-sparse weight, and it's a few lines.
def reciprocal_rank_fusion(rankings: list[list[str]], k: int = 60) -> list[str]: scores: dict[str, float] = {} for ranking in rankings: # e.g. [dense_ids, bm25_ids] for rank, doc_id in enumerate(ranking): scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank + 1) return sorted(scores, key=scores.get, reverse=True)On my repo, hybrid search fixed a whole category of failures at once: the "I searched the literal name of the thing and it wasn't in the results" bug. If you build one thing beyond naive vector search, build this. It's more reliable than swapping in a fancier embedding model, and cheaper — you're adding an index, not re-embedding the corpus.
Here's the discipline nobody wants to do because it feels like homework: measure retrieval on its own, before the generator gets involved. The generator makes everything look plausible, so if you only ever judge end-to-end answers, you're grading the wrong stage and flying blind on the stage that actually decides correctness.
I built a tiny eval set by hand. Thirty real questions I'd actually asked or expected a teammate to ask, each tagged with the file and function that genuinely answers it. It took about an hour. That hour paid for itself a dozen times over, because it turned every later decision from a vibe into a number.
Then I tracked two boring metrics as I changed the pipeline:
def recall_at_k(results: list[str], gold: set[str], k: int) -> float: return 1.0 if set(results[:k]) & gold else 0.0def mrr(results: list[str], gold: set[str]) -> float: for i, doc_id in enumerate(results): if doc_id in gold: return 1.0 / (i + 1) return 0.0
Every change — new chunker, hybrid search, a re-ranker — got scored against those 30 questions before I committed to it. Structural chunking took recall@10 from 0.61 to 0.82. Adding BM25 took it to 0.91. Neither of those wins would have been visible looking at generated answers, because the generator was busy papering over the gaps with confident prose. You cannot improve what you refuse to measure, and in RAG the thing worth measuring is upstream of the model everyone stares at.
A note on scale: thirty questions is not a benchmark, and it will miss whole categories of failure. That's fine. The point isn't statistical rigor, it's a fast, honest signal that catches regressions and makes changes comparable. Grow the set as real failures surface; don't wait for a "proper" eval before you start measuring anything.
A codebase is not a PDF. It changes every day. The tutorials treat the corpus as a fixed thing you index once, which is exactly backwards for a repo where the answer to "how does checkout work" was true last Tuesday and got refactored on Wednesday.
The failure mode is nasty because it's invisible: the system returns a real, well-formed, confidently-cited chunk that describes code you deleted a week ago. It's not hallucinating — the chunk was accurate when it was indexed. Your index is just lying about the present, which is arguably worse because the citation makes it look trustworthy.
Two things kept this under control.
Attach real metadata to every chunk. File path, language, last-commit timestamp, git blob hash, and whether it's source, test, or docs. That metadata is not decoration — it's what lets you filter ("only source files"), boost fresh code over stale, scope a query to one directory, and re-index surgically instead of rebuilding the world.
chunk_meta = { "path": "lib/payments/retry_service.dart", "commit_ts": 1719792000, "blob_sha": "9f3ac1b", # git hash-object of the file "kind": "source", # source | test | docs "symbol": "RetryService.retry",}Re-index on the diff, not the whole repo. I wired a lightweight step into CI: on merge to main, git diff the changed files, re-chunk only those, and upsert by blob_sha. Unchanged files keep their existing vectors. A full re-index of the repo took minutes and cost real embedding-API money; the diff-based one runs in seconds on every merge and is effectively free. Freshness stopped being a background anxiety and became a solved, automatic part of the deploy.
If you skip this, your assistant degrades silently. It gets a little more wrong every week and nobody notices until someone ships a bug based on a function that no longer exists.
Retrieval gives you maybe 20 candidates fast but bluntly — a single dot-product between the query vector and each chunk vector. A cross-encoder re-ranker reads the query and each candidate together in one pass and scores true relevance, catching subtleties that an independent embedding comparison misses (negation, which of two similar functions actually matches, whether a doc chunk answers the question or just mentions its words). It's noticeably better at ordering. It's also slower and adds a dependency.
The honest rule I settled on: re-rank only when your recall is already high but your MRR is mediocre. That's the exact situation where re-ranking helps — the right chunk is in your candidate set (good recall) but buried at position 8 (bad MRR). Re-ranking floats it to the top. If your recall is bad, re-ranking is just polishing the order of chunks that don't contain the answer; fix chunking and hybrid search first, then reach for a re-ranker.
from sentence_transformers import CrossEncoderreranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")def rerank(query: str, candidates: list[dict], top_n: int = 5) -> list[dict]: pairs = [(query, c["text"]) for c in candidates] scores = reranker.predict(pairs) ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True) return [c for c, _ in ranked[:top_n]]On my eval set, re-ranking bumped MRR from 0.58 to 0.74 and added roughly a tenth of a second per query. For an internal tool where I'm reading the answer anyway, that latency is invisible and the ordering win is real. For a latency-critical, user-facing path I'd think harder and maybe cache aggressively. Measure it against your own MRR before you pay for it; re-ranking is the one component people bolt on reflexively because it sounds sophisticated, when half the time their real problem is upstream in chunking or retrieval.
The single feature that changed how much I trust the thing: every claim in the answer must cite the chunk it came from, with a file path and line range I can click.
This is not cosmetic. It changes the failure mode from "silent confident wrong answer" to "answer with a citation that doesn't hold up when you look." The first is dangerous. The second I can catch in five seconds by glancing at the source. Citations turn verification from an act of faith into a click.
I enforce it in the prompt and in the plumbing:
[path:lines] at the top, so the model has a concrete handle to cite.You answer questions about our codebase using ONLY the provided chunks.After every claim, cite the source as [path:start-end].If the chunks do not contain the answer, say "Not found in the indexed code."Never invent file paths, function names, or line numbers.
That last line matters more than it looks. Without it, a helpful model will happily fabricate a plausible lib/services/auth_service.dart that doesn't exist. The "say you don't know" instruction, backed by a retrieval step that can genuinely return nothing, is what turns the assistant from a confident bluffer into an honest one. An answer I can verify in five seconds is worth ten answers I have to trust blindly.
Nobody tells you that RAG over a repo is a system you own forever, not a project you finish. The corpus moves under you constantly. Things that will drift and rot if you ignore them:
The uncomfortable truth is that a RAG system is 20% the model everyone talks about and 80% the search-and-data plumbing nobody demos. The plumbing is where your questions get answered right or wrong, and it's exactly the part that needs ongoing care.