RAG Retrieval Evaluation: Why Source Routing Beats More Search

RAG retrieval evaluation is where ambitious AI knowledge-base projects meet an uncomfortable truth: the model cannot answer from evidence it never receives. A recent local FinanceBench experiment shared on Reddit makes that point unusually clearly—simple source routing plus reranking improved exact evidence-page retrieval far more than hybrid search, graph features, or vector search alone.

The result matters beyond financial filings. Any founder building an AI assistant over customer documentation, policies, contracts, support tickets, product catalogs, or internal wikis has likely seen the same failure mode: the retrieval layer returns something plausible from the wrong document, and the language model turns that near miss into a confident answer.

The lesson is not that embeddings, BM25, knowledge graphs, HyDE, or RAG-Fusion are useless. It is that retrieval architecture should start with the question, “What documents are eligible to answer this?” before asking, “Which chunks are semantically similar?” In document collections with repeated jargon and closely related entities, eligibility can be more valuable than another retrieval trick.

The experiment: a practical RAG retrieval evaluation on a laptop

The original post came from a builder testing a local RAG stack on FinanceBench, an open-book financial question-answering benchmark. FinanceBench contains 10,231 questions, answers, and evidence strings across public-company filings; its open-source sample includes 150 annotated examples. That makes it more useful than a generic chatbot demo because it provides human-grounded evidence against which a retrieval system can be checked. (github.com)

According to the post, the author indexed roughly 55,000 chunks from 84 real 10-K and 10-Q filings, then ran everything locally with nomic-embed-text for embeddings and Llama 3.1 8B through Ollama. Ollama describes embeddings as numerical vectors that can be stored and searched for semantic retrieval, while its nomic-embed-text library entry positions the model as a dedicated embedding model with a large context window. (ollama.com)

That setup is notable for two reasons. First, it is within reach for a technical solo founder or small product team; it does not require a proprietary retrieval API or a large inference budget. Second, a collection of SEC filings is a hostile environment for naive retrieval. Companies use overlapping vocabulary—revenue, deferred revenue, impairment, debt, operating income, material weakness, fair value, and dozens of other recurring terms—while the exact number, time period, entity, and table matter enormously.

The evaluation metric was deliberately retrieval-first: did the exact evidence page appear in the top five chunks sent to the LLM? The author also tracked whether a passage from the correct filing appeared in the top five. This distinction is foundational. A RAG system can retrieve the right company filing but still miss the page that contains the evidence required for a defensible answer.

The reported results

Here is the comparison reported in the Reddit post:

The strongest configuration retrieved the exact evidence page 39% of the time, compared with 25% for vector search—a relative improvement of about 56%, or roughly 1.6 times the baseline hit rate. More importantly, it put content from the right filing into the top five 93% of the time. Those are retrieval metrics, not final answer-accuracy scores, so they should not be read as proof that the system answers 39% of questions correctly. But they are highly diagnostic: a generator has little chance of producing a cited, auditable response when the source page is absent.

Why the correct filing is not the same as the correct evidence

A common RAG dashboard reports one retrieval score, often recall at K, without separating document selection from passage ranking. That can hide two very different problems.

Document selection asks whether the system has chosen the right source file, account, project, customer, policy version, or product manual. Passage ranking asks whether it found the paragraph, table, or page inside that selected source that actually answers the question.

In the reported baseline, vector search reached the correct filing 78% of the time but recovered the exact evidence page only 25% of the time. Put differently, many queries got close enough to look reasonable in a product demo yet not close enough to support the right answer. For finance, legal, healthcare, security, or enterprise support use cases, that gap is where hallucinations and misleading citations emerge.

Consider a query such as: “What was Company X’s restructuring charge in fiscal year 2022, and how did it change year over year?” An all-corpus semantic search may retrieve chunks about restructuring charges from several businesses because the language is similar. A chunk can have strong lexical and semantic overlap while being unusable because it belongs to Company Y, refers to a different year, or discusses a different category of charge.

The important product implication is that “retrieval works” is not a binary conclusion. Your evaluation should answer at least four questions:

1. Did the system identify the correct document or source family?
2. Did it retrieve the gold evidence passage or table?
3. Did it include all evidence needed for multi-part reasoning?
4. Did the final answer stay faithful to that evidence and cite it correctly?

A retrieval score that skips the first two questions can make a weak system appear better than it is.

Source routing is a form of search-space control

The post’s best-performing intervention was almost embarrassingly simple. If the question names a company that matches a filename, search only that file. Rather than asking a retriever to distinguish one company’s generic finance language from another’s across the whole corpus, the pipeline removes the irrelevant companies before ranking begins.

This is source routing: applying deterministic or probabilistic metadata constraints to narrow the candidate set before retrieval. In the experiment, company identity was encoded in filenames. In a production application, the routing key might be a tenant ID, a workspace, a product SKU, a locale, a customer account, a document type, an effective date, a compliance region, or a version number.

Why routing often beats “smarter” retrieval

Semantic retrieval is fundamentally comparative. A vector index finds the chunks whose embeddings are closest to the question embedding. But it does not inherently understand that a question mentioning “Acme Corp.” should never be answered by “Globex Corp.” unless that entity constraint is represented and enforced.

The same limitation affects keyword retrieval. BM25 is excellent at rewarding distinctive query terms, but terms that are common across a specialized corpus are not distinctive. In filings, the phrase “net cash provided by operating activities” may occur in nearly every company’s report. In support documentation, “API key,” “webhook,” “rate limit,” and “domain verification” can recur across dozens of versions and products.

Routing changes the math of the task. Instead of trying to rank 55,000 candidate chunks, a system may rank only the chunks from the one eligible filing. The embedding model has less opportunity to confuse valid-looking alternatives, and the reranker has a smaller, cleaner pool to inspect.

This is also why routing can improve latency and cost. It generally does not require an additional generative LLM call. A rule-based filter, structured query parser, named-entity recognizer, metadata lookup, or lightweight classifier can do the work. Fewer candidates can also reduce vector database scans and reranker invocations.

The hidden connection to multi-tenant RAG

For SaaS builders, source routing is more than a quality enhancement. It is closely related to access control. A customer support copilot should not merely prefer documents from the current workspace; it should be technically unable to retrieve content from another workspace. The same metadata boundary that improves relevance can help enforce tenant isolation.

That does not mean filename matching is enough for every application. Filenames can be inconsistent, aliases can collide, and questions can mention multiple entities. But it does mean that metadata should be designed as part of retrieval, not added after the vector index ships.

Why hybrid search did not improve the baseline

Hybrid retrieval is frequently treated as the default answer to weak vector recall: combine dense semantic similarity with BM25 or another sparse keyword method, then fuse the result lists. In many collections, that is sensible. Yet the reported hybrid configuration stayed at 25% exact-page recall and fell from 78% to 71% on correct-filing recall.

The likely reason is corpus ambiguity, not a universal indictment of hybrid search. Long analyst-written finance questions contain many broad terms that match look-alike passages in every filing. Adding a keyword retriever can introduce more of those plausible but wrong-company chunks into the candidate pool.

This is a useful reminder that retrieval components are not independent points on a feature checklist. A second retriever helps only if it contributes different, discriminative signals. If both dense and sparse retrieval are confused by the same repeated language, fusion can preserve or amplify the confusion.

When hybrid search is still worth testing

Hybrid search can be valuable when queries include identifiers or vocabulary that embeddings may blur, including:

• Error codes, API parameter names, invoice numbers, model numbers, SKUs, and product IDs.
• Exact policy phrases, legal clauses, acronyms, or feature names.
• New terminology that may not be well represented in an embedding model’s training data.
• Short, keyword-heavy searches where semantic context is thin.
• Collections where users use the document’s own wording rather than natural-language questions.

The key is to evaluate it against your data rather than assuming a generic best practice will transfer. FinanceBench is especially useful as a warning: an apparently sophisticated configuration can score no better than the baseline when the real bottleneck is source disambiguation.

What the reranker contributed—and why it needs good candidates

The reported cross-encoder reranker improved exact evidence-page retrieval from 25% to 29% without routing, and routing plus reranking reached 39%. That progression matches the usual two-stage retrieval architecture: use a fast bi-encoder or vector search to retrieve a candidate pool, then use a more expensive cross-encoder to score each query-passage pair more precisely.

Sentence Transformers describes this trade-off directly. Bi-encoders scale efficiently for retrieval, while cross-encoders can produce higher-quality relevance judgments but are too costly to apply across a massive corpus one query-passage pair at a time. The typical pattern is to retrieve a limited top-N candidate set and rerank it. (sbert.net)

The FinanceBench result adds an operational nuance: rerankers cannot rescue evidence that never enters their candidate pool. If a correct page is excluded because the first-stage retriever looked across the wrong documents, a perfect reranker never sees it. Routing improves candidate recall at the document level; reranking improves precision among those candidates. Together, they address separate stages of the pipeline.

A practical retrieval sequence

For many bounded-document RAG products, a robust sequence looks like this:

1. Parse constraints from the query. Extract customer, company, product, jurisdiction, date range, document type, and version when available.
2. Apply hard access and metadata filters. Enforce tenant boundaries first, then restrict to eligible sources.
3. Retrieve broadly within the eligible set. Run dense search, sparse search, or both based on the corpus and query type.
4. Rerank the top candidates. Score query-passage pairs with a cross-encoder or domain-tuned reranker.
5. Diversify the final context. Avoid sending five near-duplicate chunks; include adjacent pages or distinct evidence for multi-hop questions.
6. Generate with citations and abstention rules. Ask the model to quote or cite retrieved evidence and decline unsupported claims.

If your product helps developers answer implementation questions, source filters should include product version, language, framework, and endpoint. For example, a retrieval assistant over an email API reference and setup guides should not answer a question about a current SDK using snippets from a deprecated endpoint or a different language package.

The entity graph lesson: complexity is not evidence

The author also reported no gain from an entity graph, despite being particularly enthusiastic about that component. This is one of the more valuable parts of the experiment because it demonstrates a discipline that RAG teams often lack: publish what failed, not just what worked.

Knowledge graphs can be powerful when questions require relationships that are not explicit in a single chunk—for example, tracing a subsidiary to its parent, connecting a person to a role across time, or joining entities across documents. But a graph only helps if entity extraction is accurate, graph edges match the question distribution, and graph traversal produces candidates that improve on baseline retrieval.

In a corpus where the primary challenge is “find the right filing for the company already named in the query,” an entity graph may be solving a more elaborate problem than necessary. Direct metadata filtering is easier to test, simpler to maintain, and less likely to introduce extraction errors.

This suggests a useful product rule: add a graph only after your evaluation identifies graph-shaped failures. Do not build one because diagrams of graph-enhanced RAG look compelling.

The benchmark exposed pipeline bugs, not just model limits

Perhaps the most actionable part of the Reddit post was not the ranking table. The evaluation harness uncovered two implementation errors that could easily survive manual spot checks:

• Thirty-nine of the 84 PDFs were encrypted and were being silently skipped during ingestion.
• A RAG-Fusion query-generation stage was accidentally sending the model’s formatting preamble—rather than only the generated alternative queries—into search.

These are not edge cases. They represent a broad category of RAG failures where teams spend weeks comparing embedding models while their ingestion, parsing, prompt extraction, or index versioning is broken.

The SEC’s EDGAR search tools offer full-text access to years of electronic filings, but source material is still heterogeneous: filings can include tables, exhibits, different formats, scanned pages, and version-specific quirks. Any document pipeline needs explicit checks that each expected source was fetched, parsed, chunked, embedded, indexed, and retrievable. (sec.gov)

Build an ingestion audit, not just a retrieval benchmark

A dependable RAG evaluation suite should generate a data-quality report before it calculates recall. At minimum, log:

• Expected documents versus successfully downloaded documents.
• Parse success, page counts, extracted character counts, and table extraction results.
• Chunk counts per document, including suspiciously low or high outliers.
• Embedding model name, digest or version, dimensions, and index build timestamp.
• Metadata completeness for source, page, tenant, document type, version, and permissions.
• Query templates and the exact normalized queries sent to each retriever.

The author also flagged possible embedding differences across two Ollama versions on the same machine. That observation is not yet a confirmed conclusion, but it points to a real reproducibility issue: if you rebuild an index with a changed model artifact, runtime, quantization, normalization method, or chunking code, you may be comparing two systems at once without realizing it.

Treat the index as a versioned build artifact. Record the dataset hash, parser version, chunking parameters, embedding model identifier, embedding dimension, vector-store configuration, reranker, and code commit. An experiment that cannot be reproduced is hard to trust and harder to improve.

How to design a useful RAG retrieval evaluation

A good benchmark does not need to start at FinanceBench scale. It needs a representative, labeled set of real questions and evidence. FinanceBench’s value comes from grounding questions in evidence strings rather than judging polished answers by vibes. Its full benchmark was designed around real financial-analysis scenarios, and the public sample makes the methodology approachable for practitioners. (github.com)

Start with 50 to 100 questions drawn from the work your users actually do. Include easy lookups, entity disambiguation, date-sensitive questions, exact terminology, numerical calculations, cross-document synthesis, and questions that should be refused because the evidence is unavailable.

Metrics that reveal where the pipeline fails

Use a layered scorecard rather than one aggregate accuracy number:

For a question with a single gold page, evidence recall@5 is simple: did that page appear among the five passages? For questions requiring multiple pieces of evidence, measure set coverage: what portion of the necessary evidence pages appeared? That prevents a system from scoring well by retrieving just one convenient fragment.

Do not overlook negative tests. Ask for a policy that is not in the indexed corpus, a product feature unavailable on a given plan, or a number for a nonexistent period. The right outcome may be an explicit “I can’t verify that from the available sources,” not a best-effort answer.

A testing roadmap for teams building document AI

The original author said they were continuing tests of HyDE, RAG-Fusion, CRAG, an “everything on” configuration, alternative embeddings, chunk sizes, and end-to-end answer accuracy. That is a sensible backlog—but the order matters.

Before experimenting with query expansion or corrective retrieval, establish a clean baseline with verified ingestion and stable metadata. Otherwise each new method adds moving parts and makes it harder to identify the real source of an improvement or regression.

A disciplined testing plan could look like this:

1. Freeze a baseline. One parser, one chunker, one embedding model, one index, one prompt, and one set of labeled questions.
2. Fix data completeness. Fail the build if a source cannot be parsed or indexed; never silently omit documents.
3. Test routing separately. Compare no filter, deterministic metadata filter, entity-based routing, and a fallback path for ambiguous queries.
4. Test retrieval separately. Evaluate dense, sparse, and hybrid retrieval at multiple candidate depths.
5. Test reranking separately. Measure gains at top 5, top 10, and top 20, along with latency.
6. Vary chunking carefully. Test chunk size, overlap, section-aware chunks, table-aware chunks, and parent-child retrieval—but change one dimension at a time.
7. Evaluate answers last. Only after evidence availability is understood should you attribute answer changes to the LLM, prompt, or context assembly.

This sequence also protects teams from an expensive anti-pattern: adding LLM calls to compensate for a simple metadata failure. If the query clearly specifies an entity, file, project, or version, deterministic routing may be faster, cheaper, and easier to explain than an agentic search loop.

What creators and SaaS founders should take from this

For founders, the most important takeaway is strategic. The defensibility of a RAG product will rarely come from saying it uses “hybrid search,” “agents,” or “graphs.” Those labels are becoming table stakes. The stronger differentiator is a system that knows what content is authorized, applicable, current, and sufficient for the user’s question—and can show its work.

For marketing teams, that means avoiding vague claims such as “ask anything about your data.” A better promise is narrower and more trustworthy: “Get answers from the correct account, document version, or policy, with source citations.” Precision is a feature users can feel when the question is costly or consequential.

For developers, put metadata modeling near the beginning of the build. Add fields you will need later even if the first interface is a simple chat box: organization ID, document ID, source URL, page number, section heading, author, product, version, locale, effective date, access level, and ingestion timestamp. Retrofitting these fields after users demand reliable answers is painful.

For local-AI enthusiasts, the experiment is encouraging. A laptop-scale setup using Ollama and open models can surface meaningful retrieval insights. The limiting factor is often not access to the largest model; it is evaluation rigor, data hygiene, and system design.

The broader conclusion: constrain first, rank second

The FinanceBench experiment should not be treated as a final leaderboard for every RAG technique. It is one reported implementation on a 150-question public sample, with results that need independent reproduction and end-to-end answer evaluation. The post itself appropriately frames several follow-up experiments as still in progress.

But its central finding is durable: when documents share language but differ by entity, source routing can be a higher-leverage improvement than adding more retrieval machinery. Reranking compounds that benefit because it can focus on a candidate set where the right answer is actually present.

The practical formula is straightforward: verify ingestion, model the metadata, route to eligible sources, retrieve enough candidates, rerank them, measure evidence coverage, and only then optimize the answer model. That sequence is less glamorous than an “everything on” RAG stack, but it is more likely to produce answers users can trust.

FAQ

What is RAG retrieval evaluation?

RAG retrieval evaluation measures whether a retrieval-augmented generation system fetches the documents, pages, chunks, or tables needed to answer a question. Strong evaluations separate source selection, evidence retrieval, answer correctness, citation support, latency, and cost rather than relying on one chatbot-quality score.

Why did source routing improve RAG performance in this test?

The reported corpus contained many financial filings with overlapping terminology. Routing limited search to the filing associated with the company named in the query, removing many wrong-company passages before vector search and reranking could be distracted by them.

Does hybrid search not work for RAG?

Hybrid search can work well, especially for identifiers, exact terms, short keyword queries, and mixed terminology. This experiment shows that it is not automatically beneficial when both semantic and lexical signals are non-discriminative across a corpus. Test it against your own labeled queries.

What does a cross-encoder reranker do?

A cross-encoder reads the query and a candidate passage together and assigns a relevance score. It is generally more precise than embedding similarity for a small set of candidates, but too computationally expensive to score every chunk in a large index, which is why it is used after first-stage retrieval. (sbert.net)

What should I measure before improving my RAG prompt?

First check document coverage, parsing success, metadata completeness, correct-source recall, and evidence recall at the number of passages you send to the model. If the evidence is missing from context, prompt engineering cannot reliably fix the answer.