Generic RAG works fine on Wikipedia. It falls apart on regulatory filings, footnoted financial statements and tables that don't fit on one page. Here's the pipeline we built when the off-the-shelf approach hit the wall.
The client was a mid-sized asset manager. The ask was straightforward: an internal assistant that analysts could query against ten years of annual reports, regulatory filings and internal research notes.
The first prototype - off-the-shelf embeddings, naive chunking, vector search - looked promising in demos and failed in evaluation. Analysts asked it for a fund's expense ratio and got back a paragraph about expense management philosophy. They asked about exposure to a sector and got matched on the word "exposure" in a risk disclaimer.
Generic RAG works on documents written like Wikipedia articles. Financial documents are not written like Wikipedia articles.
The standard RAG recipe assumes that a relevant chunk of text, retrieved by semantic similarity, is what you need to answer the question. For finance, three assumptions in that recipe break.
Chunks aren't self-contained. A line in an income statement table is meaningless without the column header and the period label, which may be six pages away. A footnote reference like "(see Note 12)" is a hyperlink in disguise.
Semantic similarity is noisy. "Revenue grew 12%" and "Revenue grew 1.2%" embed to nearly identical vectors. The number is the point. The embedding doesn't care about the number.
The question vocabulary doesn't match the document vocabulary. An analyst asks "what's the AUM?" The document says "assets under management totalled $4.7B as at fiscal year end." Embeddings handle that synonymy decently, but they handle "what changed in Q3?" - which spans temporal reasoning - badly.
Five stages, each addressing one failure mode.
1. Structure-aware parsing, not flat text extraction. We replaced the PDF-to-text pipeline with a structured extractor that preserves tables as tables, footnotes as linked references and section hierarchies as metadata. Tables get serialised to a canonical form (Markdown with explicit column headers) so the embedding sees the structure.
2. Hybrid retrieval - BM25 plus embeddings. Pure vector search is bad at exact-match queries (ticker symbols, dollar amounts, named entities). We run BM25 and dense retrieval in parallel and rerank the union. This single change fixed about half the eval failures.
def retrieve(query, k=20):
sparse = bm25_index.search(query, k=k)
dense = vector_index.search(embed(query), k=k)
candidates = dedup(sparse + dense)
return reranker.rank(query, candidates, top_k=8)
3. A domain-tuned reranker. We fine-tuned a small cross-encoder reranker on roughly 4,000 analyst-labelled (query, passage, relevance) triples from the client's internal QA logs. This was where the biggest jump in NDCG came from - more than any change to the embedding model.
4. Table-aware context expansion. When a retrieved chunk contains a cell reference or footnote pointer, we expand the context to include the referenced row or footnote before sending to the model. The model sees the local structure, not just the matched fragment.
5. Period and entity normalisation at index time. Dates, currency and entity names get a normalised form attached as metadata. The query layer can then filter ("from Q3 2025 onwards") without relying on the model to parse temporal language correctly.
We would build the evaluation harness first, before any retrieval code. We had a working pipeline before we had a representative test set and we spent two weeks tuning chunk sizes against analysts' verbal feedback. Once we had 200 labelled (question, expected answer span) pairs, every change became measurable. Most of the early "improvements" had been noise.
We would also resist the urge to use the largest available embedding model. After the reranker landed, the embedding model contributed surprisingly little to final quality. A smaller, cheaper, faster embedding model with a good reranker beat a large embedding model with no reranker on every metric we cared about.
Generic RAG is a starting point, not a solution. For any domain with structured documents, specialised vocabulary, or numerical precision requirements, the wins come from understanding the documents - not from larger models. Build the eval harness first, layer hybrid retrieval and reranking and respect the structure your source documents are already trying to give you.