Lesson 0004 · ~20 minutes
Two hard facts force the architecture. A firm's document corpus is far bigger than any context window — you cannot paste ten years of contracts into a prompt. And a bare LLM answers from training data, which is where hallucination lives — fatal in legal, where a made-up clause or invented precedent ends careers. RAG (Retrieval-Augmented Generation) is the fix: retrieve the few passages actually relevant to the question, then have the model answer only from those passages, citing them. The model becomes a reader of evidence, not an oracle.
Legal documents arrive as PDFs and scans. Amazon Textract is AWS's ML-OCR service: text, handwriting, and crucially tables and key-value pairs with structure preserved (docs) — fee schedules in contracts survive as tables, not word soup. Note how this snaps onto your Lesson 0001 whiteboard: upload lands in S3 → S3 event → SQS → worker calls Textract. Same skeleton, new payload.
You can't embed a 300-page document as one unit — retrieval needs small, focused passages (a clause, a section). So documents are split into chunks. Naive fixed-size splitting works but cuts blindly; structure-aware splitting (by section/clause boundaries — which Textract's structure gives you) is better for contracts.
The classic failure: a chunk says "The Company shall indemnify…" — which company, which agreement? Isolated chunks lose their document context and retrieval misses them. Contextual retrieval (Anthropic's technique) fixes this by having an LLM prepend a short situating sentence to every chunk before embedding — "This clause is from the 2024 Acme–Beta services agreement, section 12, indemnification…". Anthropic's benchmarks: failed retrievals drop ~49%, and ~67% when combined with reranking. Cheap at scale thanks to prompt caching. This is the single most name-droppable technique in the interview.
An embedding turns a text into a vector of its meaning; similar meanings land near each other, so "duty to cover losses" can find an indemnification clause that shares no words with the query. But legal search also needs exact matches — party names, "Section 12.3", statute numbers — where keyword search (BM25) beats semantics. Production answer: hybrid search — run both, fuse the rankings. Say "hybrid" in the interview; pure-vector search is a junior tell.
pgvector adds a vector column type + similarity search to Postgres (README). Two index modes: exact search (perfect recall, slower) or approximate (HNSW — much faster, slight recall trade-off). The Case AI argument for pgvector over a dedicated vector DB (Pinecone/Weaviate) is architectural, not performance: your chunks live in the same database as the case data — under the same row-level security. "Find similar chunks WHERE tenant_id = this firm" is one SQL query, one security model, one backup story (good comparison). Dedicated vector DBs earn their place at hundreds of millions of vectors — say that too, it shows you know the boundary.
Retrieved chunks go into the prompt with IDs: [chunk-847: Acme MSA §12.1] "…". The model is instructed to answer only from the provided chunks and to cite IDs for every claim. The app then maps IDs back to document + page for clickable citations — and can verify them programmatically (does the cited chunk actually contain the quoted text?). That check is your first taste of Lesson 0006: citation accuracy is measurable, which means it's testable, which means it belongs in CI.
Read Anthropic's Contextual Retrieval post end-to-end (~15 min). It covers chunking failure modes, embeddings + BM25 hybrid, reranking, and benchmark numbers — one post that arms you for the entire RAG conversation.