Retrieval and RAG
The reranking and hybrid search posts share one setup: 1,000 SQuAD v1.1 questions retrieved against 2,067 paragraphs, so their numbers can be compared directly. On that set, embeddings alone put the right paragraph first 63.6% of the time, BM25 74.5%, and a cross-encoder reranker over the top 20 lifted the embedding result to 85.2%.
SQuAD questions reuse words from their paragraphs, which favors BM25. Each post states its own limits next to the result.
- Hybrid search for RAG: BM25 plus embeddings, measured on 1,000 questions
Keyword search (BM25) and embedding search miss different questions. Merging their result lists with reciprocal rank fusion cut the questions whose paragraph was missing from the top 20 from 45 to 16 out of 1,000, and raised top-1 accuracy after reranking from 85.2% to 87.3%. BM25 alone was better at rank 1 than embeddings on this test.
- Reranking for RAG: a cross-encoder lifted top-1 retrieval from 64% to 85%
Bi-encoder vs cross-encoder, measured on 1,000 SQuAD questions with two small local models. Re-scoring the top 20 search results moved the right paragraph to first place 85% of the time instead of 64%, for 67 ms per query on a laptop GPU. Past 20 candidates the gain flattened while the cost kept rising.
- Do you need a vector database? Exact search vs HNSW on 134,000 real embeddings
We embedded 134k docstring paragraphs with a local model and compared exact nearest-neighbour search with an HNSW index. Exact search grew from 0.07 to 0.8 ms per query; HNSW stayed near 0.17 ms at 97% recall. And with only one candidate, HNSW's greedy walk found the true nearest neighbour less than half the time.
- Your RAG documents are data, not instructions—and not automatically true
Two harmless prompt injections failed in a local RAG test, but a false policy document still changed the answer. Separate instruction authority, document provenance, and retrieval quality instead of calling one prompt secure.
- How do you evaluate LLM answers? Exact match vs key facts vs LLM-as-judge
We graded 12 answers from a small RAG system three ways and compared each grader with a human label. Exact match agreed on 5 of 12, a key-fact check on 11 of 12, and a small LLM judge on 8 of 12, even though its total score looked right.
- How should you chunk documents for RAG? A chunk-size experiment
We split a fictional handbook six ways and measured how often retrieval found the chunk with the answer. One-sentence chunks dropped from 96% to 33% once the text used pronouns; paragraph chunks stayed at 100% in both cases.
- Prompting vs RAG vs fine-tuning: which one do you need?
We fine-tuned a small model on a fictional company handbook with LoRA and compared it with RAG. Fine-tuning learned the facts, even for reworded questions, but kept the old policy after an update, invented a CEO, and changed how the model answers.
- RAG explained in 50 lines of Python (with a local model)
Retrieval-augmented generation in 49 lines: embed documents, retrieve the closest ones, and put them in the prompt. Without RAG, a local model invented a vacation policy; with RAG it answered correctly, and one rule stopped it guessing.
- What are embeddings? Semantic search with cosine similarity in Python
An embedding turns text into a vector so that similar meanings land close together. A 30-line example with a free local model finds "My laptop gets very hot" for "Why does my notebook overheat?", with no shared words.
