RAG and Knowledge Systems

Fixed-size and recursive splitting

The baseline is a window of chunk_size units with chunk_overlap units shared between neighbours so a fact straddling a boundary still lands whole in at least one chunk. In LangChain's RecursiveCharacterTextSplitter the size is measured by length_function, which defaults to Python len(), i.e. characters. The recursive variant tries separators in order ("\n\n", "\n", " ", "": paragraph, line, word, character) so coherent units stay together for as long as possible.[3]

Structural and sentence-aware splitting

Structure-aware splitters respect boundaries the author already drew. LlamaIndex's SentenceSplitter breaks at sentence boundaries (the docs example uses chunk_size=1024 plus a chunk_overlap), and MarkdownNodeParser parses raw markdown into nodes.[4] LangChain's MarkdownHeaderTextSplitter groups text by header level and attaches the headers as metadata (for example {'Header 1': ..., 'Header 2': ...}); headers are stripped from the text by default (strip_headers=False keeps them). Chain it with a recursive splitter so overlap applies inside a section and never crosses a header boundary.[5]

from langchain_text_splitters import (
    MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter)

md = MarkdownHeaderTextSplitter(
    headers_to_split_on=[("#", "Header 1"), ("##", "Header 2")],
    strip_headers=False)          # keep headings inside the chunk text
sections = md.split_text(markdown_text)  # each Document carries header metadata

rc = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100)  # illustrative values
chunks = rc.split_documents(sections)   # overlap stays inside each section

Semantic chunking

Instead of a fixed size, semantic chunking embeds sentences and cuts where adjacent-sentence similarity drops. LlamaIndex's SemanticSplitterNodeParser chooses breakpoints from embedding similarity; its docs example uses breakpoint_percentile_threshold=95 and buffer_size=1, and credits Greg Kamradt's "5 levels of embedding chunking".[4]

Late chunking

Conventional pipelines chunk first and embed each chunk alone, so a chunk saying "the company" cannot know which company. Late chunking (Günther et al., 2024) runs a long-context embedding model over the whole text and splits "after the transformer model and just before mean pooling", so every chunk vector is pooled from token embeddings that already attended to the surrounding document. The authors state it needs no extra training, and fine-tuning can improve it further.[1] The limit is practical: it requires an embedding model whose context window can hold the document.

Contextual retrieval (Anthropic)

A different repair: have an LLM write a short, chunk-specific explanation of where the chunk sits in its document, prepend it to the chunk, and use the result for both the embedding and the BM25 index. Anthropic reports the context is roughly 50 to 100 tokens per chunk and that one-time generation costs $1.02 per million document tokens. For knowledge bases under about 200,000 tokens (around 500 pages) the post says simpler approaches, such as putting everything in the prompt, are enough.[2]

Choosing a strategy

The comparison below is my own editorial synthesis from how each method works, not a benchmark result.

ANN basics: flat, IVF and HNSW

Flat search is a linear scan: exact, low memory, sensible for small collections (Weaviate suggests it for small per-tenant data).[7] IVF partitions vectors into lists and searches only some of them; in pgvector that is lists at build time and ivfflat.probes (default 1) at query time.[8] HNSW (Malkov and Yashunin) builds multi-layer proximity graphs, similar in spirit to a skip list, with search cost that scales logarithmically.[6]

Distance metrics

pgvector exposes <-> (L2), <#> (negative inner product), <=> (cosine), <+> (L1), <~> (Hamming) and <%> (Jaccard).[8] Milvus lists L2 and IP for float vectors, Jaccard and Hamming for binary vectors, and IP or BM25 for sparse vectors.[9] Weaviate's HFresh index supports only cosine and l2-squared.[7] Match the metric to the one your embedding model was trained for; for unit-length vectors cosine and dot product rank identically.

Feature comparison (vendor docs only)

Every cell comes from the documentation pages I fetched. "Not stated" means the page I read does not say, not that the feature is absent.

-- pgvector: HNSW index with the documented defaults, cosine distance
CREATE EXTENSION vector;
CREATE TABLE items (id bigserial PRIMARY KEY, body text, embedding vector(768));
CREATE INDEX ON items USING hnsw (embedding vector_cosine_ops)
  WITH (m = 16, ef_construction = 64);

SET hnsw.ef_search = 40;                 -- query-time recall/speed knob (default 40)
SELECT id, body FROM items
ORDER BY embedding <=> '[0.12, -0.03, ...]'  -- <=> is cosine distance
LIMIT 5;

When to choose what (my own synthesis)

The two retrievers and the parallel pipeline

BM25 comes from the Probabilistic Relevance Framework (grounded in 1970s-80s work); BM25F extends it to document fields such as metadata.[17] It scores exact term matches via an inverted index. Learned sparse models such as SPLADE keep that exact-match, inverted-index efficiency and interpretability but add learned term expansion to handle vocabulary mismatch.[18] A dense retriever embeds the query and finds nearest vectors with an ANN index. Weaviate states that its hybrid search runs the vector and BM25 searches in parallel and then fuses them.[13]

Reciprocal rank fusion, worked by hand

Cormack, Clarke and Buettcher define the score as the sum, over every ranking that contains the document, of 1 / (k + r(d)), where r(d) is the document's rank. The paper states that "k = 60 was fixed during a pilot investigation and not altered during subsequent validation."[16] Elasticsearch uses score += 1.0 / (k + rank) with rank starting at 1, rank_constant defaulting to 60, and rank_window_size defaulting to the size parameter.[19] OpenSearch added RRF in 2.19 (12 Feb 2025) with a default rank constant of 60.[20] The ranks in the diagram are invented for the arithmetic.

Rank-based versus score-based fusion

Weaviate's default hybrid method is relativeScoreFusion (default since v1.24), which normalises each search's scores to 0-1 (min-max style) and weights them with alpha (0 is keyword only, 1 is vector only, server default 0.75). It notes there is no max-distance equivalent for BM25 because BM25 scores are unbounded. rankedFusion, the default through v1.23, is the rank-based alternative.[13] Qdrant's Query API supports RRF and DBSF (distribution-based score fusion), added weighted RRF in v1.17.0, and advises RRF as the default when you have no evaluation set.[21]

In code

def rrf(rankings, k=60):
    """rankings: list of ranked doc-id lists, best first. Returns fused ids."""
    scores = {}
    for ranking in rankings:
        for rank, doc in enumerate(ranking, start=1):   # ranks start at 1
            scores[doc] = scores.get(doc, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

bm25  = ["A", "B", "E", "C", "G"]
dense = ["C", "B", "D", "F", "A"]
print(rrf([bm25, dense]))   # ['B', 'C', 'A', 'E', 'D', 'F', 'G'] (D and E tie; order is insertion order)

Pinecone documents three hybrid patterns: filter-then-rank (Documents API), client-side RRF over separate keyword and dense searches, and a single index storing a dense and a sparse vector per record with alpha weighting (Vectors API). Its docs add a useful caution: "RRF is a fusion method, not a synonym for hybrid search."[22]

HyDE: retrieve with a hypothetical answer

HyDE (Hypothetical Document Embeddings) asks an instruction-following LM to write a document that answers the query. The text "captures relevance patterns but is unreal and may contain false details"; an unsupervised contrastive encoder (Contriever) embeds it, and the resulting vector is used for nearest-neighbour search over the real corpus. The authors describe the encoder as a lossy compressor that filters out the hallucinated details, and report gains over the Contriever baseline across web search, QA and fact verification, all without relevance labels.[23]

Step-back, rewriting and expansion

• Rewrite-Retrieve-Read inserts a rewriting step before retrieval. The rewriter can be a prompted LLM or a small trainable T5 tuned with reinforcement learning, and the reader can stay a frozen black-box LLM. It was tested on HotpotQA, AmbigNQ, PopQA and MMLU.[27]
• Step-back prompting has two stages: abstraction (derive a higher-level concept or question) and reasoning. With PaLM-2L the authors report +7% on MMLU physics, +11% on chemistry, +27% on TimeQA and +7% on MuSiQue. It is a reasoning method that also covers retrieval-augmented use (TimeQA), not a retrieval trick alone.[24]
• Query2doc few-shot prompts an LLM for a pseudo-document and concatenates it with the query; for BM25 the original query is repeated several times to balance the weights. Reported: 3-15% BM25 gains on in-domain sets, only 0.4-4% for dense retrievers, and larger LLMs help more.[25]
• RAG-Fusion generates several query variants, retrieves for each, and fuses the lists with Reciprocal Rank Fusion before the LLM answers.[26]

Multi-query fusion with RRF, worked through

RAG-Fusion scores each document by summing 1/(k + rank) across the lists produced by the query variants.[26] The original paper defining RRF fixed k = 60 in a pilot and did not alter it afterwards[16], and Elasticsearch defaults rank_constant to 60[19]. Vendors differ, though: Qdrant's docs state RRF's k is "set to 2 by default"[21], so check your engine's default before comparing rankings.

# Query transformation sketch: multi-query + HyDE, fused with RRF.
# llm(), embed(), dense_search() and bm25_search() are placeholders for your stack.
def rrf(lists, k=60):
    scores = {}
    for ranked in lists:
        for rank, doc_id in enumerate(ranked, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

def retrieve(question, n_variants=3, top_k=20):
    variants = llm(f"Write {n_variants} search queries for: {question}").splitlines()
    fake_doc = llm(f"Write a short passage answering: {question}")   # HyDE
    lists = [bm25_search(q, top_k) for q in [question, *variants]]
    lists.append(dense_search(embed(fake_doc), top_k))
    return rrf(lists)[:top_k]

Interleaving retrieval and reasoning (IRCoT, ReAct)

IRCoT alternates retrieval with chain-of-thought steps: the CoT so far guides what to retrieve next, and the retrieved passages improve the following CoT step. The paper reports retrieval gains of 11-21 points and QA gains of up to 15 F1 across HotpotQA, 2WikiMultihopQA, MuSiQue and IIRC, working from 0.2B to 175B models, and up to 50% fewer factual errors in the CoT.[29] ReAct interleaves reasoning traces with actions such as Wikipedia lookups; it reduced hallucination in QA and fact verification and gave +34% / +10% absolute success on ALFWorld / WebShop.[30]

# IRCoT-style loop (sketch). retrieve() and llm() are placeholders.
def ircot(question, max_steps=5, k=4):
    evidence, cot = [], []
    query = question
    for _ in range(max_steps):
        evidence += retrieve(query, k)                    # 1. retrieve with the current query
        step = llm(question, evidence, cot)               # 2. write ONE next reasoning sentence
        cot.append(step)
        if "answer is" in step:                         # 3. stop when the model commits
            return step, evidence
        query = step                                    # 4. last sentence drives the next search
    return llm(question, evidence, cot), evidence

Self-reflection and routing (Self-RAG, CRAG, Adaptive-RAG)

• Self-RAG: the LM retrieves on demand and critiques both the passages and its own output using special "reflection tokens". The 7B and 13B models are reported to outperform ChatGPT and retrieval-augmented Llama2-chat on open-domain QA, reasoning and fact verification, with better citation accuracy.[31]
• CRAG (Corrective RAG): a lightweight retrieval evaluator returns a confidence; if the static corpus retrieval is inadequate, it falls back to large-scale web search, and a decompose-then-recompose step filters noise. It is plug-and-play over an existing RAG pipeline.[32]
• Adaptive-RAG: a smaller LM classifier predicts query complexity and routes among no retrieval, single-step retrieval and iterative retrieval, improving efficiency and accuracy against adaptive baselines (NAACL 2024).[33]

Agentic retrieval and its cost

Anthropic distinguishes workflows (LLMs and tools orchestrated through predefined code paths) from agents (the LLM directs its own process and tool use), and recommends starting simple because a single optimized LLM call with retrieval is often enough; it also stresses investing in tool definitions.[34] OpenAI's file_search tool in the Responses API is a hosted retrieval tool the model chooses to invoke, doing semantic and keyword search over vector stores, with max_num_results and metadata filters; its docs do not detail internal query rewriting or ranking.[35]

In Anthropic's multi-agent research system a lead agent spawns parallel subagents that search iteratively. They report agents using about 4x the tokens of chat and multi-agent systems about 15x, token usage explaining 80% of performance variance on a browsing evaluation, and an Opus 4 lead with Sonnet 4 subagents beating single-agent Opus 4 by 90.2% on their internal evaluation.[36]

Three ways to score a (query, document) pair

Bi-encoder vs cross-encoder vs late interaction

Per the Sentence-Transformers docs, a bi-encoder embeds each text independently and compares vectors with cosine similarity, while a cross-encoder processes the pair jointly and outputs a score; it produces no reusable embedding, and it is more accurate but slower.[43] The same docs give a concrete cost example: clustering 10,000 sentences with a cross-encoder needs about 50 million pairwise comparisons, roughly 65 hours.[43] BGE describes its reranker the same way: it takes the question and document as input and directly outputs a similarity, not an embedding, and a full system is a first-stage retriever followed by the reranker.[47]

ColBERT sits in between. Query and document are encoded independently by BERT, so document representations can be computed offline, and the score is MaxSim: for each query token take the maximum cosine against the document's token vectors, then sum.[48] The paper reports about 170x faster and 14,000x fewer FLOPs per query than BERT rerankers at competitive quality on MS MARCO and TREC CAR, and shows it used either as a reranker over BM25 top-1000 or end-to-end with FAISS.[48]

The two-stage funnel

The sizing of N and k is an engineering choice you tune against your own evaluation set. The vendor and paper limits above are hard ceilings; the commonly quoted "retrieve 50 to 100, keep 5 to 10" is practitioner folklore that none of the pages cited here state, so treat it as a starting point, not a rule. Because cross-encoder cost is linear in N, doubling N roughly doubles reranking latency and cost (this follows from one forward pass per pair).

Hosted and open rerankers: API facts

# Hosted: Cohere Rerank (v2 API)
import cohere
co = cohere.ClientV2()          # reads the API key from the environment
docs = ["Refund policy: 30 days.", "Shipping takes 3-5 days.", "Refunds need a receipt."]
resp = co.rerank(
    model="rerank-v4.0-pro",       # model id from the API reference example; check the current list
    query="how do I get my money back?",
    documents=docs,                  # docs recommends at most 1,000
    top_n=2,
)
for r in resp.results:
    print(r.index, r.relevance_score, docs[r.index])

# Local: BGE reranker (FlagEmbedding package)
from FlagEmbedding import FlagReranker
reranker = FlagReranker("BAAI/bge-reranker-v2-m3", use_fp16=True)
pairs = [["how do I get my money back?", d] for d in docs]
scores = reranker.compute_score(pairs, normalize=True)   # sigmoid -> [0, 1]
ranked = sorted(zip(scores, docs), reverse=True)[:2]

Pre-filter, post-filter, and filtering inside the graph

Qdrant's filtering guide states the trade-off plainly: at scale, pre-filtering "breaks too many links in the HNSW graph, causing lower accuracy", while post-filtering with a low-cardinality filter means you often end up discarding a large portion of the results.[50] Its answer is filterable HNSW, which adds links between the leftover points, plus a payload index that lets it switch from HNSW to a payload-index search when the filter's cardinality is low.[50] A newer Qdrant post describes in-place filtering during HNSW traversal, filterable HNSW with extra index-time edges, and ACORN as a query-time repair, with a query planner choosing the path from estimated selectivity.[52]

What each engine does

Self-querying: let the LLM write the filter

Users type natural language, not filter JSON. A self-querying retriever has an LLM parse the request into a semantic query plus a structured metadata filter, then runs both against embeddings and metadata; Elastic's example is "science fiction movies released after 2000 with rating above 8".[55] The LLM must be told the available fields, their types, and the allowed operators, and its output should be validated against that schema before it reaches the database.

# 1) LLM turns text into {query, filter}  (llm_json = your structured-output call)
parsed = llm_json(
    "Fields: genre(str), year(int), rating(float). "
    "Return JSON {query, filter} for: sci-fi movies after 2000 rated above 8"
)
# parsed = {"query": "science fiction",
#           "filter": {"$and": [{"genre": {"$eq": "sci-fi"}},
#                               {"year": {"$gt": 2000}}, {"rating": {"$gt": 8}}]}}

# 2a) Pinecone: filter goes straight into the query
res = index.query(vector=embed(parsed["query"]), top_k=5,
                  filter=parsed["filter"], include_metadata=True)

# 2b) Elasticsearch: put the filter INSIDE knn so k matching docs return
body = {"knn": {
    "field": "embedding", "query_vector": embed(parsed["query"]),
    "k": 5, "num_candidates": 50,
    "filter": {"range": {"year": {"gt": 2000}}},
}}

When the question is really a database query

Counts, sums, joins and "top customer by revenue" are exact computations, not similarity problems; the right tool is text-to-SQL, where an LLM writes SQL against a schema. The standard benchmark is Spider (Yu et al., EMNLP 2018), built so that test databases have schemas the model never saw.[56]

Spider's 2018 figure of 9.7% for the best model is a historical baseline showing how hard cross-domain SQL generation was then, not a statement about current models.[56] In practice, give the model the schema, validate generated SQL (read-only credentials, allow-listed tables, row limits) and log every query.

Microsoft GraphRAG: index, then local or global search

The documented indexing pipeline segments documents into TextUnits, extracts entities, relationships and claims, runs Leiden hierarchical clustering, and summarizes each community. Query modes include Global search (reasoning over community summaries), Local search (expanding from matched entities to their neighbours) and DRIFT (local search enriched with community context). The docs recommend prompt tuning.[41] The original paper tested on corpora of about 1M tokens and reported gains in answer comprehensiveness and diversity over standard RAG.[37]

Lighter alternatives: LightRAG, HippoRAG, Neo4j

• LightRAG builds graph structures into both indexing and retrieval, with dual-level (low-level and high-level) retrieval, combines graph and vector retrieval, supports incremental updates, and is open source.[38]
• HippoRAG is hippocampus-inspired: an LLM, a knowledge graph and Personalized PageRank. It reports up to 20% better multi-hop QA, and single-step retrieval comparable to or better than iterative retrieval while 10-30x cheaper and 6-13x faster.[39]
• Neo4j's neo4j-graphrag Python package is Neo4j's official GraphRAG library (successor to neo4j-genai): a knowledge-graph builder, retrievers (VectorRetriever and others), and a GraphRAG class. It needs Neo4j 5.18.1 or later and supports LLM providers including OpenAI, Anthropic, Cohere, Mistral, Vertex AI and Ollama, plus external vector stores such as Weaviate, Pinecone and Qdrant.[40]

# neo4j-graphrag: vector retrieval wrapped by a RAG class (sketch; check the docs for exact setup)
from neo4j import GraphDatabase
from neo4j_graphrag.retrievers import VectorRetriever
from neo4j_graphrag.generation import GraphRAG

driver = GraphDatabase.driver("neo4j://localhost:7687", auth=("neo4j", "password"))
retriever = VectorRetriever(driver, index_name="chunk_embeddings", embedder=my_embedder)
rag = GraphRAG(retriever=retriever, llm=my_llm)       # my_embedder / my_llm: provider wrappers
answer = rag.search(query_text="Which suppliers connect to both plants?")

When graphs help versus cost (synthesis)

The following is my synthesis from the sources above, not a measured result. Global, whole-corpus sensemaking questions and multi-hop questions about connected entities are where the graph papers report their gains[37][39]. The cost is LLM extraction and summarization at index time, which Microsoft itself warns can be expensive[42]. LightRAG and HippoRAG emphasize incremental updates and cheaper retrieval respectively[38][39]. For simple point lookups, a plain vector or hybrid pipeline is the natural default.

The canonical pipeline

Docling, Unstructured, LlamaParse side by side

The table below contains only what each project's own documentation or paper states; it is not a benchmark.

Choosing a strategy in Unstructured

# Docling: convert, then export structured Markdown
from docling.document_converter import DocumentConverter
doc = DocumentConverter().convert("report.pdf").document
markdown = doc.export_to_markdown()

# Unstructured: choose a strategy explicitly, get typed elements
from unstructured.partition.pdf import partition_pdf
elements = partition_pdf(filename="report.pdf", strategy="hi_res")
tables = [e for e in elements if e.category == "Table"]

Where parsing fails

Three architectures

LangChain's write-up lists these as: (1) multimodal embeddings such as CLIP, passing raw images to a multimodal LLM; (2) LLM summaries of images and tables embedded as text; (3) summaries embedded with a reference to the raw image, which is handed to a multimodal LLM at synthesis time. Option 3 relies on a multi-vector retriever, which decouples the retrieval key from the document used for synthesis.[63]

# Option 3 in plain Python: summary is the key, raw object is the value
docstore = {}          # id -> raw table / image (kept as-is)
index = []             # (vector, id) pairs built from summaries
for obj_id, raw in parsed_objects:
    summary = summarize_with_llm(raw)       # placeholder: your LLM call
    docstore[obj_id] = raw
    index.append((embed_text(summary), obj_id))   # placeholder: your embedder

hits = nearest(index, embed_text(question), k=3)
context = [docstore[obj_id] for obj_id in hits]   # raw objects go to a multimodal LLM

Multimodal embedding models

CLIP was trained on 400M image-text pairs and transfers zero-shot to 30+ vision datasets.[64] Newer hosted models take mixed inputs through one backbone. Voyage notes that a single backbone avoids a CLIP-style problem where text and image vectors align with same-modality items in mixed search.[65]

Jina's vendor page also lists approximate image token cost per 600x600 image: 363 (v5-omni), 4,840 (v4) and 16,000 (clip-v2), which matters when you embed millions of pages.[67]

Page-image retrieval with late interaction

ColPali introduces the ViDoRe benchmark for visually rich document retrieval and a vision-language model that produces multi-vector embeddings directly from page images, skipping text extraction. The authors report that it largely outperforms modern document retrieval pipelines; it appeared at ICLR 2025.[62]

Tables and unified frameworks

Tables are the hardest object: a plain-text flattening loses row and column relationships. Three practical routes follow from the architectures above: parse the table into Markdown or HTML (previous section), summarise it for retrieval while keeping the raw table for the answer step (option 3), or retrieve the page image (ColPali). RAG-Anything takes a broader view: a unified framework over text, images, tables and equations with dual-graph construction (cross-modal plus textual) and cross-modal hybrid retrieval; code is open source at HKUDS/RAG-Anything.[68]