What is the difference between recursive and semantic chunking?

Recursive chunking splits on document structure, trying headings, paragraphs, sentences, then characters until chunks fit a size limit. Semantic chunking uses embeddings to detect topic changes. Recursive is cheaper and works well on structured documents; semantic helps with long unstructured text like transcripts.

What is hierarchical chunking?

Chunking at multiple levels, such as small passages and their parent sections. Search runs on the small passages for precise matches, and the system returns the larger parent so the model receives enough context. It combines well with either recursive or semantic splitting.

How do you implement semantic chunking?

Split text into sentences, embed each sentence or small window, compute similarity between neighbours, start a new chunk where similarity falls below a threshold, and enforce minimum and maximum chunk sizes. LangChain and LlamaIndex provide utilities, or you can implement it directly with any embedding model.

What Is Semantic Chunking in RAG?

Semantic chunking splits a document into pieces at points where the meaning changes, rather than every fixed number of characters or tokens. It typically embeds sentences, measures similarity between neighbours, and starts a new chunk where similarity drops. The goal is chunks that each cover one coherent idea, which embed and retrieve more precisely.

How semantic chunking works

1. Split into sentences. The document is broken into sentences or short units.

2. Embed each unit. Each sentence, or a small window of neighbouring sentences, is embedded.

3. Compare neighbours. The similarity between each sentence and the next is calculated.

4. Find breakpoints. Where similarity drops below a threshold, or falls sharply relative to the document's typical similarity, the topic is assumed to change and a new chunk begins.

5. Apply size limits. Chunks that grow too large are split further; very small ones may be merged, so chunks stay within the embedding model's input limit and remain useful.

The result is chunks whose boundaries follow the content rather than an arbitrary character count. A section describing installation stays together; the next section about configuration starts a new chunk even if the first was short.

Frameworks including LangChain and LlamaIndex provide semantic chunking utilities, and the approach can be implemented directly with any embedding model.

Choosing a breakpoint threshold

Fixed similarity thresholds behave differently across documents, so many implementations use a percentile of the document's own similarity scores instead. Inspect a sample of chunk boundaries on real documents after changing the threshold, then confirm the effect on retrieval with labelled queries.

Semantic, recursive, fixed, and hierarchical chunking

Fixed-size chunking splits every set number of tokens, often with overlap between chunks. It is fast and predictable, and it cuts through sentences, tables, and code blocks without regard for meaning.

Recursive chunking tries a list of separators in order, such as section headings, then paragraph breaks, then sentences, then characters, splitting at the largest structure that keeps chunks under the size limit. LangChain's RecursiveCharacterTextSplitter is a widely used implementation. For well-structured documents it produces coherent chunks cheaply, because authors already put breaks where topics change.

Semantic chunking uses embeddings to find topic changes. It helps most where structure is weak or misleading: transcripts, long unbroken prose, notes, and conversations.

Hierarchical chunking creates chunks at several levels, such as small passages for precise matching and their parent sections for context. Search runs on small chunks; the system then returns the larger parent so the model sees enough surrounding material.

Structure-aware chunking for code and markup splits on functions, classes, or document elements rather than text boundaries, keeping units that are meaningful to a developer intact.

These methods combine. A common strong setup is structure-aware or recursive splitting first, semantic refinement for long unstructured sections, and hierarchical retrieval for context.

When semantic chunking is not worth it

It requires embedding every sentence during indexing, which adds cost and time, and threshold choices affect results. On documents with clear headings and paragraphs, recursive splitting often performs about as well at a fraction of the cost. Test both on your own queries before committing.

Practical guidance

Size for the question, not the document. Chunks should be large enough to answer a typical query on their own and small enough that their embedding represents one idea.

Keep context attached. Store the document title, section heading, and source path as metadata, or prepend them to chunk text, so a chunk that says configure the timeout still carries which system it refers to.

Use overlap carefully. Small overlaps prevent answers being cut in half at boundaries; large overlaps waste storage and return near-duplicates.

Respect special content. Keep tables, code blocks, and lists intact rather than splitting them mid-structure.

Evaluate retrieval, not chunks. The measure of a chunking strategy is whether the right content comes back for real queries. Compare strategies using the same embedding model and a labelled query set.

Re-chunk when sources change shape. New document types, such as transcripts added to a documentation index, may need a different strategy.

In RDK, indexed vault files are split into encrypted private chunks, and chunk quality is a large part of why reference-heavy questions can be answered from retrieval instead of a fresh model call.

Chunking code

Source code chunks best along syntax: functions, classes, and modules, with the file path and enclosing class kept as context. Splitting code by character count separates a function from its signature or docstring and produces chunks that match queries poorly.

Frequently asked questions

What is semantic chunking in RAG?: A way to split documents for retrieval at points where the meaning changes. Sentences are embedded, similarity between neighbours is measured, and a new chunk begins where similarity drops, producing chunks that each cover one coherent idea and therefore retrieve more precisely than arbitrary fixed-length pieces.
What is the difference between recursive and semantic chunking?: Recursive chunking splits on document structure, trying headings, paragraphs, sentences, then characters until chunks fit a size limit. Semantic chunking uses embeddings to detect topic changes. Recursive is cheaper and works well on structured documents; semantic helps with long unstructured text like transcripts.
What is hierarchical chunking?: Chunking at multiple levels, such as small passages and their parent sections. Search runs on the small passages for precise matches, and the system returns the larger parent so the model receives enough context. It combines well with either recursive or semantic splitting.
How do you implement semantic chunking?: Split text into sentences, embed each sentence or small window, compute similarity between neighbours, start a new chunk where similarity falls below a threshold, and enforce minimum and maximum chunk sizes. LangChain and LlamaIndex provide utilities, or you can implement it directly with any embedding model.