Semantic vs Hierarchical Chunking
Semantic chunking splits text at points where meaning changes, producing one level of topic-aligned chunks. Hierarchical chunking creates nested levels, such as small child chunks inside larger parents, retrieving on the small ones for precision and returning the parent for context. Semantic chunking improves boundaries; hierarchical chunking balances precision and context. They can be combined.
Semantic chunking in brief
Sentences are embedded, similarity between neighbours is computed, and boundaries go where similarity drops. The output is a flat list of chunks, each roughly one topic. Retrieval returns those chunks directly.
How hierarchical chunking works
The document is split into large parent chunks, such as sections, and each parent is split into smaller child chunks, sometimes over three or more levels. Only children are embedded for search. When a child matches, the system returns its parent, or merges sibling children, so the model sees the surrounding context. Frameworks call this parent document retrieval or auto-merging retrieval.
Precision versus context
Small chunks match questions precisely because they contain little else. Large chunks give the model enough surrounding text to answer correctly. Flat chunking forces one size to do both jobs. Hierarchical chunking separates them: search small, read large.
Indexing cost of each
Semantic chunking spends embeddings at index time to place boundaries. Hierarchical chunking stores more records and links, and returns more tokens per hit because parents are larger. Token budgets need a cap on how many parents are sent.
Using both together
Semantic boundaries can define parents, such as topic sections, while fixed or semantic splits define children. This keeps parents coherent and children precise. Document structure, such as headings in Markdown notes, is often the best parent boundary of all.
When to use which
Use semantic chunking alone for prose-heavy corpora where questions target single ideas. Use hierarchical chunking when answers need surrounding context, such as procedures, specifications, and long notes. Evaluate both on real questions, measuring answer accuracy as well as retrieval recall, since hierarchical retrieval mainly improves what the model reads.
Hierarchical chunking for code
Code has a natural hierarchy: repository, directory, file, class, function. Searching at function level and returning the enclosing class or file header gives an agent the signature, imports, and neighbouring methods it needs. Semantic boundaries add little here, because syntax already defines the units.
Evaluating the choice
Build a question set with known answers and the sections that contain them. Measure retrieval recall for the matched unit and answer accuracy for the full pipeline. Hierarchical retrieval often shows its value in accuracy rather than recall, because the model reads more surrounding context. Track tokens per answer as well, so the context gain is weighed against cost.
Operational notes
Hierarchical indexes need parent IDs on every child and a fast lookup from child to parent. When documents change, update children and parents together to avoid stale links. Deduplicate when several children from the same parent match, so the same section is not sent twice.
Agentic chunking
A newer variant uses a language model to decide boundaries or group propositions into chunks. It can produce very coherent units but costs model calls for every document, which is hard to justify at scale. It is best reserved for small, high-value corpora.
| Aspect | Semantic | Hierarchical |
|---|---|---|
| Main idea | Cut where meaning shifts | Nest small chunks inside larger ones |
| Levels | One | Two or more |
| Search unit | The chunk | Small child chunks |
| Returned to the model | The chunk | The parent or merged children |
| Main cost | Embeddings to find boundaries | Larger index, more tokens per hit |
Frequently asked questions
- What is hierarchical chunking?
- Hierarchical chunking splits documents into nested levels, such as sections and paragraphs. Search runs on the small chunks for precision, and the system returns the larger parent chunk so the model has enough context. It is also called parent document or auto-merging retrieval.
- Can semantic and hierarchical chunking be combined?
- Yes. Semantic or structural boundaries can define parent chunks, while smaller splits inside them become children. This keeps parents topic-coherent and children precise. Headings in Markdown or documentation are often natural parent boundaries. Code can use files or classes as parents and functions as children.
- Does hierarchical chunking use more tokens?
- Often, because parents are larger than the matched child chunks. Limit how many parents are sent per query and deduplicate overlapping parents. The extra tokens are worthwhile when they prevent wrong or incomplete answers. Measure answer accuracy alongside token use to judge the trade.