Fixed-Size vs Semantic Chunking
Fixed-size chunking splits text into pieces of a set token or character length, usually with overlap. Semantic chunking splits where the topic changes, typically by comparing embeddings of adjacent sentences. Fixed-size is simple, fast, and predictable; semantic chunking keeps ideas together but costs more to compute and produces uneven sizes. Test both on your own queries.
How fixed-size chunking works
Text is cut every N tokens or characters, often with an overlap of a fraction of the chunk so that content at the boundary appears in both neighbours. Recursive variants try separators in order, such as paragraphs, then sentences, then words, to avoid cutting mid-sentence while staying near the target size. It needs no model calls and runs at any scale.
How semantic chunking works
The text is split into sentences, each sentence or small window is embedded, and similarity between neighbours is measured. Where similarity drops sharply, a boundary is placed. The result is chunks that each cover one topic, with sizes that vary with the writing. Thresholds are usually set by percentile or standard deviation of the similarity drops.
Retrieval quality
Semantic chunks tend to keep an explanation and its conclusion together, which helps when questions target a specific idea. Fixed-size chunks can split an answer across two pieces, so the retriever returns half of it. On structured documents, such as code, tables, or short records, the advantage often disappears, and structure-aware splitting beats both.
Cost and operations
Fixed-size chunking is deterministic and free. Semantic chunking needs an embedding per sentence at index time, plus the final chunk embeddings, and re-chunking when content changes. Its variable sizes complicate token budgeting: one chunk might be a sentence, another a page.
How to choose
Start with recursive fixed-size chunking at a moderate size with overlap. Build a small set of real questions with known answer locations. Measure recall at k for fixed-size and semantic variants. Switch only if semantic chunking measurably improves retrieval on your content, and cap maximum chunk size either way.
Chunking in RDK
RDK indexes vault files and code as chunks, stored as encrypted private chunks. Good chunk boundaries matter more there, because agents retrieve chunks instead of asking the model to regenerate answers, and a chunk that holds a complete answer is one fewer LLM call.
Structure-aware splitting
Many corpora have structure that beats both methods. Markdown headings, code functions and classes, table rows, and email threads all give natural boundaries. Splitting on structure first, then applying fixed or semantic splitting within long sections, usually produces the most retrievable chunks. Obsidian vaults are a good example: headings already mark topics.
Metadata matters as much as boundaries
Store each chunk with its source file, heading path, position, and timestamps. Metadata lets retrieval filter by project or date, lets answers cite sources, and lets you rebuild context around a hit. Poor metadata makes even perfect boundaries hard to use.
Hybrid approaches
Many pipelines combine methods: split on structure, apply recursive fixed-size splitting to long sections, and use semantic checks only to avoid cutting inside a closely related run of sentences. This keeps indexing costs low while still fixing the worst boundary errors on long prose sections.
| Aspect | Fixed-size | Semantic |
|---|---|---|
| Split rule | Token or character count | Drop in similarity between neighbours |
| Chunk size | Uniform | Variable |
| Indexing cost | None beyond final embeddings | Embedding per sentence plus final embeddings |
| Strength | Simple, predictable, fast | Keeps ideas together |
| Weakness | Can split answers across chunks | Uneven sizes, threshold tuning |
Frequently asked questions
- Is semantic chunking better than fixed-size chunking?
- Sometimes. Semantic chunking often helps on long prose where topics shift, but it costs more and produces uneven chunks. On code, tables, and short structured records, structure-aware or fixed-size splitting can work as well or better. Measure recall on real queries before switching.
- What chunk size should I use for RAG?
- There is no universal size. Smaller chunks give precise matches but may lack context; larger chunks carry context but dilute relevance. Start moderate with some overlap, then test several sizes against real questions and keep the one with the best retrieval results.
- Does chunk overlap help?
- Usually. Overlap repeats boundary text in neighbouring chunks, so a sentence split by a fixed boundary still appears whole in one of them. It increases index size slightly. Semantic chunking needs less overlap because boundaries already follow topic changes. Overlap is usually set as a small fraction of chunk size.