How to Do Semantic Chunking in Python

Step 1: split into sentences

Use a sentence splitter suited to your text. Simple regular expressions work for clean prose; libraries such as NLTK or spaCy handle abbreviations and edge cases better. For Markdown notes, split on headings first and chunk within each section, so semantic boundaries never cross a heading.

Step 2: embed sentences with a small window

Embedding single sentences is noisy, because short sentences carry little meaning. A common approach embeds each sentence together with its neighbours, for example one before and one after, and normalises the vectors. Any embedding model works; local models from the sentence-transformers library avoid API costs during experimentation.

Step 3: compute similarity between neighbours

For normalised vectors, cosine similarity is a dot product. Compute it for each adjacent pair, then convert to distance as one minus similarity. Peaks in distance mark likely topic changes.

Step 4: choose boundaries

Set a threshold at a high percentile of the distances in the document, so only the sharpest drops become boundaries. A percentile adapts to each document's writing style. Lower percentiles create more, smaller chunks; higher ones create fewer, larger chunks.

A minimal implementation

import re
import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

def semantic_chunks(text, window=1, pct=90, max_chars=2000):
    sents = [s.strip() for s in re.split(r"(?<=[.!?])\s+", text) if s.strip()]
    if len(sents) < 3:
        return [" ".join(sents)]
    ctx = [" ".join(sents[max(0, i - window): i + window + 1]) for i in range(len(sents))]
    emb = model.encode(ctx, normalize_embeddings=True)
    dist = 1 - np.sum(emb[:-1] * emb[1:], axis=1)
    cut = np.percentile(dist, pct)
    chunks, cur = [], [sents[0]]
    for i, d in enumerate(dist):
        nxt = sents[i + 1]
        if d > cut or len(" ".join(cur)) + len(nxt) > max_chars:
            chunks.append(" ".join(cur))
            cur = []
        cur.append(nxt)
    chunks.append(" ".join(cur))
    return chunks

Step 5: enforce size limits

Semantic boundaries can create one-sentence fragments or very long chunks. Merge fragments below a minimum size into a neighbour, and split anything above the maximum, as the example does with a character cap. Keep the maximum below the embedding model's input limit, or the end of the chunk is silently ignored.

Step 6: embed chunks and test retrieval

Embed the final chunks, store them with source and position metadata, and run a set of real questions whose answer locations you know. Measure how often the right chunk appears in the top results. Compare against recursive fixed-size chunking on the same questions. Keep semantic chunking only if it wins.

Libraries that do this

LangChain and LlamaIndex both include semantic chunkers built on the same idea, with options for breakpoint thresholds. Writing the loop yourself is still useful: it makes the threshold and windowing visible, which is where most tuning happens.

Feeding chunks into RDK

RDK indexes vault files and code into chunks on the RDK network, stored as encrypted private chunks that agents search before calling a model. Clean, self-contained chunks raise the share of questions answered from retrieval, which is where RetroDeck's reported 80 to 90 percent token savings on repeated work come from.

Tuning tips

Increase the window if boundaries look random; short sentences need more context. Lower the percentile if chunks are too long, raise it if they are too fragmented. Strip boilerplate such as navigation text and repeated footers before chunking, since it distorts similarity. For multilingual content, use a multilingual embedding model for boundary detection.

Handling updates

Store a hash of each source file. When a file changes, re-chunk and re-embed only that file, then replace its chunks in the index. Semantic boundaries can shift after edits, so replace all chunks for the file rather than patching individual ones.

Where chunking code goes wrong

Typical mistakes include embedding single short sentences without a window, using a fixed similarity threshold across documents with different styles, forgetting to cap chunk length, chunking across headings, and judging quality by reading a few chunks instead of measuring retrieval. Each produces chunks that look reasonable but retrieve poorly. Another is using one embedding model for boundaries and a different one for search without testing.

Storing chunks with metadata

Save each chunk with its source path, heading path, start and end sentence index, and file hash. Metadata lets you cite sources, filter by project, rebuild surrounding context, and replace chunks cleanly when a file changes.