RAG Chunking Strategy SOP: How Big to Cut, How to Cut, How to Evaluate

The most frustrating failure after shipping a RAG system isn't "it can't answer"-it's "it answers wrong." The knowledge base has the relevant docs, but retrieval doesn't surface them, or the chunks that come back are tangential to the question. The first instinct is usually to swap the embedding model, add a reranker, or crank up top-k. All valid, but many teams skip the variable at the very top of the pipeline: chunking. How you split documents decides what semantic unit the embedding sees. Too large, and a chunk packs multiple topics into one diluted vector, dragging similarity down. Too small, and context fractures, leaving answers stitched from fragments. Get chunking wrong and every downstream tweak is a patch on a leaking pipe.

This SOP skips the theory and walks through chunking strategy selection and parameter tuning end to end: how to choose among four mainstream strategies, how to set chunk_size and overlap, how to write the code, how to evaluate, and how to iterate. Unlike our "RAG System Evaluation SOP," which covers evaluating the whole RAG pipeline, this one zooms in on the "cut" at the pipeline's upstream-chunking is the foundation of retrieval quality; get it right, and embedding and reranking have room to shine.

1. Why Chunking Is the Gatekeeper of RAG Retrieval Quality

RAG retrieval works on embedding similarity: turn the user query and each document chunk into a vector, compute cosine similarity, return the top-k. The embedding's input is never the whole document-it's the chunk you cut. The chunk is the atomic unit of retrieval, and its size and boundaries dictate what semantic range the vector "sees."

Both extremes break:

• Chunk too large: one chunk mashes together multiple topics (say, three contract clauses). The embedding compresses them into a single vector, diluting the signal. The user asks about "liquidated damages," but this chunk's similarity is dragged down by the unrelated clauses, so it ranks low or never surfaces. Large chunks also eat the LLM's context window, reducing how many chunks fit in top-k and the effective information delivered.
• Chunk too small: a chunk holds a single sentence or fragment, losing context. The user asks "how does annual leave work," retrieval surfaces a chunk that only says "10 days at 1 year tenure," missing "15 days at 10 years" (which landed in a different chunk), so the answer is incomplete.

The hard truth: chunking isn't a throwaway preprocessing step-it's the decision that caps retrieval quality. Cut it right, and the embedding can express semantics accurately. Cut it wrong, and no amount of model swapping saves you.

2. Four Chunking Strategies: Principles and Use Cases

Four mainstream strategies, each with its own fit. No silver bullet.

Each in turn.

1. Fixed-size. Hard-cut every N tokens or characters (e.g., 500 tokens per chunk). Simplest, built into every framework. But the cost: a complete sentence or clause can be split mid-way, leaving both halves semantically incomplete. Fine for quick validation on uniform plain text, but almost never the right choice in production.

2. Recursive. Try a hierarchy of separators in order: first paragraph breaks (\n\n), then line breaks (\n), then sentence enders (.), finally spaces and empty strings as fallback. The core idea: "cut at natural boundaries when possible, hard-cut only when necessary." LangChain's RecursiveCharacterTextSplitter is the canonical implementation. It's the safe default for general text and Markdown-chunks stay within size while avoiding mid-sentence cuts wherever possible.

3. Semantic. Instead of fixed length, split the document into sentences, compute embedding similarity between adjacent sentences, and cut at "breakpoints" where similarity drops sharply. The principle: semantically coherent sentences cluster into a chunk; a semantic jump marks the boundary. Advantage: cut points naturally align with topic shifts. Disadvantage: requires running embeddings over every document, adding cost and latency. Best for long documents with wide topic range (e.g., a whitepaper mixing product intro, pricing, and case studies).

4. Structure-aware. Don't cut by length-cut by the document's own hierarchy: Markdown by headings (#, ##), HTML by DOM nodes, PDF by paragraphs and sections. Each chunk carries its full hierarchical context ("this fragment belongs to this section under this heading"), making retrieval semantically complete. LlamaIndex's MarkdownNodeParser and Unstructured's partitioning fall in this category. Best for structured technical docs, product manuals, regulatory text.

Selection in one line: when there's structure, use structure-aware; without structure, recursive is the safe fallback; for long docs with wide topic range, consider semantic; fixed-size only for quick validation.

3. Key Parameters: How to Set chunk_size and overlap

Once the strategy is chosen, two parameters decide the outcome.

chunk_size. The empirical range is 256-1024 tokens. No universal value-tune by corpus type:

• FAQ, SOP entries, short Q&A: 300-500 tokens, since each entry is a self-contained semantic unit.
• General docs, product descriptions, blog posts: 500-800 tokens, balancing semantic completeness and retrieval precision.
• Contract clauses, regulations, long manuals: 800-1200 tokens, to keep a clause intact.

Two traps to remember: bigger isn't better (vector dilution), smaller isn't better (context fracture). 256-1024 is the safe band, but the final value must come from testing your own corpus and eval set-no gut-feeling silver bullet.

overlap. The overlap between adjacent chunks, there to prevent losing context at cut boundaries. The empirical value is 10%-20% of chunk_size:

• chunk_size 500 -> overlap 50-100
• chunk_size 1000 -> overlap 100-200

Overlap of 0 is the most common pitfall: a sentence gets split in half, the first half in chunk A, the second in chunk B, and retrieving A gives an incomplete answer. But overlap shouldn't be too large either-excessive overlap duplicates content across chunks, wasting storage, diluting retrieval precision, and potentially causing top-k to return redundant content. 10%-20% is the sweet spot.

4. Code Example: Recursive Chunking in Practice

Using LangChain's RecursiveCharacterTextSplitter (API per official docs):

# pip install langchain-text-splitters
from langchain_text_splitters import RecursiveCharacterTextSplitter

# For Chinese docs, include CJK sentence-ending punctuation in the separator hierarchy
splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,
    separators=["\n\n", "\n", "。", "！", "？", "；", "，", " ", ""],
    length_function=len,
)

chunks = splitter.split_text(document_text)

print(f"Split into {len(chunks)} chunks")
for i, chunk in enumerate(chunks[:3]):
    print(f"--- chunk {i} ({len(chunk)} chars) ---\n{chunk[:80]}...")

Key points:

• separators is a ranked list. The splitter tries them in order: cut by \n\n first; if a piece still exceeds chunk_size, fall back to \n; then 。; and so on.
• For Chinese documents, always include CJK sentence-ending punctuation (。！？；) in the separators. Otherwise the splitter only recognizes English punctuation and hard-cuts mid-sentence.
• length_function=len counts characters. To count by tokens, swap in length_function=tiktoken_len (requires tiktoken), making chunk_size in token units.
• LlamaIndex's equivalent is SentenceSplitter (from llama_index.core.node_parser import SentenceSplitter), same parameter names, method get_nodes_from_documents(documents). API per official docs.

5. Advanced: Parent-Child Chunking and Late Chunking

Basic strategies answer "how to cut." Advanced strategies address the tension between retrieval precision and context completeness.

Parent-Child Chunking. Core idea: cut documents into large chunks (parent), then split each parent into smaller chunks (child). At retrieval, use child embeddings for precise hits, but feed the parent chunk to the LLM. Retrieval precision is guaranteed by small chunks (focused semantics, no dilution), while answer context is guaranteed by the parent (complete paragraph). LangChain's ParentDocumentRetriever implements this pattern. Best for long documents where you need both precise recall and complete answers.

Late Chunking. Traditional chunking is "cut first, then embed." Late chunking reverses it: run the whole document through the embedding model to get token-level representations, then cut by boundaries and aggregate token vectors into chunk vectors. The benefit: each chunk's vector carries whole-document context (not isolated encoding), improving retrieval on long documents. Jina introduced this approach in 2024 and open-sourced the implementation. Best for long documents with strong cross-paragraph semantic links.

In one line: parent-child uses "division of labor between sizes" to resolve the precision-context trade-off; late chunking uses "encode first, cut later" to resolve the isolation problem. Both add complexity-exhaust basic strategies first.

6. Evaluation: Let Data Decide Whether Chunking Works

Don't ship chunking changes on "looks like recall is fine"-quantify it. Consistent with our "RAG System Evaluation SOP," chunking directly affects the retrieval quality dimension:

• context_recall: was all the information in the gold answer actually retrieved? Chunks too large miss recall (diluted semantics rank low); too small also miss (fractured context, incomplete). This is the metric to watch most when tuning chunking.
• context_precision: do relevant chunks rank near the top? Chunks too large let irrelevant content sneak into the same chunk, dragging precision down.
• Hit rate: does top-k contain the correct chunk? Simple and direct, ideal for quick A/B comparison of different chunk_size values.

Evaluation flow: prepare 50-100 labeled samples (with question and ground_truth), run through Ragas or DeepEval, compare context_recall and context_precision across chunking parameters. For code and metric reading, see our "RAG System Evaluation SOP"-not duplicated here. Key principle: change only one variable at a time (only chunk_size, or only overlap), or you can't attribute results.

7. Five-Step Chunking Tuning Workflow

Stringing it all together into a repeatable process:

1. Analyze document type. Is the corpus structured technical docs, Markdown, or plain text? How wide is the topic range? Are entries self-contained or long-form continuous? Document type drives strategy selection.
2. Choose a strategy. Structured docs -> structure-aware; no structure -> recursive (safe default); wide topic range in long docs -> consider semantic. Start with recursive-it's the safe default.
3. Set chunk_size and overlap. Start from empirical values by corpus type (general docs 500/50, FAQ 300/30, long docs 1000/100), overlap at 10% of chunk_size.
4. Evaluate. Run context_recall / context_precision / hit rate on labeled samples, comparing parameters. Change one variable at a time.
5. Iterate. If scores fall short, go back and tune parameters or switch strategies. Loop until targets are met. After launch, re-run evaluation when documents update-chunking can silently break on new docs.

8. Four Pitfalls

1. One-size-fits-all fixed-size. Taking the easy route and hard-cutting all documents with fixed length, splitting sentences and clauses mid-way, collapsing recall quality. Fixed-size is only for quick validation-in production, at least use recursive.

2. Ignoring document structure. A Markdown doc has heading hierarchy, but you cut it as plain text, splitting a heading from its body into separate chunks. Retrieval surfaces a body fragment missing the heading context. Structured docs must use structure-aware strategies.

3. Setting overlap to 0. Saving storage by setting overlap to 0, losing context at cut boundaries. Cross-chunk information (a sentence split in two) means retrieval hits only half, giving incomplete answers. Overlap should be at least 10% of chunk_size.

4. Shipping without evaluation. Tuning parameters by feel, deciding "looks like recall is fine," and shipping-only to have users surface missed key docs immediately. Chunking parameters must be quantitatively validated on a labeled eval set, no exceptions. See our "RAG System Evaluation SOP" for methods.

FAQ

Q1: How big should a chunk be? No universal value; the empirical range is 256-1024 tokens. FAQ/short entries 300-500, general docs 500-800, long clauses 800-1200. The final value must come from testing your own corpus and eval set. Principle: big enough to hold a complete semantic unit, small enough not to be diluted by multiple topics.

Q2: How much overlap? Empirically 10%-20% of chunk_size (e.g., chunk_size 500 -> overlap 50-100). Its job is preventing context loss at cut boundaries. Setting it to 0 loses boundary information; setting it too high (over 30%) duplicates content across chunks, wasting storage and diluting retrieval precision.

Q3: Which of the four strategies is best? No silver bullet. With document structure (Markdown/HTML), use structure-aware; for general text, recursive is the safe default; for long docs with wide topic range, use semantic; fixed-size only for quick validation. Start with recursive and switch to advanced strategies only when you hit a ceiling.

Q4: How do I evaluate whether chunking is good? Run retrieval metrics on labeled samples: context_recall (missed retrieval), context_precision (are relevant chunks ranked high), hit rate (did top-k hit). Compare scores across chunk_size/overlap values, changing one variable at a time. For tools and code, see our "RAG System Evaluation SOP."

Q5: Any special notes for Chinese documents? Two things. First, separators: the recursive chunker's separator list must include CJK sentence-ending punctuation (。！？；), or the splitter only recognizes English punctuation and hard-cuts mid-sentence. Second, length measurement: length_function=len counts characters, where one Chinese character counts as 1. If you want to count by tokens, use tiktoken, since one Chinese character typically maps to 1-2 tokens-chunks sized by character count may have a larger actual token count than expected.

Reference Sources

• LangChain Docs - RecursiveCharacterTextSplitter
• LangChain Docs - Text Splitters
• LlamaIndex Docs - Node Parsers
• Unstructured Docs
• Jina AI - Late Chunking