The Goldilocks Cut: How Big Should a Chunk Be?

The most common question I get about chunking is also the most reasonable-sounding: "What's the right chunk size?"

It arrives expecting a number. Five hundred tokens. A thousand. Some blessed value the professionals use. And the questioner's face always does the same thing when I answer honestly: there is no number — there's a zone, and your documents decide where it is.

That's not evasion; it's the actual shape of the craft. Part 1 established that retrieval finds chunks, not answers, and that a mis-cut answer stops existing. Today: the size dimension of the cut — the two opposite ways to ruin the deck, the zone between them, and the practical moves that keep you inside it.

The flashcard standard

The parent series gave us the measuring stick, and everything today calibrates against it: a good chunk is a good flashcard — one complete idea, carrying enough context to make sense to a stranger reading it cold.

"Stranger reading it cold" is the load-bearing phrase. The model is a stranger, every single time. It doesn't know what came before the chunk, what section it lived in, what "this" refers to. A chunk succeeds if it can be picked up alone, understood alone, and used alone. Hold that standard; both failure modes are just ways of losing it.

Failure one: confetti

Cut too small and ideas shatter mid-thought. The parent series named this the Confetti Problem, and its cruelest property bears repeating: confetti isn't just useless when retrieved — it's unfindable. A fragment like "It requires written approval from the manager" is about... what? Approval of what? Which manager? The words are too generic to earn a sharp address on the map of meaning, so it gets parked in a vague neighborhood no query visits. Cut a policy into confetti and you haven't made it harder to find. You've erased it from the searchable world.

Confetti happens when fixed-size cutting ignores where thoughts end — the industrial shredder from the parent series, slicing every N words through sentences, lists, and reason itself.

Failure two: the phone book

Cut too big and every chunk becomes a phone book — technically containing the answer, practically burying it. The five-page chunk covering returns and shipping and warranties fails twice. It fails at findability: its address lands at the blurry average of three neighborhoods, matching everything weakly (readers of The Map of Meaning will recognize the mushy-address problem from the finale). And it fails at usefulness: even when retrieved, the model must locate one relevant sentence in a haystack of forty irrelevant ones — paying for the whole haystack, at premium prices, while the answer risks drowning in its own context.

Small kills findability by vagueness. Big kills it by dilution. The zone lives between: big enough to stand alone, small enough to be about one thing.

Finding your zone

Three practical moves, in the order I'd apply them.

Match the cut to the question, not the document. The right size depends on how your users ask. Support queries ("what's the return window?") want tight, paragraph-scale chunks — one rule, one card. Research-flavored queries ("how does our warranty compare to the returns policy?") want section-scale chunks with room to breathe. Look at fifty real questions before choosing scissors; the questions describe the pieces they're hoping to find.

Overlap the edges. Let each card repeat a little of its neighbor's ending. A thought straddling a boundary then survives whole on at least one card. Overlap is cheap insurance against the split-answer failure from Part 1 — the standard trick, and it works.

Respect the two-idea alarm. When a chunk covers two distinct ideas, cut it — even if it's "under the size limit." When a cut would behead a thought, don't — even if the chunk runs "over." The zone is defined by idea boundaries, not word counts. Size limits are guardrails; ideas are the road.

The catch

Every rule above has a saboteur: documents whose ideas don't come in tidy paragraph sizes. A legal clause can be one idea across two dense pages. A FAQ is forty ideas, three lines each. A data table is one idea that dies the moment you split it anywhere (that horror gets its own article — Part 4). One document set, three incompatible zones — which is why "what's the right chunk size?" has no number, and why the next two parts move past size entirely: where to cut matters more than how much. The author already marked the spots.

The one line to remember

A chunk should be big enough to stand alone and small enough to be about one thing — the size is a zone, and your users' questions tell you where it is.

Next — Part 3: "The Author's Seams." Every heading is the author telling you where to cut. The cheapest chunking upgrade in existence, and why most pipelines ignore it.