The Art of Cutting Up a Book: Chunking

Here's a confession from someone who builds RAG systems for a living: when a client says "our AI gives terrible answers," my first suspect is never the AI. It's never the vector database either. Nine times out of ten, I go straight to the least glamorous step in the whole pipeline — the one that happens before anything intelligent even starts.

Somebody cut the books wrong.

Every document in your open-book library got there in pieces. Not whole. A 40-page policy doesn't float in the shelf-less library as one book — it floats as dozens of passages, cut up in advance by a process with the profoundly unsexy name of chunking. Whoever — or whatever — did that cutting made hundreds of tiny editorial decisions about where one idea ends and the next begins.

And here's the brutal rule that makes those decisions matter: retrieval can only ever hand the genius the pieces that exist. If the answer to a question got split across two pieces, or buried in the middle of a bloated one, no amount of intelligence downstream can fix it. The world's best librarian cannot fetch a page nobody cut.

Why cut at all?

Two hard constraints force chunking on us; it isn't a stylistic choice.

First, retrieval returns pieces, not books. The whole point of Part 2 was bringing the genius the right three pages, not the whole bookshelf. Something has to define what "a page" is.

Second, the desk is small — and rented. The model reads whatever you place in front of it per question, and you pay for every word, every time (and long inputs make it slower and more distractible, a problem we'll meet properly in Part 9). Hand it three crisp passages and it reasons beautifully. Hand it three bloated ones and the answer is buried in noise, at premium prices.

So: cut we must. The question is only where the scissors land.

Flashcards, done right and done wrong

The best mental model for chunking is one you already know: turning a textbook into flashcards.

A good flashcard holds one complete idea. Enough context to stand alone — you can pick up any single card, cold, and it makes sense. That's exactly what a good chunk is: a passage that a stranger (and the genius is a stranger, every single time) can read in isolation and correctly use.

Now the two classic ways to ruin the deck. Chunking's entire craft lives between them.

Cut too small, and you get confetti. Slice mid-thought and you produce cards that read: "It requires written approval from the manager." What requires approval? Which manager? The card is grammatically fine and informationally dead — its meaning lived in the sentence before the cut. I call this the Confetti Problem, and it is silently rampant. The nastiest version: the map of meaning (Part 3) can't place confetti properly either — "It requires approval" has such vague coordinates that it gets parked in a meaningless neighborhood, findable by nobody. Confetti isn't just useless when retrieved. It's unfindable.

Cut too big, and you get the phone book. Make each card five pages long and every retrieval drags in mostly-irrelevant text. The one sentence that answers the question sits buried under forty that don't. You pay for all of it, the genius must resist being distracted by all of it, and the map coordinates go mushy for the opposite reason — a five-page chunk about returns and shipping and warranties lives at the blurry average of three neighborhoods, matching everything a little and nothing well.

Between confetti and phone book sits the goldilocks zone: chunks big enough to be self-contained, small enough to be about one thing. Two practical tricks help you live there. Overlap — letting each card repeat a little of its neighbor's edge, so a thought straddling the boundary survives on at least one card whole. And breathing with the document's own structure, which deserves its own section.

Cut where the author already cut

Here's the insight that separates decent chunking from naive chunking: documents come pre-cut. The author already divided the thing into sections, headers, paragraphs, list items — every heading is a label reading "new idea starts here." The single biggest chunking upgrade is simply to respect those seams: cut at the section boundary, not at "every 500 words regardless of anything," which is the industrial shredder approach — fixed-width cuts straight through sentences, tables, and reason itself.

Even better: carry the labels along. A chunk that begins "Returns are accepted within 14 days" is good. A chunk that begins "Refund Policy → Sale Items: Returns are accepted within 14 days" is gold — it self-locates. The genius knows exactly what it's holding, and the map files it in a sharper neighborhood.

And when structure is unreliable — transcripts, emails, scanned legacy files — there's a cleverer scissors: let the map itself decide. Walk the document watching the meaning-coordinates drift; when consecutive passages jump neighborhoods, that jump is the seam. Cutting by meaning instead of by formatting. (If you want the full practitioner's depth on all of this — hierarchy-based versus semantic strategies, choosing between them, the PDF horror stories — I've written a whole separate series on chunking on LinkedIn. This article is the doorway; that series is the workshop.)

From the trenches

The failure that made me respect chunking forever: a client's HR bot, asked about notice periods, kept answering with the interns' policy — for everyone, including directors. The retrieval looked perfect. The chunk was clean. But the notice periods lived in a table, and the shredder had cut it from its heading row two pages earlier: sixty rows of numbers, no column names in sight. The genius, handed a table with amnesia about its own headers, did what geniuses do — confidently picked a plausible row.

Nothing in that pipeline was "broken." Every component performed. The books were just cut wrong, and everything downstream inherited the wound, wearing a straight face.

The catch

The catch with chunking is that there's no universal right answer — the perfect cut for legal contracts mangles chat transcripts; the ideal for API docs ruins financial tables. Chunking is less like configuring software, more like editing: judgment, applied to your documents, tested against your real questions. Anyone selling a one-size-fits-all chunk size is selling the shredder with better marketing.

The one line to remember

Your AI can only ever be as smart as the pieces you cut — retrieval finds chunks, not answers, and confetti is unfindable.

So: the books are cut well, filed by meaning, floating in the shelf-less library. A question is about to arrive. Now comes the librarian's actual moment — out of two million well-cut pieces, pick the five that matter, in the time it takes you to blink.

Next — Part 6: "Ctrl+F for Meaning." Why the best retrieval uses two kinds of search at once, and the elegant hiring-funnel trick — the Second Interview — that catches what both of them miss.