When RAG Lies: Why \"AI With Sources\" Still Gets It Wrong

RAG was supposed to end hallucinations. That was the pitch, remember? Ground the model in real documents, make it cite sources, and the confident-nonsense problem goes away.

So explain this one: a client's support bot — proper RAG, well-built, citations and all — told a customer their enterprise plan included a feature that had been discontinued for two years. The answer had a citation. The citation was real. It pointed to an actual document in the company's own library. The bot didn't hallucinate anything.

The document was just wrong — a pricing PDF from two product generations ago, sitting in the library like expired medicine in a cabinet, because nobody's job was to throw it out.

This is the most important article in the series, and the one vendors would prefer I skip. Because here's the honest status of RAG: it dramatically reduces the genius-inventing-things problem — and replaces it with a new family of quieter failures, each one arriving with a straight face and a citation. There are five. You've already met most of their ingredients; today we watch them cause accidents.

Failure 1: The Photocopier Problem

The open-book student's nightmare: perfect exam technique, wrong pages photocopied.

Retrieval fetched confidently — and fetched the wrong thing. The question was about termination notice periods; the funnel returned chunks about probation notice periods. Same neighborhood on the map of meaning, same vocabulary, wrong answer. The genius, doing exactly his job, synthesizes what he was handed, beautifully, with citations.

This is the purest RAG failure because every component worked as designed. The map placed things reasonably; the passages simply live close together. And the negation trap from Part 3 is this failure's sharpest knife: "cannot be refunded" retrieved for a refund-eligibility question. Nothing crashed. The answer is grounded — in the wrong ground.

Failure 2: The Confetti Problem, in production

You know this one from Part 5 — bad chunking — but here's how it presents at the emergency room: the bot quotes a table row with no header, applies "the following exceptions" without the exceptions list that lived in the next chunk, or states a rule whose "unless" clause got cut onto different confetti. The citation points at a real document. The document even contains the truth. The truth just got beheaded on its way into the library, and the model answered from a partial quote — the AI equivalent of quoting someone mid-sentence.

Failure 3: The expired-medicine cabinet

The opening story. Nothing about a library makes old documents leave it. The 2023 pricing sheet, the superseded policy, the draft that never shipped, seven copies of the handbook with two contradicting each other — retrieval doesn't know which is current unless something tells it. The map of meaning is no help at all: an outdated policy and its replacement are nearly identical in meaning, which means they float side by side, and the coin-flip of which gets retrieved decides what your company tells a customer.

This failure isn't really an AI problem. It's a librarianship problem — and it's the single most common one in real deployments, because every company's document hygiene is worse than it believes.

Failure 4: The stubborn genius

Sometimes the desk is set perfectly — right chunks, current documents, clean cuts — and the model answers from memory anyway. The retrieved passage says your product's limit is 50 users; the model, having read ten thousand SaaS pricing pages during training, breezily says 100, because that's what such products usually say. The book was open; the genius glanced at it and trusted his gut.

This tug-of-war between what the model was handed and what it "already believes" is a real, studied phenomenon — and it worsens exactly when retrieval quality drops, because a passage that only half-answers gives the model room to fill gaps the old-fashioned way: by plausible invention. The amnesiac genius from Part 1 never fully left the building. RAG built him a library; it didn't remove his imagination.

Failure 5: The mangler

The subtlest one. Right passages, model genuinely reading them — and the synthesis is wrong. Two clauses merged into a rule that appears in neither ("14-day returns" + "defective items: 30 days" → "defective sale items: 44 days" is a caricature, but the real versions are exactly this shape). A condition from one paragraph applied to a different paragraph's rule. A number transcribed correctly and attributed to the wrong plan. Each source chunk: accurate. The citation: valid. The combination: fiction. This is the failure that survives spot-checks longest, because verifying it requires actually reading the sources side-by-side with the answer — which is precisely the work the bot was bought to eliminate.

The turn: why this is actually the good news

Five ways to be confidently wrong, each wearing a citation. Grim? Here's the reframe that makes this the optimistic article in the series.

Go back to Part 1's closed-book genius. When he invented an answer, what could you do? Nothing. No trail, no stations, no anywhere to look. The failure was a fog inside a hundred billion model weights.

Now look at the five failures again. Every single one has an address in the kitchen: wrong pages → inspect the funnel and add the Second Interview. Confetti → fix the scissors. Expired medicine → document hygiene, freshness dates, deprecation rules for the library. Stubborn genius → sharpen the exam rules ("answer only from the passages; say so if they don't cover it") and grant explicit permission for "I don't know" — a right the eager-to-please genius must be given, loudly. The mangler → automated checks that re-read answers against their own citations before they ship, which mature systems increasingly run as a matter of course.

You don't have to remember the fixes. Remember the shape: RAG converts hallucination from a mystery into a maintenance schedule. Failures moved out of the fog and onto stations you can name, test, and repair. That's not a smaller achievement than "no more errors." In engineering, it's the achievement that actually exists.

The one line to remember

RAG doesn't make AI honest. It makes AI's dishonesty inspectable — and inspectable problems are the only kind that ever get fixed.

By now, though, a fair question may be forming: this open-book machinery is real work to build and maintain. Is it always the right tool? Couldn't you just... train the model on your documents? Or dump everything into one of those million-token desks and skip the library? Every AI team has this exact argument, usually expensively.

Next — Part 9: "Book, Study, or Bigger Desk?" RAG vs. fine-tuning vs. long context — a decision your team can make in five minutes with one framework, instead of five months with one regret.