Your AI Is a Genius With Amnesia

Here's an experiment that takes thirty seconds. Open ChatGPT — or Claude, or Gemini — and ask it about your company's refund policy. Or your leave policy. Or anything only your organization would actually know.

You'll get an answer, and it will be lovely. Clear structure, friendly tone, maybe a 30-day return window with free shipping on exchanges. Quite possibly better written than your real policy.

It will also, quite possibly, be fiction. The AI didn't look anything up — it has never seen your policy. It wrote what a refund policy usually sounds like, with total confidence, and handed it over like a fact. When a founder I know ran this exact test on his e-commerce store, the AI invented a return window twice as long as his real one. If a customer-facing bot had said that, it wouldn't have been a typo. It would have been a commitment.

If you've used AI for anything involving your own work, you've hit some version of this. The tool that can pass medical licensing exams and write working code somehow knows nothing about your product, your documents, or the email you sent yesterday. And instead of saying "I don't know," it improvises.

This series is about the idea that fixes this. It's called RAG — and it's quietly becoming the most important pattern in applied AI. But before the fix means anything, you need to feel the problem properly. That's today's job.

Meet the genius

Imagine you could hire a consultant with an absurd résumé. They've read essentially the entire public internet — every textbook, every Wikipedia article, millions of contracts, codebases, and customer service transcripts. Ask them anything about marketing, law, chemistry, or 14th-century poetry, and they answer instantly, fluently, in any language you like.

There's just one condition, buried in the contract:

The consultant has amnesia.

Not total amnesia — a very specific kind. Everything they read, they read before a certain date, and they've learned nothing since. They've never seen a single file inside your company. They can't remember your previous conversations. And here's the strangest symptom: when they don't know something, they don't feel the gap. Their brain fills it in with something plausible, and to them, the invented answer feels exactly like a remembered one.

That's a large language model. That's ChatGPT, Claude, Gemini — all of them. The amnesia isn't a bug someone forgot to fix. It comes from how these systems are built.

Why the amnesia is built in

A language model is trained in one enormous, expensive event. The lab gathers a mountain of text, and over weeks of computation, the model absorbs the patterns in it — how ideas connect, how arguments flow, what a refund policy tends to say. Then training stops, and the model is frozen. Shipped. Done.

Three consequences fall straight out of that, and together they explain almost every disappointing AI interaction you've ever had.

1. Its knowledge has an expiry date. The model knows the world as it looked when training ended — the "knowledge cutoff" you may have seen mentioned. Anything after that simply doesn't exist for it. Your industry shifted last quarter? The model is answering from last year.

2. It has never seen your stuff. Your policies, your contracts, your product docs, your internal wiki — none of it was in the training mountain, because it's private. The genius read the whole library but never once stepped inside your building. This is why "AI for your business" is harder than it looks: the intelligence is general, but your questions are specific.

3. It doesn't know what it doesn't know. This is the one that actually causes damage. A language model works by predicting what text should come next. When it knows the answer, prediction produces the truth. When it doesn't, prediction produces something that sounds like the truth — same fluent tone, same confidence, zero flashing warning light. People call these hallucinations. I think that word makes it sound rarer than it is. It's not a glitch in the machinery; it's the machinery, running exactly as designed, on a question it has no material for.

A US lawyer famously learned this the hard way by submitting a legal brief containing court cases ChatGPT had invented — complete with convincing case numbers and quotes. The judge checked. The cases didn't exist. The AI hadn't lied, exactly. It had done what it always does: produce the most plausible-sounding text. Plausible and true are only sometimes the same thing.

"Can't we just make it smarter?"

The instinctive fix: wait for the next model. Surely intelligence solves this?

It doesn't — because this isn't an intelligence problem. Make the amnesiac consultant twice as brilliant and you get more eloquent guesses about your refund policy. Brilliance was never the missing piece. Access was.

Think about how a knowledgeable human answers a question at work. Almost never purely from memory. They say the five most useful words in professional life: "Let me look that up." They open the actual policy document, find the relevant paragraph, and answer based on what's in front of them. If the document doesn't cover it, they say so.

Notice what happened there. The human's raw intelligence didn't change mid-conversation. What changed the quality of the answer was a simple behavior: check the source before you speak.

So here's the reframe this whole series rests on: the problem with AI at work isn't a lack of intelligence. It's an exam-technique problem. We've been forcing a genius to take every exam closed-book — answer from memory, no notes, no looking things up — on subjects it never studied. And then acting surprised when it improvises.

The fix is exactly what it would be for a person: let it take the exam open-book. Give it your documents. Teach it to find the right page before answering. Let it cite where the answer came from — and admit when the book doesn't say.

That's the whole idea behind RAG — Retrieval-Augmented Generation. Three intimidating words for "look it up first." When you ask a question, a RAG system retrieves the relevant pieces from your actual documents, hands them to the model, and only then lets it generate an answer — grounded in what it just read, not what it vaguely remembers.

You've already seen it in action, probably today. AI search engines that answer with little citation links. Support chatbots that quote the actual help-center article. Coding assistants that reference your codebase instead of generic tutorials. Every one is the open-book pattern: retrieve first, answer second.

The catch (because there's always a catch)

I'll be straight with you, because this series will always be: RAG is not a magic honesty switch. Building the open-book exam well turns out to be its own craft. How do you find the right page among a million pages when the question and the document share no words in common? How do you cut documents into pieces without beheading sentences? What happens when the book itself is out of date?

Get those wrong and you can build a system that fails in new and creative ways — confidently citing the wrong page instead of confidently inventing one. (I run AI R&D at a software company; I have watched a demo bot cite a pricing document that had been dead for two years. The citation looked great.)

That craft — how the lookup actually works, where it breaks, and how to get it right — is what the rest of this series unpacks, one plain-English idea at a time. No code required. Just ideas, explained the way I wish someone had explained them to me.

The one line to remember

If you take a single sentence from today, take this:

Your AI isn't unreliable because it's not smart enough. It's unreliable because it's answering from memory — and the fix is to hand it the book.

Next time a chatbot confidently tells you something wrong about your own business, you'll know exactly what's happening: a genius with amnesia, improvising. And you'll know what it's missing isn't brainpower. It's a library card.

Next in the series — Part 2: "The Open-Book Exam: What RAG Actually Is." We open the hood on those three steps — store, find, answer — and by the end, you'll be able to explain RAG to anyone in five minutes flat.