The Ruler Problem: What \"Near\" Actually Means

A team I know shipped a support-search feature with one safety rule: only show results with a similarity score above 0.8. Sensible-sounding. Clean. The kind of threshold that goes in a config file and gets forgotten.

In production, the system alternated between two failure modes. For some questions it returned nothing — the right document was sitting at 0.74. For others it confidently returned junk — because on that query, everything scored above 0.8, including the cafeteria menu.

The bug wasn't the number. The bug was believing the number meant something by itself. Parts 2 and 3 gave every thought an address on a self-drawn map. Today: the ruler — how "near" actually gets measured, and why similarity scores behave less like grades and more like distances between towns.

Direction beats distance

First, a genuinely useful subtlety, in plain English.

Picture two movie reviews. One is a single sentence: "Loved it, gorgeous film." The other is three paragraphs of the same opinion — same warmth, more words. On the map of meaning, more text pushes an address further out along its theme; the long review sits further from the center than the short one. Measure the straight-line gap between them and they look moderately far apart — even though they're saying the same thing.

But look from the map's origin, and both reviews lie in exactly the same direction — the "loved this film" direction. One is just further down the road.

That's why the standard similarity measure in AI compares direction, not straight-line distance. Two texts pointing the same way are similar, whether one is a headline and the other an essay. (You've heard its name — cosine similarity — and that's the entire idea: same heading, high score. The trigonometry is somebody else's problem.) It's the reason your three-word query can match a three-paragraph answer: they point the same way.

The Ruler Problem

Now the trap from the opening story. A similarity comparison outputs a score — often something like 0.83 — and every human instinct says to read it like a grade. Above 0.8, excellent. Below 0.5, poor.

The instinct is wrong, and here's the plainest way I know to see it: the ruler stretches depending on the question. Ask something crisp and well-represented — "refund policy for sale items" — and genuinely relevant passages might score 0.85 while irrelevant ones sit at 0.4. Yawning gap, easy call. Ask something vague — "problem with my thing" — and the map, doing its honest best with a vague address, scores everything in a mushy 0.55-to-0.65 band. Ask about jargon the mapmaker barely saw (Part 3's blind spot) and even the perfect document may top out at 0.7.

Same ruler, wildly different readings — because the score isn't a quality grade. It's a relative position on this particular query's landscape. I call this the Ruler Problem, and the fix is a discipline, not a number: read scores against their neighbors, never against a universal bar. The question that matters is not "did it score 0.8?" but "does the top result stand clearly apart from the pack?" A 0.72 towering over a field of 0.45s is a strong signal. A 0.84 in a crowd of 0.83s means the map is shrugging.

Retrieval systems that work embrace this: take the top few by rank, watch the gaps between scores, and treat a flat, gapless score profile as what it truly is — the system saying "I'm not sure," in the only language it has.

Rank is the honest output

This reframe earns you one very practical instinct: absolute scores are for machines; rankings are for decisions. When the Open-Book AI series walked through retrieval, the funnel never asked "which chunks scored above X?" It asked "which fifty rank highest — and then which five survive a careful second read?" Ranking sidesteps the stretching ruler entirely: whatever today's landscape looks like, the best-available candidates rise, and a downstream reader (the reranker — the parent series' "Second Interview") judges whether best-available is actually good.

That division of labor is the mature design: the map proposes by rank; a reader disposes by reading. The score's job is to sort, not to certify.

The catch

None of this makes the measurement smart. Direction-comparison is arithmetic on addresses — it inherits every flaw the addresses have. If the map placed "cannot be refunded" beside "can be refunded" (it did), the ruler will dutifully report them as nearly identical, at any threshold, under any reading discipline. A great measuring stick on a flawed map measures the flaws with precision.

Which sharpens a question that's been building for three articles: whose map are you measuring on? The internet's general-purpose map is spectacular for general language. Your business does not speak general language.

The one line to remember

Similarity scores aren't grades — they're distances on a landscape that changes with every question. Read them against their neighbors, never against a bar.

Next — Part 5: "The Local Map." When the internet's geography fails your company's jargon — and the three honest options: pick a specialist mapmaker, redraw the map, or give your text better company.