Why RAG must be evaluated in parts

"The RAG system is bad."

Fine, which part? A single end-to-end quality score is a diagnosis-free diagnostic. It tells you something is wrong and nothing about where, so teams default to fixing the layer they can see, which is the prompt. Weeks of prompt iteration later, retrieval was broken the whole time.

It fails in parts, so measure it in parts

A RAG pipeline is at least four systems wearing one interface, and they fail independently.

Diagnose in order, because failures masquerade

This is the part that saves the most time. Upstream failures disguise themselves as downstream ones.

If the right passage was never retrieved, everything after is irrelevant, and the symptom will look exactly like hallucination, because the model had nothing correct to work from and produced something plausible instead. A large share of "the model is making things up" tickets resolve to "recall@20 was 0.4 and nobody was measuring it."

So work upstream to downstream. Was it in the index? Was it retrieved? Was it ranked highly enough to be read? Did the answer use it? Only when all four pass is a generation problem actually a generation problem.

The minimum viable setup

Fifty labelled questions. For each: the question, the passage that should answer it, and what a good answer contains.

That is it. Fifty questions, per-layer metrics, run on every meaningful change (chunking strategy, embedding model, retrieval config, prompt). It takes an afternoon to build and it converts every subsequent argument into a measurement.

The specific thing it buys you: re-chunking stops being a gamble. Change the strategy, run the set, read the diff. Without it, every structural change to your pipeline is a leap of faith you find out about from users.

Closing thought

You cannot fix what you cannot locate. Component metrics tell you where the fault is. The end-to-end score only tells you that there is one.