Why RAG quality drops without anyone noticing

A RAG system almost never fails with an error. It fails by answering a little worse than it did last week. Someone changes the chunk size, swaps an embedding model, or edits a prompt to fix one complaint, and every spot check still looks fine. Nothing throws. Nothing alerts.

That is the trap. Spot checks only cover the questions you happen to think of, and the questions you think of are the ones the change was meant to fix. The answers that got worse are somewhere else.

The fix is boring and it works: keep a golden set of questions with known-good answers, and run every version of the system against it before it ships. Treat it like a test suite, because it is one. If the score drops, the change does not go out.

A score alone is not enough, though. "Quality fell four points" tells you something is wrong but not where to look. So each failure gets a cause: the right passage was never retrieved, the passage was retrieved but the model ignored it, or the model answered from nothing at all. Those three need three different fixes, and guessing between them is how a team loses a week.

That is the idea behind RAGWatch. It is less a clever model than a habit made automatic: compare this version to the last one, on the same questions, and say plainly what broke.