Rag

Large language models have a habit of making things up, and the standard fix introduces its own problems.

The fix is called retrieval-augmented generation, and the idea is simple and reasonable. Instead of letting the model answer from whatever it happened to absorb during training, you hand it the relevant documents at the moment of the question, and you ask it to answer based on them. In education, this means you can point the system at a particular textbook, a particular course, a particular set of approved materials, and have it generate explanations drawn from there. The hallucination problem shrinks, not because the model has changed but because you have given it a smaller and more specific haystack to look in. For many uses this is a genuine improvement, and it is the reason most serious educational applications of AI now involve some form of retrieval.

The trouble is that it introduces new problems. The model might retrieve the right passage and still misinterpret it. It might pick the wrong passage, plausibly, from a set of documents that did not quite match the question. It might quietly blend information from different sources in a way that produces something no single source actually said. And when it gets things wrong, the error is dressed in the authority of the original material, which makes the mistake harder to catch than a pure hallucination would be. A retrieval-augmented system is not a system that tells the truth. It is a system that tells you things grounded in its sources, which is better, and not the same. When you last used an AI tool that claimed to be grounded in reliable documents, how often did you actually check whether it quoted them correctly?

Last week's Act opens by saying something has shifted in the last few years that the industry is still pretending has not. This week we look at the most common fix for AI making things up, and at the new problems it brings.