Explainable Ai

When a machine tells you why it made a decision, it is often telling you a story about a different machine.

This is one of the stranger facts about the field called explainable AI, and most people who use these systems have no idea. The problem researchers were trying to solve is real. Complicated models make decisions that affect people's lives, and the people affected deserve to know why. So tools were built to produce explanations, little highlighted regions, little lists of what features mattered most, little graphs showing how the decision would change if one input changed. They look authoritative, and they give the person on the receiving end a sense that the black box has been opened. In many cases, it has not been. What has actually happened is that a second, simpler model has been trained to approximate what the first one was doing, and the explanation is an explanation of the simpler one. Close enough, sometimes. Not the same thing, often.

The difference matters when the decision matters. If a system explains that it denied you a loan because of your income and your history, and the real model was also using something it picked up indirectly about your postcode, your explanation is clean and your denial is not. A researcher named Cynthia Rudin has argued for years that the answer is not better explanations of opaque systems but systems that were interpretable from the start. Her argument is inconvenient because interpretable models sometimes perform slightly worse on benchmarks, so most of the industry keeps choosing the opaque ones and dressing them in explanations that are close enough. When a decision is being made about you, how much does it matter to you that the explanation is the real one?

Last week we said opening the box does not always produce an explanation. This week we look at what is really happening when an AI tells you why it decided something.