Ex Ante, Ex Post

We evaluate almost everything after it is built, and then we are surprised when it does not work.

Think about the last big thing your organisation launched. A new system, a new policy, a new platform, a new way of doing something. The evaluation probably came months later, once it had already been rolled out, once people were already using it, once changing direction would cost enormous amounts of time and money. By then the evaluation was a report card rather than a steering wheel, because the decisions that shaped the outcome had been made long before anyone thought to check whether they made sense. The review said what happened. It could not change what was going to happen next, because that had already been set in motion.

There is a distinction in design science between evaluating something before it is fully built, based on whether the design logic is sound, and evaluating it after, based on whether the outcomes matched the hopes. Both are necessary, but only the first has any chance of catching the really expensive mistakes while they can still be fixed. We are good at the second and almost hopeless at the first, because the first requires admitting that you might be wrong while there is still time to change course. The second lets you mostly defend what you already did. The trade is comfort now for expense later, and most institutions keep making it. In the last thing you worked on that did not go well, when did you first suspect it was not going to, and what stopped you from saying so?

Last week we looked at the gap between research and practice. This week we look at the difference between checking your work before you start and judging it only after it is too late to change.