Replication Governance

Most educational technology gets sold on a single impressive study, and single impressive studies are unreliable.

A vendor runs a pilot, or commissions one, or partners with a university, and a few months later there is a study showing their product produced a meaningful improvement in some outcome. The study goes into the sales deck. Schools buy the product on the strength of it. Years later, a handful of other researchers try to replicate the effect in different contexts, and the results are mixed, or the effect shrinks, or it disappears altogether. By then the product has been purchased by thousands of institutions, installed, integrated, and made part of how they teach, and nobody is in a hurry to rip it out. The mismatch between the breathless promise and the steadier reality is a pattern that repeats across the entire industry.

The way out is not to be cynical about research. It is to change what we accept as evidence. A single study, however well designed, is a report on what happened with one group of people in one setting. Genuine confidence requires replication in different settings, with different learners, by different research teams, ideally including some that are independent of the vendor. Replication governance is the discipline of treating a single study as the start of evidence rather than the end. Almost nobody in education technology procurement actually works this way, because it is slow and inconvenient and would require rejecting a lot of exciting product pitches. The alternative is continuing to buy things because the first study looked good. When a decision at your organisation was made on the strength of one piece of impressive-looking evidence, how did it turn out a few years later?

Last week we asked what makes a small study big enough. This week we look at why one impressive study is almost never enough, and at the slow business of finding out whether something really works.