Psychometric Validation

An AI that writes quiz questions for you is doing half of what a proper assessment designer does.

When a modern language model generates a quiz question in seconds, the thing it produces usually looks fine. The grammar is correct. The options are plausible. The structure matches what a quiz question should look like. If you read a dozen of these and were not paying close attention, you would almost certainly assume they had come from a human. The machine is good at producing items that look like assessment items. What it is not automatically doing, and what almost nobody checks, is the second part, which is whether those items actually measure what they claim to measure, reliably, fairly, across different learners. That part is called psychometric validation, and it is slow, expensive, and mostly invisible to the people using the tests.

A proper assessment goes through a long process after it is written. It is tried on real learners. Statistics are run to check whether the right people get it right and the wrong people get it wrong. It is examined for bias against particular groups. It is refined, sometimes discarded, sometimes kept. Only the items that pass this gauntlet end up in serious use. AI-generated questions skip this entirely by default. They look like assessments without actually being assessments in the technical sense, and when they are used for real decisions, the test measures something, but nobody quite knows what. The tool is brilliant for generating drafts. Treating the drafts as finished tests, without the quiet work of validation, is how AI turns assessment from a science back into guesswork. When a test result you received did not match how you felt about your performance, how much of that was you and how much was the test?

Last week we said good tests start from what you want to claim about the learner. This week we look at what happens when an AI writes the test for you.