Model Cards

Every AI system has a quiet set of assumptions built into it, and you almost never get to see them.

A model was trained on something. It was tested against something. It is appropriate for some uses and not others, and it performs differently on different groups of people, and it has known weak spots where its confidence is unreliable. All of these facts exist, somewhere in somebody's notebook, on the team that built the thing. When the system is deployed into the world and starts making decisions about students, patients, job applicants, or loan applications, none of these facts travel with it. The user sees a confident output and has no way of knowing that it was trained on data from ten years ago, or that it has never been validated on the group of people it is now being used on, or that its accuracy drops sharply in a particular corner of its input space.

A pair of researchers named Margaret Mitchell and Timnit Gebru proposed a simple idea in 2019. Every model should ship with a one-page card describing what it was trained on, what it was tested on, what it is for, what it is not for, and how it performs on different populations. The idea is modest and the benefits are large. It does not change how the model works. It just forces the people who built it to state, plainly, what they already know about its limits. A few organisations have adopted this. Most have not, because it is voluntary, and voluntary disclosure in a competitive market is a hard habit to grow. When you next encounter a product that promises to decide something important about you, how useful would a plain one-page honest description of its limits actually be?

Last week we said fairness is not one thing. This week we look at the simple idea of attaching a plain page of honest description to every model, and at why almost nobody does it.