Knowledge Tracing

When a student gets an answer right, we do not actually know what happened.

Did they know it, solidly, because they had truly understood the material and could reproduce the answer on any given day. Did they guess, and happen to guess correctly. Did they remember it shakily from the last review session without really understanding why the answer was what it was. Did they have a half-formed intuition that happened to point in the right direction this time and would mislead them on a slightly different problem. From the outside, all four look the same. One tick on a screen. The student moves on, the system notes a correct answer, and a statistic somewhere gets incremented by one. Nobody, not the student, not the system, not the teacher reading a report later, can tell these cases apart.

Knowledge tracing is the statistical attempt to solve this problem. You model each student as probably having or probably not having each skill, and you keep updating that probability as they answer more questions. The idea goes back to work by Corbett and Anderson in the 1990s, and it has become more elaborate ever since. It is useful, because it is better than pretending one correct answer equals one mastered skill. It is also fragile, because the whole apparatus rests on assumptions about how learning works that are only approximately true. A student might appear mastered because they happened to encounter their strong topics first, or appear weak because they were tired for three sessions in a row. The probability is a fiction that is closer to the truth than the one-tick view. A good teacher still sees more. When you think about something you are sure you know well, how would you prove it to somebody who wanted to be genuinely convinced?

Last week we asked what a real tutor actually does. This week we look at what your screen sees and does not see when you get an answer right.