Ethical Proportionality

Not every decision an algorithm makes deserves the same amount of scrutiny.

A recommendation engine that suggests what film you might enjoy tonight is doing something very different from a system that decides whether a child should be flagged for special intervention at school, even if both are technically running on similar machine learning. If something goes slightly wrong with the film recommendation, you watch something you do not enjoy and move on. If something goes slightly wrong with the child flagging, a real family gets contacted, a teacher starts treating a student differently, a record follows that child through years of their schooling. The stakes are not comparable, and yet many of the AI ethics frameworks we have built apply exactly the same governance to both. This sounds rigorous in theory, but in practice it either makes the low stakes systems impossible to ship or makes the high stakes systems barely better checked than the film recommender.

The fix is not more rules or fewer rules. It is proportionality. The depth of scrutiny should match the depth of consequence. A lightweight decision with reversible effects can be governed lightly, so that useful things actually get built. A heavy decision with hard to reverse effects needs intensive review, human oversight, and the ability to escalate when something feels off. And any decision in a situation nobody anticipated, regardless of its supposed stakes, should quietly pause itself and ask for a human, because the one thing an algorithm cannot do is notice that it is out of its depth. Where in your own life would you want a machine to stop and ask for help rather than just doing its best?

Last week we said every product arrives with someone's values inside. This week we look at the difference between a small decision and a serious one, and at how rules should change between them.