Mastodon Feed
Boosted by glyph ("Glyph"):
0xabad1dea@infosec.exchange ("abadidea") wrote:
wow, discord acknowledged on the Bad Place that it was true that an overfitted CSAM detection AI started flagging grids (spreadsheets, game inventory screenshots, etc) as illegal and instabanning people.
8200 people got banned this way over several weeks before someone managed to get their attention on reddit. But the good news is they have in fact manually rolled back all the bans.
On the one hand, this is the one problem where you absolutely want as little hands-on human moderation as possible. On the other, this is a very clear-cut overfitting failure mode on a small dataset (of the images that governments provide to major websites for automated detection) with disastrous consequences for random innocent people. And they presumably had to go back and manually check all bans in this category for the last few weeks, thus maximizing the moderators' exposure too. 🤦♀️