Last year, Instagram introduced an enhanced comment filter that uses machine learning to spot offensive words and phrases in challenging contexts. Now, the company is expanding similar coverage to photos and captions. Today, it announced that it will use AI to “proactively detect bullying” before sending content to human moderators for review.
The new feature will roll out to users in the coming weeks, launching in time for October’s National Bullying Prevention Month in the US and just before Anti-Bullying Week in the UK. The same technology is also being added to live videos to filter comments there as well.
This is the first product announcement under new Instagram chief Adam Mosseri who took over following the hasty departure of co-founders Kevin Systrom and Mike Krieger last month. The split was reportedly due to simmering tensions between the pair and parent company Facebook, which has frequently meddled with Instagram’s product.
With public trust in Facebook continuing to fall, Instagram remains the bright spot in the company’s product lineup. It’s popular, profitable, and it has yet to be tainted by the scandals that have undermined Facebook. In this context, using AI to help weed out offensive content and keep Instagram a home for good vibes is extremely important.
A story published in Wired last year explained some of the details of Instagram’s machine learning comment filters, but it’s well-established that this sort of technology is no silver bullet for content moderation. AI is cheap to deploy at scale, yes, but it still has trouble dealing with human context and nuance. That’s why it’s good that these new bullying filters also send content to human moderators to perform the final check. Automation without oversight is a recipe for disaster.
It’s unclear whether this new filter analyzes only written content in captions to identify bullying or if it’s also using machine vision to study the photos themselves. The latter would be a much tricker (and more ambitious) task. We’ve reached out to Instagram about this and will update this story if we hear more.
Along with the new filters, Instagram is also launching a “kindness camera effect,” which sounds like it’s a way to spread a positive message as a method to boost user engagement. While using the rear camera, the effects fill the screen with an overlay of “kind comments in many languages.” Switch to your front-facing camera, and you get a shimmer of hearts and a polite encouragement to “tag a friend you want to support.”