Facebook AI has announced the release of a new challenge and dataset called “Hateful Memes”. In their blog post, researchers from Facebook describe the new challenge as another one in the series of challenges, coming after the Deepfakes Detection Challenge, whose goal is again, to engage the community and catalyze research in a novel area.
According to them, progress in AI and AI solutions to many problems can come from open collaboration, which is part of the reason behind the release of the new dataset and challenge. As researchers mention, the Hateful Memes Challenge was created to make AI a more effective tool for detecting hate speech and to bring AI models closer to the way humans understand multimodal content.
“In order for AI to become a more effective tool for detecting hate speech, it must be able to understand content the way people do: holistically. When viewing a meme, for example, we don’t think about the words and photo independently of each other; we understand the combined meaning together. This is extremely challenging for machines, however, because it means they can’t just analyze the text and the image separately.”
The dataset contains more than 10 000 new examples created by researchers at Facebook which represent images accompanied by a short text as a meme. The creators of the Hateful Memes dataset paid special attention to making all the samples in the dataset in a way that unimodal models would struggle to classify them correctly.
Together with the dataset, Facebook AI announced the new challenge, a first-of-its-kind, on detecting hateful memes. It is hosted on the DrivenData platform with a total prize pool of 100 000$. The competition is designed as a binary classification problem, given a dataset of memes – an image plus a string of text. Solutions will be evaluated and scored by using the simple AUC ROC metric for binary classification.
