Mapping Memes to Words for Multimodal Hateful Meme Classification

Multimodal image-text memes are prevalent on the inter- net, serving as a unique form of communication that com- bines visual and textual elements to convey humor, ideas, or emotions. However, some memes take a malicious turn, promoting hateful content and perpetuating discrimination. Detecting hateful memes within this multimodal context is a challenging task that requires understanding the inter- twined meaning of text and images. In this work, we address this issue by proposing a novel approach named ISSUES for multimodal hateful meme classification. ISSUES lever- ages a pre-trained CLIP vision-language model and the textual inversion technique to effectively capture the mul- timodal semantic content of the memes. The experiments show that our method achieves state-of-the-art results on the Hateful Memes Challenge and HarMeme datasets. The code and the pre-trained models are publicly avail