Borrowing Human Senses: Comment-Aware Self-Training for Social Media Multimodal Classification

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5644 - 5656 December 7-11, 2022 ©2022 Association for Computational Linguistics Borrowing Human Senses: Comment-Aware Self-Training for Social Media Multimodal Classification Chunpu Xu, Jing Li∗ Department of Computing, The Hong Kong Polytechnic University, China chun-pu.xu@connect.polyu.hk jing-amelia.li@polyu.edu.hk Abstract Social media is daily creating massive multime- dia content with paired image and text, present- ing the pressing need to automate the vision and language understanding for various mul- timodal classification tasks. Compared to the commonly researched visual-lingual data, so- cial media posts tend to exhibit more implicit image-text relations. To better glue the cross- modal semantics therein, we capture hinting features from user comments, which are re- trieved via jointly leveraging visual and lin- gual similarity. Afterwards, the classification tasks are explored via self-training in a teacher- student framework, motivated by the usually limited labeled data scales in existing bench- marks. Substantial experiments are conducted on four multimodal social media benchmarks for image-text relation classification, sarcasm detection, sentiment classification, and hate speech detection. The results show that our method further advances the performance of previous state-of-the-art models, which do not employ comment modeling or self-training. 1 Introduction Interpersonal communications in multimedia are gaining growing popularity on social media. More and more social media users are turning to pair im- ages to text and vice versa to better voice opinions, exchange information, and share ideas, exhibiting rich and ever-updating resources in multimedia. While potentially benefiting people’s everyday de- cision making, the huge volume of multimedia con- tent might also challenge users in finding what they need. Towards a more efficient and effective way to process the online multimodal data, substantial efforts have been made to automatically understand the vision and language on social media through a broad range of multimodal classification tasks for predicting image-text relations (Vempala and Preotiuc-Pietro, 2019), sarcasm (Cai et al., 2019), ∗ Corresponding author Text: was going to take mollie for a walk in the park today . pennsylvania weather cooperated as per usual Retrieved comments: 1. damn ! that 's a lot of snow to still be around for this time of year , no 2. snow in your area we 'd love to see pictures of it ! 3. heavier snow now shifting to your east … Figure 1: A sample tweet with its image on the left. On the right, the tweet text is shown on the top, followed by the comments retrieved from similar tweets. The word “snow” (in blue) in comments helpfully hint the implicitly shared semantics between image and text. metaphor (Zhang et al., 2021), point-of-interest (Villegas and Aletras, 2021), hate speech (Botelho et al., 2021), sentiment (Yu and Jiang, 2019), etc. Despite the success of visual-lingual understand- ing witnessed in common domains (Huang et al., 2020; Shi et al., 2020; Wang et al., 2021), existing models’ performance is likely to be compromised on social media posts. The possible reason lies in the relatively more implicit and obscure image- text …

Content truncated to 500 words