Different Data, Different Modalities! Reinforced Data Splitting for Effective Multimodal Information Extraction from Social Media Posts

Proceedings of the 29th International Conference on Computational Linguistics, pages 1855–1864 October 12–17, 2022. 1855 Different Data, Different Modalities! Reinforced Data Splitting for Effective Multimodal Information Extraction from Social Media Posts Bo Xu1, Shizhou Huang1, Ming Du1,∗, Hongya Wang1, Hui Song1, Chaofeng Sha2 and Yanghua Xiao2 1School of Computer Science and Technology, Donghua University 2Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University xubo@dhu.edu.cn, 2202408@mail.dhu.edu.cn, {duming, hywang, songhui}@dhu.edu.cn {cfsha, shawyh}@fudan.edu.cn Abstract Recently, multimodal information extraction from social media posts has gained increas- ing attention in the natural language processing community. Despite their success, current ap- proaches overestimate the significance of im- ages. In this paper, we argue that different social media posts should consider different modalities for multimodal information extrac- tion. Multimodal models cannot always outper- form unimodal models. Some posts are more suitable for the multimodal model, while oth- ers are more suitable for the unimodal model. Therefore, we propose a general data splitting strategy to divide the social media posts into two sets so that these two sets can achieve bet- ter performance under the information extrac- tion models of the corresponding modalities. Specifically, for an information extraction task, we first propose a data discriminator that di- vides social media posts into a multimodal and a unimodal set. Then we feed these sets into the corresponding models. Finally, we combine the results of these two models to obtain the final extraction results. Due to the lack of explicit knowledge, we use reinforcement learning to train the data discriminator. Experiments on two different multimodal information extrac- tion tasks demonstrate the effectiveness of our method. The source code of this paper can be found in https://github.com/xubodhu/RDS. 1 Introduction Social media, with its wealth of user-generated posts, provides a rich platform for understanding events, opinions and preferences of groups and indi- viduals (Moon et al., 2018). Information extraction, such as named entity recognition (Yu et al., 2020), ∗ Corresponding author. This paper was supported by the National Natural Science Foundation of China (No. 61906035), Information Development Project of Shanghai Economic and Information Commission (No. 202002009), Natural Science Foundation of Shanghai (No. 22ZR1402000) and Shanghai Science and Technology Innovation Action Plan (No. 19511120400) relation extraction (Zheng et al., 2021a) and senti- ment detection (Yang et al., 2021), is a critical step in uncovering these hidden insights in social media posts. In social media scenarios, information is ex- pressed not only through textual modality, but through multiple modalities (e.g., text, image, etc.). Considering only text modality may lead to in- accurate information extraction. For example in Fig. 1(a) and Fig. 1(c), the text-based named entity recognition model cannot recognize Kolo as a dog, and the text-based relation extraction model cannot determine that Meghan Markle and Prince Harry are couples. Therefore, many multimodal models for infor- mation extraction have been proposed which are using visual modality to complement text modality. They mainly focus on using a multimodal inter- action mechanism to combine text representation with image representation. For example, Zhang et …

Content truncated to 500 words