Rethinking Multimodal Entity and Relation Extraction from a Translation Point of View

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6810–6824 July 9-14, 2023 ©2023 Association for Computational Linguistics Rethinking Multimodal Entity and Relation Extraction from a Translation Point of View Changmeng Zheng1, Junhao Feng3, Yi Cai3, Xiao-Yong Wei2,1∗, Qing Li1 1Department of Computing, The Hong Kong Polytechnic University, Hong Kong SAR, China 2Department of Computer Science, Sichuan University, China 3School of Software Engineering, South China University of Technology, China csczheng@comp.polyu.edu.hk, cswei@scu.edu.cn Abstract We revisit the multimodal entity and relation extraction from a translation point of view. Spe- cial attention is paid on the misalignment issue in text-image datasets which may mislead the learning. We are motivated by the fact that the cross-modal misalignment is a similar prob- lem of cross-lingual divergence issue in ma- chine translation. The problem can then be transformed and existing solutions can be bor- rowed by treating a text and its paired image as the translation to each other. We implement a multimodal back-translation using diffusion- based generative models for pseudo-paralleled pairs and a divergence estimator by construct- ing a high-resource corpora as a bridge for low-resource learners. Fine-grained confidence scores are generated to indicate both types and degrees of alignments with which better repre- sentations are obtained. The method has been validated in the experiments by outperforming 14 state-of-the-art methods in both entity and re- lation extraction tasks. The source code is avail- able at https://github.com/thecharm/TMR. 1 Introduction Multimodal language understanding has received intensive attention recently for its advantage of mining semantics by collaborating the cross-modal inference (Yang et al., 2019a). Examples include methods for multimodal name entity recognition (MNER) (Zhang et al., 2018) and multimodal rela- tion extraction (MRE) (Zheng et al., 2021a). Both benefit from the collaborative reasoning based on the alignment of textual and visual content. How- ever, statistics on commonly adopted text-image relation benchmarks (e.g., TRC (Vempala and Preo¸tiuc-Pietro, 2019) and Twitter100k (Hu et al., 2017)) shows that the misalignment rate between images and texts is as high as 60%. Noise intro- duced by the misalignment can mislead the learning and degrade the performance of resulting models. ∗Corresponding author A pair (𝒕𝟐,𝒗𝟐) of Irrelevant Alignment A pair (𝒕𝟏,𝒗𝟏) of Partial Alignment Taylor Swift sets new AMAs record, urges people to vote via @ReutersTV. Entities: Taylor Swift (MISC) Predictions w/o knowing misalignment Entities: Taylor Swift (PER) “Fair trade now fool trade,” Trump vents anger on NATO allies. Entities: Trump (PER), NATO (PER) Relations: <Trump, peer, NATO> Entities: Trump (PER), NATO (ORG) Relations: <Trump, member_of, NATO> 𝒑(𝒕𝟏,𝒗𝟏) 𝒑(𝒕𝟐,𝒗𝟐) 𝒑(𝒗𝟏 " |𝒕𝟏) 𝒑(𝒗𝟐 " |𝒕𝟐) Predictions with the back-translation as the reference and aware of misalignments Back-translation Back-translation 𝒑(𝒕𝟏,𝒗𝟏,𝒗𝟏 " ) 𝒑(𝒕𝟐,𝒗𝟐,𝒗𝟐 " ) Figure 1: Partial (left) and irrelevant (right) alignments in text-image pairs and the results of using generative back-translation to help the inference in multimodal entity and relation extraction tasks. As shown in Fig. 1, the misalignment can be categorized into partial and irrelevant alignment. In case of incomplete alignment, textual entities (e.g., …

Content truncated to 500 words