MultiTACRED: A Multilingual Version of the TAC Relation Extraction Dataset

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3785–3801 July 9-14, 2023 ©2023 Association for Computational Linguistics MultiTACRED: A Multilingual Version of the TAC Relation Extraction Dataset Leonhard Hennig Philippe Thomas Sebastian Möller German Research Center for Artificial Intelligence (DFKI) Speech and Language Technology Lab {leonhard.hennig, philippe.thomas, sebastian.moeller}@dfki.de Abstract Relation extraction (RE) is a fundamental task in information extraction, whose exten- sion to multilingual settings has been hindered by the lack of supervised resources compara- ble in size to large English datasets such as TACRED (Zhang et al., 2017). To address this gap, we introduce the MultiTACRED dataset, covering 12 typologically diverse languages from 9 language families, which is created by machine-translating TACRED instances and automatically projecting their entity annota- tions. We analyze translation and annotation projection quality, identify error categories, and experimentally evaluate fine-tuned pretrained mono- and multilingual language models in common transfer learning scenarios. Our anal- yses show that machine translation is a viable strategy to transfer RE instances, with native speakers judging more than 83% of the trans- lated instances to be linguistically and seman- tically acceptable. We find monolingual RE model performance to be comparable to the English original for many of the target lan- guages, and that multilingual models trained on a combination of English and target lan- guage data can outperform their monolingual counterparts. However, we also observe a va- riety of translation and annotation projection errors, both due to the MT systems and lin- guistic features of the target languages, such as pronoun-dropping, compounding and inflec- tion, that degrade dataset quality and RE model performance. 1 Introduction Relation extraction (RE), defined as the task of identifying and classifying semantic relationships between entities from text (cf. Figure 1), is a funda- mental task in information extraction (Doddington et al., 2004). Extending RE to multilingual set- tings has recently received increased interest (Zou et al., 2018; Nag et al., 2021; Chen et al., 2022c), both to address the urgent need for more inclusive NLP systems that cover more languages than just English (Ruder et al., 2019; Hu et al., 2020), as well as to investigate language-specific phenom- ena and challenges relevant to this task. The main bottleneck for multilingual RE is the lack of super- vised resources, comparable in size to large English datasets (Riedel et al., 2010; Zhang et al., 2017), as annotation for new languages is very costly. Most of the few existing multilingual RE datasets are dis- tantly supervised (Köksal and Özgür, 2020; Seganti et al., 2021; Bhartiya et al., 2022), and hence suffer from noisy labels that may reduce the prediction quality of models (Riedel et al., 2010; Xie et al., 2021). Available fully-supervised datasets are small, and cover either very few domain-specific re- lation types (Arviv et al., 2021; Khaldi et al., 2022), or only a small set of languages (Nag et al., 2021). To address this gap, and to incentivize research on supervised multilingual RE, we introduce a multilingual version …

Content truncated to 500 words