REDFM: a Filtered and Multilingual Relation Extraction Dataset

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4326–4343 July 9-14, 2023 ©2023 Association for Computational Linguistics REDFM: a Filtered and Multilingual Relation Extraction Dataset Pere-Llu´ıs Huguet Cabot Babelscape, Italy & Sapienza University of Rome huguetcabot@babelscape.com Simone Tedeschi Babelscape, Italy & Sapienza University of Rome tedeschi@babelscape.com Axel-Cyrille Ngonga Ngomo Paderborn University axel.ngonga@upb.de Roberto Navigli Sapienza University of Rome navigli@diag.uniroma1.it Abstract Relation Extraction (RE) is a task that iden- tifies relationships between entities in a text, enabling the acquisition of relational facts and bridging the gap between natural language and structured knowledge. However, current RE models often rely on small datasets with low coverage of relation types, particularly when working with languages other than English. In this paper, we address the above issue and provide two new resources that enable the train- ing and evaluation of multilingual RE systems. First, we present SREDFM, an automatically an- notated dataset covering 18 languages, 400 re- lation types, 13 entity types, totaling more than 40 million triplet instances. Second, we pro- pose REDFM, a smaller, human-revised dataset for seven languages that allows for the evalu- ation of multilingual RE systems. To demon- strate the utility of these novel datasets, we ex- periment with the first end-to-end multilingual RE model, mREBEL, that extracts triplets, in- cluding entity types, in multiple languages. We release our resources and model checkpoints at https://www.github.com/babelscape/rebel. 1 Introduction The vast majority of online and offline content con- sists of raw, natural language text containing fac- tual information. Current Large Language Mod- els (LLMs) are pretrained on such text, allowing reasoning over it through tasks such as Question Answering (Bouziane et al., 2015) or Text Summa- rization (El-Kassas et al., 2021). On the other hand, structured resources such as Knowledge Graphs en- able knowledge-based, explainable, machine-ready reasoning over their content. Both approaches are important and are widely used within Natural Lan- guage Processing systems, with recent trends look- ing at combining them (Yamada et al., 2020; Sun et al., 2021). Information Extraction tackles the need for sys- tems that extract structured information from raw text. Specifically, end-to-end Relation Extraction extracts the relational information between enti- ties in a given text, providing a structured pre- diction. However, although some highly capable systems have been released (Wang and Lu, 2020; Paolini et al., 2021; Huguet Cabot and Navigli, 2021), there are few high-quality, contemporary resources. Current RE datasets are outdated, be- hind paywalls, have design flaws, or only consider English. While multilingual datasets exist, such as ACE051 or SMiLER (Seganti et al., 2021), the former covers only six relation types, and requires a paid license for its use. The latter is more recent, bigger, and has a higher coverage of relation types, but it does not contain human-annotated samples that permit reliable evaluation and is not conducive to train End-to-End Relation Extraction systems. Instead, the availability of large high-quality re- sources is fundamental in order to allow LLMs to be trained and evaluated on trustworthy …

Content truncated to 500 words