A Relation Extraction Dataset for Knowledge Extraction from Web Tables

Proceedings of the 29th International Conference on Computational Linguistics, pages 2319–2327 October 12–17, 2022. 2319 REDTab: A Relation Extraction Dataset for Knowledge Extraction from Web Tables Siffi Singh 1, Alham Fikri Aji 2, Gaurav Singh 3,∗, and Christos Christodoulopoulos 2 1Amazon AWS AI, 2Amazon Alexa AI {siffis,afaji,chrchrs}@amazon.com 3Binance gaurav.singh@binance.com Abstract Relational web-tables are significant sources of structural information that are widely used for relation extraction and population of facts into knowledge graphs. To transform the web- table data into knowledge, we need to iden- tify the relations that exist between column pairs. Currently, there are only a handful of publicly available datasets with relations annotated against natural web-tables. Most datasets are constructed using synthetic tables that lack valuable metadata information, or are limited in size to be considered as a challeng- ing evaluation set. In this paper, we present REDTab, the largest natural-table relation ex- traction dataset. We have annotated ~9K tables and ~22K column pairs using crowd sourced annotators from MTurk, which has 50x larger number of column pairs than the existing human-annotated benchmark. Our test set is specially designed to be challeng- ing as observed in our experiment results using TaBERT. 1 Introduction Web-tables contain a lot of knowledge (Cafarella et al., 2008; Wang and Hu, 2002) that can be uti- lized for various tasks such as question answering (Chakrabarti et al., 2020; Zhang et al., 2020), build- ing knowledge graphs (Sekhavat et al., 2014; Wang et al., 2012), web-search (Sun et al., 2016; Kopliku et al., 2011), data-mining (Gatterbauer et al., 2007) and so on. Relation Extraction (RE) is one way to extract table information by capturing relations be- tween two table columns (Figure 1). However, the current RE datasets available are either synthetic, or very small in size. In this paper, we propose a new human-annotated dataset for the evaluation of RE models. ∗Work done while at Amazon The existing benchmark datasets for column pair RE are T2Dv2 and SemTab. However, T2Dv2 (Ritze and Bizer, 2017) only contains 236 tables and 435 column pairs annotated with 118 relations where about 50% of relations have only 1 column pair sample. The highest number of sam- ples for a relation is only 28. Hence, due to its small size and less diversity in column pairs per relation, it is difficult to be treated as a reliable eval- uation set. In contrast, the SemTab 2020 challenge (Jiménez-Ruiz et al., 2020a) provides a large bench- mark dataset, but synthetically generated, which is arguably not representative of the real web-tables. Other table-based datasets (Limaye et al., 2010; Efthymiou et al., 2017; Zhang, 2017; Kacprzak et al., 2018) are designed for a binary relation ex- traction task. REDTab1 is a human annotated dataset consist- ing of 9,149 tables and 22,236 column pairs with 23 relations. The table was collected from Web Data Commons (WDC) (Lehmberg et al., 2016) relational tables for Music and Literature domains. These domains are two of most popular domains for question answering (Rajpurkar et al., 2016; Mihaylov et …

Content truncated to 500 words