From Dataset Recycling to Multi-Property Extraction and Beyond

Proceedings of the 24th Conference on Computational Natural Language Learning, pages 641–651 Online, November 19-20, 2020. c⃝2020 Association for Computational Linguistics https://doi.org/10.18653/v1/P17 641 From Dataset Recycling to Multi-Property Extraction and Beyond Tomasz Dwojak1,2, Michał Pietruszka1,3, Łukasz Borchmann1,4, Jakub Chł˛edowski1,3, and Filip Grali´nski1,2 1Applica.ai 2Faculty of Mathematics and Computer Science, Adam Mickiewicz University in Poznan 3Faculty of Mathematics and Computer Science, Jagiellonian University 4Institute of Computing Science, Poznan University of Technology tomasz.dwojak@applica.ai Abstract This paper investigates various Transformer architectures on the WikiReading Informa- tion Extraction and Machine Reading Com- prehension dataset. The proposed dual-source model outperforms the current state-of-the- art by a large margin. Next, we intro- duce WikiReading Recycled—a newly devel- oped public dataset, and the task of multiple- property extraction. It uses the same data as WikiReading but does not inherit its predeces- sor’s identified disadvantages. In addition, we provide a human-annotated test set with diag- nostic subsets for a detailed analysis of model performance. 1 Introduction The emergence of attention-based models has rev- olutionized Natural Language Processing (Young et al., 2018). Pretraining these models on large cor- pora like BookCorpus (Zhu et al., 2015) has been shown to yield a reliable and robust base for down- stream tasks. These include Natural Language In- ference (Bowman et al., 2015), Question Answer- ing (Rajpurkar et al., 2016), Named Entity Recogni- tion (Yadav and Bethard, 2018; Goyal et al., 2018; Li et al., 2020), and Property Extraction (Hewlett et al., 2016). The creation of large supervised datasets often comes with trade-offs, such as one between the quality and quantity of data. For instance, the WikiReading dataset (Hewlett et al., 2016) has been created in such a way that WikiData annotations were treated as the expected answers for related Wikipedia articles. However, the above datasets were created separately, and the information con- tent of both sources overlaps only partially. Hence, the resulting dataset may contain noise. The best models can achieve results better than the human baseline across many NLP datasets such as MSCQAs (Wang et al., 2018), STS-B, QNLI (Raffel et al., 2020), CoLA or MRPC (Wang et al., 2020). However, as a consequence of differ- ent kinds of noise in the data, they rarely maximize the score metric (Stanislawek et al., 2019). While current work in NLP is focused on preparing new datasets, we regard recycling the current ones as equally important as creating a new one. Thus, after outperforming previous state-of-the-art on WikiReading, we investigated the dataset’s weak- nesses and created an entirely new, more challeng- ing Multi-Property Extraction task with improved data splits and a reliable, human-annotated test set. Contribution. The specific contributions of this work are the following. We analyzed the WikiRead- ing dataset and pointed out its weaknesses. We introduced a Multi-Property Extraction task by cre- ating a new dataset: WikiReading Recycled. Our dataset contains a human-annotated test set, with multiple subsets aimed to benchmark qualities such as generalization on unseen properties. We in- troduced a Mean-Multi-Property-F1 score suited for the new Multi-Property Extraction task. We …

Content truncated to 500 words