Does it Really Generalize Well on Unseen Data? Systematic Evaluation of Relational Triple Extraction Methods

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3849 - 3858 July 10-15, 2022 ©2022 Association for Computational Linguistics Does it Really Generalize Well on Unseen Data? Systematic Evaluation of Relational Triple Extraction Methods Juhyuk Lee1∗ †, Min-Joong Lee2†, June Yong Yang3†, Eunho Yang3 Samsung Research, Samsung Electronics, South Korea1, Samsung Advanced Institute of Technology, Samsung Electronics, South Korea2, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea3 {juhyuk.lee, minjoong.lee}@samsung.com {laoconeth, eunhoy}@kaist.ac.kr Abstract The ability to extract entities and their relations from unstructured text is essential for the auto- mated maintenance of large-scale knowledge graphs. To keep a knowledge graph up-to-date, an extractor needs not only the ability to recall the triples it encountered during training, but also the ability to extract the new triples from the context that it has never seen before. In this paper, we show that although existing extrac- tion models are able to easily memorize and recall already seen triples, they cannot general- ize effectively for unseen triples. This alarming observation was previously unknown due to the composition of the test sets of the go-to bench- mark datasets, which turns out to contain only 2% unseen data, rendering them incapable to measure the generalization performance. To separately measure the generalization perfor- mance from the memorization performance, we emphasize unseen data by rearranging datasets, sifting out training instances, or augmenting test sets. In addition to that, we present a simple yet effective augmentation technique to promote generalization of existing extraction models, and experimentally confirm that the proposed method can significantly increase the generalization performance of existing models. 1 Introduction Relational Triple Extraction (RTE), a more gener- alized version of Relation Extraction, is the task of extracting all relational triples in the form of (sub- ject, relation, object) from a given sentence. The ability to extract such triples is much required in the construction and maintenance of knowledge graphs such as Dbpedia (Auer et al., 2007), Freebase (Bol- lacker et al., 2008), and Wikidata (Vrandeˇci´c and Krötzsch, 2014) from documents containing a large number of new and emerging information. ∗This work was done when Juhyuk Lee was with KAIST as a student. †Equal contribution. With language model pretraining (Devlin et al., 2019; Radford et al., 2019), RTE methods achieved a new state-of-the-art (Wei et al., 2020; Wang et al., 2020; Zheng et al., 2021). However, whether the performance of these methods attributes to their capabilities of recalling already seen data or their ability to generalize and extract relations from un- seen data is yet to be scrutinized. To separately evaluate memorization and gen- eralization, we categorize the triples in the test set into three types: entirely seen (completely overlaps with triples in their respective training sets), partially seen (overlaps partially), and un- seen (completely new). We analyze common RTE benchmark datasets NYT (Riedel et al., 2010) and WebNLG (Gardent et al., 2017) using these cate- gories, and find that 89.61% and 91.10% of …

Content truncated to 500 words