Constructing Taxonomies from Pretrained Language Models

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4687–4700 June 6–11, 2021. ©2021 Association for Computational Linguistics 4687 Constructing Taxonomies from Pretrained Language Models Catherine Chen∗ Kevin Lin∗ University of California, BerkeleyDan Klein {cathychen,k-lin,klein}@berkeley.edu Abstract We present a method for constructing tax- onomic trees (e.g., WORDNET) using pre- trained language models. Our approach is composed of two modules, one that predicts parenthood relations and another that recon- ciles those predictions into trees. The par- enthood prediction module produces likeli- hood scores for each potential parent-child pair, creating a graph of parent-child rela- tion scores. The tree reconciliation module treats the task as a graph optimization prob- lem and outputs the maximum spanning tree of this graph. We train our model on subtrees sampled from WORDNET, and test on non- overlapping WORDNET subtrees. We show that incorporating web-retrieved glosses can further improve performance. On the task of constructing subtrees of English WORDNET, the model achieves 66.7 ancestor F1, a 20.0% relative increase over the previous best pub- lished result on this task. In addition, we convert the original English dataset into nine other languages using OPEN MULTILINGUAL WORDNET and extend our results across these languages. 1 Introduction A variety of NLP tasks use taxonomic information, including question answering (Miller, 1998) and information retrieval (Yang and Wu, 2012). Tax- onomies are also used as a resource for building knowledge and systematicity into neural models (Peters et al., 2019; Geiger et al., 2020; Talmor et al., 2020). NLP systems often retrieve taxonomic information from lexical databases such as WORD- NET (Miller, 1998), which consists of taxonomies that contain semantic relations across many do- mains. While manually curated taxonomies pro- vide useful information, they are incomplete and expensive to maintain (Hovy et al., 2009). * indicates equal contribution Traditionally, methods for automatic taxonomy construction have relied on statistics of web-scale corpora. These models generally apply lexico- syntactic patterns (Hearst, 1992) to large corpora, and use corpus statistics to construct taxonomic trees (e.g., Snow et al., 2005; Kozareva and Hovy, 2010; Bansal et al., 2014; Mao et al., 2018; Shang et al., 2020). Figure 1: An example subtree from the WORDNET hi- erarchy. In this work, we propose an approach that constructs taxonomic trees using pretrained lan- guage models (CTP). Our results show that direct access to corpus statistics at test time is not neces- sary. Indeed, the re-representation latent in large- scale models of such corpora can be beneficial in constructing taxonomies. We focus on the task pro- posed by Bansal et al. (2014), where the task is to organize a set of input terms into a taxonomic tree. We convert this dataset into nine other languages using synset alignments collected in OPEN MULTI- LINGUAL WORDNET and evaluate our approach in these languages. CTP first finetunes pretrained language mod- els to predict the likelihood of pairwise parent- child relations, producing a graph of parenthood scores. Then it reconciles these predictions with a …

Content truncated to 500 words