Low-resource Taxonomy Enrichment with Pretrained Language Models

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2747–2758 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 2747 Low-resource Taxonomy Enrichment with Pretrained Language Models Kunihiro Takeoka, Kosuke Akimoto, Masafumi Oyamada Data Science Research Laboratories, NEC Corporation {k_takeoka, kosuke_a, oyamada}@nec.com Abstract Taxonomies are symbolic representations of hierarchical relationships between terms or en- tities. While taxonomies are useful in broad applications, manually updating or maintain- ing them is labor-intensive and difficult to scale in practice. Conventional supervised methods for this enrichment task fail to find optimal parents of new terms in low-resource settings where only small taxonomies are avail- able because of overfitting to hierarchical rela- tionships in the taxonomies. To tackle the problem of low-resource taxon- omy enrichment, we propose Musubu, an ef- ficient framework for taxonomy enrichment in low-resource settings with pretrained language models (LMs) as knowledge bases to compen- sate for the shortage of information. Musubu leverages an LM-based classifier to determine whether or not inputted term pairs have hier- archical relationships. Musubu also utilizes Hearst patterns to generate queries to leverage implicit knowledge from the LM efficiently for more accurate prediction. We empirically demonstrate the effectiveness of our method in extensive experiments on taxonomies from both a SemEval task and real-world retailer datasets. 1 Introduction Taxonomies, which represent the hierarchical rela- tionships between terms, have been widely utilized in tasks related to information retrieval, recommen- dation, and classification (Agrawal et al., 2009; Huang et al., 2019; Babbar et al., 2016). Because the target domain of a specific taxonomy changes over time, taxonomies must be kept up to date so that newly introduced categories and hierarchical relationships can be properly integrated. However, manually constructing and maintaining taxonomies is a costly task due to their labor-intensive and domain-specific nature (Gao et al., 2018; Shen et al., 2018). Figure 1: Enrichment of taxonomy with sufficient re- sources (top) and low resources (bottom). Conven- tional supervised methods for taxonomy enrichment fail to enrich small seed taxonomies because the num- ber of hierarchical relations of term pairs used as train- ing samples is limited. The goal of a taxonomy enrichment task is to automate this costly maintenance (Jurgens and Pile- hvar, 2016; Shen et al., 2018), as shown in Fig. 1. Taxonomy enrichment methods enrich a taxonomy that may be incomplete, i.e., a seed taxonomy, by predicting new hierarchical relationships between terms in the seed taxonomy and new terms. We focus on low-resource taxonomy enrichment tasks, in which there are few terms in the seed taxonomy, specifically less than 10,000. Previous studies on hypernymy detection used a similar definition for low-resource settings (Yu et al., 2020). Although conventional methods for taxonomy enrichment use large taxonomies (e.g., WordNet taxonomy (90,000 terms) (Miller, 1995), Microsoft Academic Graph (355,000 terms) (Sinha et al., 2015), most of the ones used in realistic situations (as seen in 2748 §4) have around a thousand terms, so they are cate- gorized as low-resource settings. Existing supervised methods (Baroni et al., 2012; Shen et al., …

Content truncated to 500 words