TEMP: Taxonomy Expansion with Dynamic Margin Loss through Taxonomy-Paths

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3854–3863 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3854 TEMP: Taxonomy Expansion with Dynamic Margin Loss through Taxonomy-Paths Zichen Liu1, Hongyuan Xu1, Yanlong Wen1 ∗, Ning Jiang2, Haiying Wu2, Xiaojie Yuan1 1TKLNDST, College of Computer Science, Nankai University, China 2Mashang Consumer Finance Co, Ltd {liuzichen, xuhongyuan, wenyl, yuanxj}@dbis.nankai.edu.cn {ning.jiang, haiying.wu02}@msxf.com Abstract As an essential form of knowledge represen- tation, taxonomies are widely used in vari- ous downstream natural language processing tasks. However, with the continuously rising of new concepts, many existing taxonomies are unable to maintain coverage by manual ex- pansion. In this paper, we propose TEMP, a self-supervised taxonomy expansion method, which predicts the position of new concepts by ranking the generated taxonomy-paths. For the first time, TEMP employs pre-trained con- textual encoders in taxonomy construction and hypernym detection problems. Experiments prove that pre-trained contextual embeddings are able to capture hypernym-hyponym rela- tions. To learn more detailed differences be- tween taxonomy-paths, we train the model with dynamic margin loss by a novel dynamic margin function. Extensive evaluations exhibit that TEMP outperforms prior state-of-the-art taxonomy expansion approaches by 14.3% in accuracy and 15.8% in mean reciprocal rank on three public benchmarks. 1 Introduction Taxonomies, tree-structured semantic hierarchies that organize entities by hypernym-hyponym (is-a) relations, play an important role in many NLP tasks such as question answering (Yang et al., 2017), query understanding (Hua et al., 2016) and infor- mation extraction (Demeester et al., 2016). Manually curated taxonomies usually face the limited coverage issue, especially when new con- cepts arise continuously. A low coverage taxonomy can largely hurt the performance of downstream tasks relied on it. Moreover, for maintaining and expanding existing taxonomies, the curation pro- cess that requires domain experts is expensive and time-consuming. Thus, we study the automatic tax- onomy expansion task (Figure 1): given an existing taxonomy, a text corpus, and a set of concepts, the ∗ Corresponding author goal is to expand the taxonomy by inserting con- cepts into it. Two common strategies used to study taxonomy construction and expansion are pattern-based meth- ods (e.g. the Hearst pattern (Hearst, 1992)) and distributional methods (Yu et al., 2015). Recent evidence suggests that the semantic information or structural features encoding in the representa- tion is an effective way to solve the task, especially probability statistics from a large corpus (Mikolov et al., 2013), semantic information extracted from text data (Yin and Roth, 2018), and properties of hypernym-hyponym relations such as strict partial order (Dash et al., 2020). Since taxonomies can be formulated as directed acyclic graphs (DAGs), the graph structure has been seen as important information for taxonomy expansion and construction in recent works (Shang et al., 2020; Shen et al., 2020). However, according to our observation, the path composed of ancestor nodes in a taxonomy is a more appropriately en- coded object in hypernym-hyponym relations. In a tree-structured taxonomy, all the ancestor nodes have an is-a relation with the child node. In Figure 1, …

Content truncated to 500 words