Tri-Train: Automatic Pre-Fine Tuning between Pre-Training and Fine-Tuning for SciNER

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 460–475 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 460 A Fully Hyperbolic Neural Model for Hierarchical Multi-Class Classification Federico L´opez Michael Strube Heidelberg Institute for Theoretical Studies Research Training Group AIPHES firstname.lastname@h-its.org Abstract Label inventories for fine-grained entity typing have grown in size and complexity. Nonethe- less, they exhibit a hierarchical structure. Hy- perbolic spaces offer a mathematically appeal- ing approach for learning hierarchical repre- sentations of symbolic data. However, it is not clear how to integrate hyperbolic components into downstream tasks. This is the first work that proposes a fully hyperbolic model for multi-class multi-label classification, which performs all operations in hyperbolic space. We evaluate the proposed model on two chal- lenging datasets and compare to different base- lines that operate under Euclidean assump- tions. Our hyperbolic model infers the latent hierarchy from the class distribution, captures implicit hyponymic relations in the inventory, and shows performance on par with state-of- the-art methods on fine-grained classification with remarkable reduction of the parameter size. A thorough analysis sheds light on the impact of each component in the final predic- tion and showcases its ease of integration with Euclidean layers. 1 1 Introduction Entity typing classifies textual mentions of enti- ties, according to their semantic class, within a set of labels (or classes) organized in an inventory. The task has progressed from recognizing a few coarse classes (Sang and De Meulder, 2003), to extremely large inventories, with hundreds (Gillick et al., 2014) or thousands of labels (Choi et al., 2018). Therefore, exploiting inter-label correla- tions has become critical to improve performance. Large inventories tend to exhibit a hierarchical structure, either by an explicit tree-like arrange- ment of the labels (coarse labels at the top, fine- grained at the bottom), or implicitly through the 1Code available at: https://github.com/nlpAThits/hyfi Figure 1: Tree embedded in hyperbolic space. Items at the top of the hierarchy are placed near the ori- gin of the space, and lower items near the boundary. Moreover, the hyperbolic distance (Eq. 1) between sib- ling nodes resembles the one through the common an- cestor, analogous to the distance in the tree. That is dD(D, E) ≈ dD(D, B) + dD(B, E). label distribution in the dataset (coarse labels ap- pear more frequently than fine-grained ones). Prior work has integrated only explicit hierarchical in- formation by formulating a hierarchy-aware loss (Murty et al., 2018; Xu and Barbosa, 2018) or by representing instances and labels in a joint Eu- clidean embedding space (Shimaoka et al., 2017; Abhishek et al., 2017). However, the resulting space is hard to interpret, and these methods fail to capture implicit relations in the label inventory. Hy- perbolic space is naturally equipped for embedding symbolic data with hierarchical structures (Nickel and Kiela, 2017). Intuitively, that is because the amount of space grows exponentially as points move away from the origin. This mirrors the expo- nential growth of the number of nodes in trees with increasing distance from …

Content truncated to 500 words