Inferring Concept Hierarchies from Text Corpora via Hyperbolic Embeddings

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3231–3241 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3231 3232 embeddings (Nickel and Kiela, 2017, 2018) as follows: Roller et al. (2018) showed that Hearst patterns can provide important constraints for hy- pernymy extraction from distributional contexts. However, it is also well-known that Hearst patterns suffer from missing and incorrect extractions, as words must co-occur in exactly the right pattern to be detected successfully. For this reason, we first extract potential is-a relationships from a corpus using Hearst patterns and build a directed weighted graph from these extractions. We then embed this Hearst Graph in hyperbolic space to infer missing hypernymy relations and remove wrong extractions. By using hyperbolic space for the embedding, we can exploit the following important advantages: Consistency Hyperbolic entailment cones (Ganea et al., 2018) allow us to enforce transitivity of is-a-relations in the entire embedding space. This improves the taxonomic consistency of the model, as it enforces that (x, is-a, z) if (x, is-a, y) and (y, is-a, z). To improve optimization properties, we also propose a new method to compute hyperbolic entailment cones in the Lorentz model of hyperbolic space. Efficiency Hyperbolic space allows for very low dimensional embeddings of graphs with latent hierarchies and heavy-tailed degree distribu- tions. For embedding large Hearst graphs – which exhibit both properties (e.g., see Fig- ure 2) – this is an important advantage. In our experiments, we will show that hyperbolic em- beddings allow us to decrease the embedding dimension by over an order of magnitude while outperforming SVD-based methods. Interpretability In hyperbolic embeddings, simi- larity is captured via distance while the gener- ality of terms is captured through their norms. This makes it easy to interpret the embeddings with regard to their hierarchical structure and allows us to get additional insights, e.g., about a term’s degree of generality. Figure 1 shows an example of a two-dimensional embedding of the Hearst graph that we use in our experiments. Although we will use higher dimen- sionalities for our final embedding, the visualiza- tion serves as a good illustration of the hierarchical structure that is obtained through the embedding. 2 Related Work Hypernym detection Detecting is-a-relations from text is a long-standing task in natural language processing. A popular approach is to exploit high- precision lexico-syntactic patterns as first proposed by Hearst (1992). These patterns may be prede- fined or learned automatically (Snow et al., 2005; Shwartz et al., 2016; Nakashole et al., 2012). How- ever, it is well known that such pattern-based meth- ods suffer significantly from missing extractions as terms must occur in exactly the right configuration to be detected (Shwartz et al., 2016; Roller et al., 2018). Recent works improve coverage by lever- aging search engines (Kozareva and Hovy, 2010) or by exploiting web-scale corpora (Seitner et al., 2016); but also come with precision trade-offs. To overcome the sparse extractions of pattern- based methods, focus has recently shifted to …

Content truncated to 500 words