Embedding Semantic Taxonomies

Proceedings of the 28th International Conference on Computational Linguistics, pages 1279–1291 Barcelona, Spain (Online), December 8-13, 2020 1279 Embedding Semantic Taxonomies Alyssa Lees Google Research alyssalees@google.com Chris Welty Google Research cawelty@gmail.com Jacek Korycki Google korycki@gmail.com Sara Mc Carthy Google Research sara.m.mccarthy@gmail.com Shubin Zhao Google Research shubin@google.com Abstract A common step in developing an understanding of a vertical domain, e.g. shopping, dining, movies, medicine, etc., is curating a taxonomy of categories specific to the domain. These hu- man created artifacts have been the subject of research in embeddings that attempt to encode aspects of the partial ordering property of taxonomies. We compare Box Embeddings, a natu- ral containment-based representation of taxonomies, to partial-order embeddings and a baseline Bayes Net, in the context of representing the Medical Subject Headings (MeSH) taxonomy given a set of 300K PubMed articles with subject labels from MeSH. We deeply explore the experimen- tal properties of training box embeddings, including preparation of the training data, sampling ratios and class balance, initialization strategies, and propose a fix to the original box objective. We then present first results in using these techniques for representing a bipartite learning prob- lem (i.e. collaborative filtering) in the presence of taxonomic relations within each partition, inferring disease (anatomical) locations from their use as subject labels in journal articles. Our box model substantially outperforms all baselines for taxonomic reconstruction and bipartite re- lationship experiments. This performance improvement is observed both in overall accuracy and the weighted spread by true taxonomic depth. 1 Introduction Recent work on hierarchical representational structures in machine learning promise to blend the value of human curated taxonomies with the power and flexibility of machine learning systems. A plethora of such taxonomies exist across many domains as seen in libraries, linguistic resources, medicine and popular culture, yet they rely on an assumption of discreteness – category membership is either true or false – and this assumption does not generally lend itself to modelling with continuous valued systems or spatial embeddings. Taxonomic knowledge can be leveraged in the semantic space to extract hierarchical relations between words, and this simple observation has been the basis of many computational resources for linguistics, such as WordNet, EuroWordNet, FrameNet, etc. For example, the similarity of the words lion and tiger may be indistinguishable from the similarity of lion and animal in a naive text embedding space, as they may simply be the same distance apart, but in WordNet the difference and similarity are made clear through the taxonomy. In many cases, this explicit knowledge can be derived from existing taxonomies, but without some way of understanding this knowledge in the embedding space, it is just another sim- ilarity signal. Recently, order embeddings addressed this issue by orienting similarity in two axes and adding a constraint to similarity embeddings that ensured more general terms had to be closer to the origin (Liu et al., 2012). More recently we’ve seen the introduction of box lattice embeddings, which treat categories in a taxonomy as n-dimensional boxes, with a constraint …

Content truncated to 500 words