HyperText: Endowing FastText with Hyperbolic Geometry

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1166–1171 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1166 HyperText: Endowing FastText with Hyperbolic Geometry Yudong Zhu Di Zhou Jinghui Xiao Xin Jiang Xiao Chen Qun Liu Huawei Noah’s Ark Lab {zhuyudong3,zhoudi7,xiaojinghui4,Jiang.Xin, chen.xiao2,qun.liu}@huawei.com Abstract Natural language data exhibit tree-like hi- erarchical structures such as the hypernym- hyponym relations in WordNet. FastText, as the state-of-the-art text classifier based on shal- low neural network in Euclidean space, may not model such hierarchies precisely with lim- ited representation capacity. Considering that hyperbolic space is naturally suitable for mod- eling tree-like hierarchical data, we propose a new model named HyperText for efficient text classification by endowing FastText with hy- perbolic geometry. Empirically, we show that HyperText outperforms FastText on a range of text classification tasks with much reduced pa- rameters. 1 Introduction FastText (Joulin et al., 2016) is a simple and effi- cient neural network for text classification, which achieves comparable performance to many deep models like char-CNN (Zhang et al., 2015) and VDCNN (Conneau et al., 2016), with a much lower computational cost in training and inference. How- ever, natural language data exhibit tree-like hierar- chies in several respects (Dhingra et al., 2018) such as the taxonomy of WordNet. In Euclidean space the representation capacity of a model is strictly bounded by the number of parameters. Thus, con- ventional shallow neural networks (e.g., FastText) may not represent tree-like hierarchies efficiently given limited model complexity. Fortunately, hyperbolic space is naturally suit- able for modeling the tree-like hierarchical data. Theoretically, hyperbolic space can be viewed as a continuous analogue of trees, and it can easily embed trees with arbitrarily low distortion (Kri- oukov et al., 2010). Experimentally, Nickel and Kiela (2017) first used the Poincar´e ball model to embed hierarchical data into hyperbolic space and Average Pooling Linear Classifier Poincar Embedding Word Embedding Einstein midpoint Hyperbolic Classifier M bius Linear visualize midpoint tree-like embedding Figure 1: The architecture comparison of FastText (up- per) and HyperText (lower). achieved promising results on learning word em- beddings in WordNet. Inspired by their work, we propose HyperText for text classification by endowing FastText with hyperbolic geometry. We base our method on the Poincar´e ball model of hyperbolic space. Specif- ically, we exploit the Poincar´e ball embedding of words or ngrams to capture the latent hierarchies in natural language sentences. Further, we use the Einstein midpoint (Gulcehre et al., 2018) as the pooling method to emphasize semantically spe- cific words which usually contain more information but occur less frequently than general ones (Dhin- gra et al., 2018). Finally, we employ M¨obius lin- ear transformation (Ganea et al., 2018) to play the part of the hyperbolic classifier. We evaluate the performance of our approach on text classifi- cation task using ten standard datasets. We ob- serve HyperText outperforms FastText on eight of them. Besides, HyperText is much more parameter- efficient. Across different tasks, only 17% ∼ 50% parameters of FastText are needed for HyperText to achieve comparable performance. …

Content truncated to 500 words