Investigating Capsule Network and Semantic Feature on Hyperplanes for Text Classification

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 456–465, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 456 Investigating Capsule Network and Semantic Feature on Hyperplanes for Text Classification 12Chunning Du ∗, 12Haifeng Sun∗, 12Jingyu Wang†, 12Qi Qi, 12Jianxin Liao 12Chun Wang, 12Bing Ma 1State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing 100876, China 2EBUPT Information Technology Co., Ltd., Beijing 100191, China {duchunning,sunhaifeng 1,wangjingyu,qiqi}@ebupt.com Abstract As an essential component of natural language processing, text classification relies on deep learning in recent years. Various neural net- works are designed for text classification on the basis of word embedding. However, pol- ysemy is a fundamental feature of the natu- ral language, which brings challenges to text classification. One polysemic word contains more than one sense, while the word embed- ding procedure conflates different senses of a polysemic word into a single vector. Extract- ing the distinct representation for the specific sense could thus lead to fine-grained models with strong generalization ability. It has been demonstrated that multiple senses of a word actually reside in linear superposition within the word embedding so that specific senses can be extracted from the original word embed- ding. Therefore, we propose to use capsule networks to construct the vectorized represen- tation of semantics and utilize hyperplanes to decompose each capsule to acquire the spe- cific senses. A novel dynamic routing mech- anism named ‘routing-on-hyperplane’ will se- lect the proper sense for the downstream clas- sification task. Our model is evaluated on 6 different datasets, and the experimental re- sults show that our model is capable of ex- tracting more discriminative semantic features and yields a significant performance gain com- pared to other baseline methods. 1 Introduction Text classification is a crucial task in natural language processing, which has many applica- tions, such as sentiment analysis, intent identifica- tion and topic labeling[Aggarwal and Zhai, 2012; Wang and Manning, 2012a]. Recent years, many studies rely on neural networks and have shown promising performance. ∗ Authors contributed equally. †Corresponding author. The success of deep learning model for NLP is based on the progress in learning distributed word representations in semantic vector space, where each word is mapped to a vector called a word embedding. The word’s representation is calcu- lated relying on the distributional hypothesis - the assumption that semantically similar or related words appear in similar contexts [Mikolov et al., 2013; Langendoen, 1959]. Normally, each word’s representation is constructed by counting all its context features. However, for the polysemic word which contains multiple senses, the context fea- tures of different senses are mixed together, lead- ing to inaccurate word representation. As demon- strated in [Arora et al., 2018], multiple senses of a word actually reside in linear superposition within the word embedding: v ≈ α1vsense1+α2vsense2+α3vsense3+· · · , (1) where coefficients αi are nonnegative and vsense1, vsense2... are the hypothetical embed- dings of different …

Content truncated to 500 words