Towards Scalable and Reliable Capsule Networks for Challenging NLP Applications

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1549–1559 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1549 Towards Scalable and Reliable Capsule Networks for Challenging NLP Applications Wei Zhao†, Haiyun Peng‡, Steffen Eger†, Erik Cambria‡ and Min YangΦ † Computer Science Department, Technische Universit¨at Darmstadt, Germany ‡ School of Computer Science and Engineering, Nanyang Technological University, Singapore Φ Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, China www.aiphes.tu-darmstadt.de Abstract Obstacles hindering the development of cap- sule networks for challenging NLP applica- tions include poor scalability to large out- put spaces and less reliable routing processes. In this paper, we introduce (i) an agreement score to evaluate the performance of routing processes at instance level; (ii) an adaptive optimizer to enhance the reliability of rout- ing; (iii) capsule compression and partial rout- ing to improve the scalability of capsule net- works. We validate our approach on two NLP tasks, namely: multi-label text classification and question answering. Experimental results show that our approach considerably improves over strong competitors on both tasks. In ad- dition, we gain the best results in low-resource settings with few training instances.1 1 Introduction In recent years, deep neural networks have achieved outstanding success in natural language processing (NLP), computer vision and speech recognition. However, these deep models are data- hungry and generalize poorly from small datasets, very much unlike humans (Lake et al., 2015). This is an important issue in NLP since sen- tences with different surface forms can convey the same meaning (paraphrases) and not all of them can be enumerated in the training set. For exam- ple, Peter did not accept the offer and Peter turned down the offer are semantically equivalent, but use different surface realizations. In image classification, progress on the gener- alization ability of deep networks has been made by capsule networks (Sabour et al., 2017; Hinton et al., 2018). They are capable of generalizing to the same object in different 3D images with vari- ous viewpoints. 1Our code is publicly available at http://bit.ly/311Dcod Jerry completed his project. Jerry managed to finish his project. Jerry succeeded in finishing his project. Extrapolate Extrapolated sentences Unseen sentences Observed sentences Extrapolate operation Extrapolation regime Jerry is sleeping. Figure 1: The extrapolation regime for an observed sentence can be found during training. Then, the un- seen sentences in this regime may be generalized suc- cessfully. Such generalization capability can be learned from examples with few viewpoints by extrapo- lation (Hinton et al., 2011). This suggests that capsule networks can similarly abstract away from different surface realizations in NLP applications. Figure 1 illustrates this idea of how observed sentences in the training set are generalized to un- seen sentences by extrapolation. In contrast, tra- ditional neural networks require massive amounts of training samples for generalization. This is especially true in the case of convolutional neu- ral networks (CNNs), where pooling operations wrongly discard positional information and do not consider hierarchical relationships between local features (Sabour …

Content truncated to 500 words