PRADO: Projection Attention Networks for Document Classification On-Device

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5012–5021, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5012 PRADO: Projection Attention Networks for Document Classification On-Device Prabhu Kaliamoorthi Google Research Mountain View, CA, USA prabhumk@google.com Sujith Ravi Google Research Mountain View, CA, USA sravi@google.com Zornitsa Kozareva Google Mountain View, CA, USA zornitsa@kozareva.com Abstract Recently, there has been a great interest in the development of small and accurate neural net- works that run entirely on devices such as mo- bile phones, smart watches and IoT. This en- ables user privacy, consistent user experience and low latency. Although a wide range of ap- plications have been targeted from wake word detection to short text classification, yet there are no on-device networks for long text classi- fication. We propose a novel projection attention neural network PRADO that combines trainable pro- jections with attention and convolutions. We evaluate our approach on multiple large docu- ment text classification tasks. Our results show the effectiveness of the trainable projection model in finding semantically similar phrases and reaching high performance while main- taining compact size. Using this approach, we train tiny neural networks just 200 Kilobytes in size that improve over prior CNN and LSTM models and achieve near state of the art per- formance on multiple long document classi- fication tasks. We also apply our model for transfer learning, show its robustness and abil- ity to further improve the performance in lim- ited data scenarios. 1 Introduction One of the fundamental tasks in Natural Language Processing is related to long text classification. Given a document, the goal is to assign one or more categories of interest to the text. This task is of high importance as it has wide applications in spam detection (Jindal and Liu, 2007), product categorization (Kozareva, 2015), sentiment classi- fication (Pang and Lee, 2008) and it also plays an important role for improving document retrieval and ranking (Deerwester et al., 1990). For a long time, the most successful text clas- sification approaches relied on sparse lexical fea- tures such as n-grams, which are later used by lin- ear or kernel models (Joachims, 1998; McCallum and Nigam, 1998; Joulin et al., 2016). However, with the recent advancements in deep learning, various neural network architectures like CNN (Kim, 2014), LSTM (Zhang et al., 2015), hierar- chical attention mechanisms (Yang et al., 2016) showed improvement in performance. Recently, (Ravi and Kozareva, 2018) and (Ravi and Kozareva, 2019) showed the importance of building on-device neural models for short text classification, which preserve user privacy, en- able consistent user experience and most impor- tantly perform inference on the device. One of the biggest challenges is how to fit these large and complex neural networks on devices with limited memory and computation capac- ity while still maintaining high performance. (Ravi and Kozareva, 2018, 2019) developed on- device self-governing neural networks (SGNN) and (SGNN++) based on locality-sensitive projec- tions (Ravi, 2017, …

Content truncated to 500 words