ProSeqo: Projection Sequence Networks for On-Device Text Classification

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3894–3903, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3894 ProSeqo: Projection Sequence Networks for On-Device Text Classification Zornitsa Kozareva Google Mountain View, CA, USA zornitsa@kozareva.com Sujith Ravi Google Research Mountain View, CA, USA sravi@google.com Abstract We propose a novel on-device sequence model for text classification using recurrent projec- tions. Our model ProSeqo uses dynamic re- current projections without the need to store or look up any pre-trained embeddings. This re- sults in fast and compact neural networks that can perform on-device inference for complex short and long text classification tasks. We conducted exhaustive evaluation on multi- ple text classification tasks. Results show that ProSeqo outperformed state-of-the-art neural and on-device approaches for short text clas- sification tasks such as dialog act and intent prediction. To the best of our knowledge, ProSeqo is the first on-device long text clas- sification neural model. It achieved compara- ble results to previous neural approaches for news article, answers and product categoriza- tion, while preserving small memory footprint and maintaining high accuracy. 1 Introduction In the last decade, research in deep neural net- works has lead to tremendous advances and state- of-the-art performance on wide range of Natu- ral Language Processing (NLP), speech and vi- sion applications. Coupled with the tremendous growth and adoption of smart devices such as mo- bile phones, watches, Internet of Things, conver- sational AI devices like Alexa and Google Home, these deep learning improvements have resulted in text and speech becoming the primary modes of communication (Cohen, 2008). To date, the majority of the models run on the server side by taking user input, sending the data to the server (or cloud) for processing and returning the results back to the user. This often results in latency de- lays, inconsistent user experiences and concerns over user privacy. One way to surmount these challenges is to de- velop and deploy text and speech models that run inference entirely on-device and return accurate predictions in real-time. To make this work, the on-device models have to be very small in size (few kilobytes or megabytes) to fit on-devices like mobile phone, watch and IoT; to have low latency and be as accurate as server side models. These on-device challenges opened up new active area of research, which recently has shown promising results for speech (Lin et al., 2018), wake word detection (He et al., 2017), dialog act (Ravi and Kozareva, 2018) and intent prediction short text classification (Ravi and Kozareva, 2019). Previous attempts for on-device text classifica- tion used hashed <input text, output class> pairs. Unfortunately, such models cannot handle exam- ples that are not part of the lookup table and cannot generalize to more complex tasks like the ones we solve in this paper – long text classi- fication, conversational intent prediction. (Bui et al., 2018) combined graphs with neural net- works to improve the robustness …

Content truncated to 500 words