Robust Text Classifier on Test-Time Budgets

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1167–1172, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1167 Robust Text Classifier on Test-Time Budgets Md Rizwan Parvez University of California Los Angeles rizwan@cs.ucla.edu Kai-Wei Chang University of California Los Angeles kwchang@cs.ucla.edu Tolga Bolukbasi Boston University tolgab@bu.edu Venkatesh Saligrama Boston University srv@bu.edu Abstract We design a generic framework for learning a robust text classification model that achieves high accuracy under different selection bud- gets (a.k.a selection rates) at test-time. We take a different approach from existing methods and learn to dynamically filter a large fraction of unimportant words by a low-complexity se- lector such that any high-complexity classifier only needs to process a small fraction of text, relevant for the target task. To this end, we propose a data aggregation method for training the classifier, allowing it to achieve competi- tive performance on fractured sentences. On four benchmark text classification tasks, we demonstrate that the framework gains consis- tent speedup with little degradation in accu- racy on various selection budgets. 1 Introduction Recent advances in deep neural networks (DNNs) have achieved high accuracy on many text classifi- cation tasks. These approaches process the entire text and encode words and phrases in order to per- form target tasks. While these models realize high accuracy, the computational time scales linearly with the size of the documents, which can be slow for a long document. In this context, various ap- proaches based on modifying the RNN or LSTM architecture have been proposed to speed up the process (Seo et al., 2018; Yu et al., 2017). How- ever, the processing in these models is still fun- damentally sequential and needs to operate on the whole document which limits the computational gain. In contrast to previous approaches, we pro- pose a novel framework for efficient text classifi- cation on long documents that mitigates sequential processing. The framework consists of a selector and a classifier. Given a selection budget as in- put, the selector performs a coarse one-shot selec- tion deleting unimportant words and pass the re- selector-n selector-2 selector-1 Great location but the food is not Great location but the food is not Great but not Great location but food not Great location food not Original input sentence Full text Output Aggregated corpus Output Output Fast and Robust Classifier Prediction Trained on aggregated corpus Figure 1: Our proposed framework. Given a selection rate, a selector is designed to select relevant words and pass them to the classifier. To make the classifier robust against fractured sentences, we aggregate outputs from different selectors and train the classifier on the aggregated corpus. mainder to the classifier. The classifier then takes the sentence fragments as an input and performs the target task. Figure 1 illustrates the proce- dure. This framework is general and agnostic to the architecture of the downstream classifier (e.g., RNN, CNN, Transformer). However, three challenges arise. First, to build …

Content truncated to 500 words