How Large a Vocabulary Does Text Classification Need? A Variational Approach to Vocabulary Selection

Proceedings of NAACL-HLT 2019, pages 3487–3497 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3487 How Large a Vocabulary Does Text Classification Need? A Variational Approach to Vocabulary Selection Wenhu Chen†, Yu Su¶, Yilin Shen‡, Zhiyu Chen†, Xifeng Yan†, William Wang† University of California, Santa Barbara† Ohio State University, Columbus¶ Samsung Research, Moutain View‡ {wenhuchen,zhiyuchen,xyan,william}@cs.ucsb.edu su.809@osu.edu yilin.shen@samsung.com Abstract With the rapid development in deep learn- ing, deep neural networks have been widely adopted in many real-life natural language ap- plications. Under deep neural networks, a pre- defined vocabulary is required to vectorize text inputs. The canonical approach to select pre- defined vocabulary is based on the word fre- quency, where a threshold is selected to cut off the long tail distribution. However, we observed that such a simple approach could easily lead to under-sized vocabulary or over- sized vocabulary issues. Therefore, we are interested in understanding how the end-task classification accuracy is related to the vocab- ulary size and what is the minimum required vocabulary size to achieve a specific perfor- mance. In this paper, we provide a more sophisticated variational vocabulary dropout (VVD) based on variational dropout to per- form vocabulary selection, which can intelli- gently select the subset of the vocabulary to achieve the required performance. To eval- uate different algorithms on the newly pro- posed vocabulary selection problem, we pro- pose two new metrics: Area Under Accuracy- Vocab Curve and Vocab Size under X% Ac- curacy Drop. Through extensive experiments on various NLP classification tasks, our varia- tional framework is shown to significantly out- perform the frequency-based and other selec- tion baselines on these metrics. 1 Introduction Over the past decade, deep neural networks have become arguably the most popular model choice for a vast number of natural language processing (NLP) tasks and have constantly been delivering state-of-the-art results. Because neural network models assume continuous data, to apply a neu- ral network on any text data, the first step is to Cutoff Freq. Vocab Remain Vocab #Emb #CNN #Emb Ratio 1 60K 100% 15M 0.36M 97.6% 5 40K 21.7% 10M 0.36M 95.6% 10 24K 13% 6M 0.36M 94.3% 20 14K 9.4% 3.5M 0.36M 90% 100 4K 2.7% 1M 0.36M 73% Table 1: Illustration of the frequency-based vocabulary selection heuristic on a typical CNN-based document classification model (Section 4.1). #Emb is the num- ber of parameters in the word embedding matrix (256 dimensions), and #CNN is that in the CNN model. vectorize the discrete text input with a word em- bedding matrix through look-up operation, which in turn assumes a pre-defined vocabulary set. For many NLP tasks, the vocabulary size can easily go up to the order of tens of thousands, which poten- tially makes the word embedding the largest por- tion of the trainable parameters. For example, a document classification task like AG-news (Zhang et al., 2015) can include up to 60K unique words, with the embedding matrix accounting for 97.6% of the trainable parameters (Table 1), which leads to under-representation of …

Content truncated to 500 words