Stochastic Tokenization with a Language Model for Neural Text Classification

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1620–1629 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1620 Stochastic Tokenization with a Language Model for Neural Text Classification Tatsuya Hiraoka1, Hiroyuki Shindo2, and Yuji Matsumoto2,3 1Tokyo Institute of Technology 2Nara Institute of Science and Technology 3RIKEN Center for Advanced Intelligence Project (AIP) 1tatsuya.hiraoka@nlp.c.titech.ac.jp 2{shindo,matsu}@is.naist.jp Abstract For unsegmented languages such as Japanese and Chinese, tokenization of a sentence has a significant impact on the performance of text classification. Sentences are usually seg- mented with words or subwords by a morpho- logical analyzer or byte pair encoding and then encoded with word (or subword) representa- tions for neural networks. However, segmenta- tion is potentially ambiguous, and it is unclear whether the segmented tokens achieve the best performance for the target task. In this pa- per, we propose a method to simultaneously learn tokenization and text classification to ad- dress these problems. Our model incorporates a language model for unsupervised tokeniza- tion into a text classifier and then trains both models simultaneously. To make the model robust against infrequent tokens, we sampled segmentation for each sentence stochastically during training, which resulted in improved performance of text classification. We con- ducted experiments on sentiment analysis as a text classification task and show that our method achieves better performance than pre- vious methods. 1 Introduction Tokenization is a fundamental problem in text classification such as sentiment analysis (Tang et al., 2014; Kim, 2014; dos Santos and Gatti, 2014), topic detection (Lai et al., 2015; Zhang et al., 2015), and spam detection (Liu and Jia, 2012; Liu et al., 2016). In text classification with neural networks, sentence representation is cal- culated based on tokens that compose the sen- tence. Specifically, a sentence is first tokenized into meaningful units such as characters, words, and subwords (Zhang et al., 2015; Joulin et al., 2017). Then, the token embeddings are looked up and fed into a neural network encoder such as a feed-forward neural network (Iyyer et al., 2015), a convolutional neural network (CNN) (Kim, 2014; Kalchbrenner et al., 2014), or a long short-term memory (LSTM) network (Wang et al., 2016a,b). For English and other languages that use the Latin alphabet, the whitespace is a good indica- tor of word segmentation. However, tokenization is a non-trivial problem in unsegmented languages such as Chinese and Japanese since they have no explicit word boundaries. For these languages, tokenizers based on supervised machine learning with a dictionary (Zhang et al., 2003; Kudo, 2006) have been used to segment a sentence into units (Lai et al., 2015). In addition, we use a neural network-based word segmenter to tokenize a raw corpus in Chinese text classification (Zhou et al., 2016; Zhang and Yang, 2018). In machine trans- lation, subword tokenization with byte pair en- coding (BPE) addresses the problem of unknown words and improves performance (Sennrich et al., 2016). However, segmentation is potentially ambigu- ous, and it is unclear whether preset tokeniza- tion offers …

Content truncated to 500 words