An Investigation of Transfer Learning-Based Sentiment Analysis in Japanese

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4652–4657 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4652 An Investigation of Transfer Learning-Based Sentiment Analysis in Japanese Enkhbold Bataa ExaWizards, Inc. / Tokyo, Japan enkhbold.bataa@exwzd.com Joshua Wu ExaWizards, Inc. / Tokyo, Japan joshua.wu@exwzd.com Abstract Text classification approaches have usually re- quired task-specific model architectures and huge labeled datasets. Recently, thanks to the rise of text-based transfer learning techniques, it is possible to pre-train a language model in an unsupervised manner and leverage them to perform effectively on downstream tasks. In this work we focus on Japanese and show the potential use of transfer learning techniques in text classification. Specifically, we perform bi- nary and multi-class sentiment classification on the Rakuten product review and Yahoo movie review datasets. We show that trans- fer learning-based approaches perform better than task-specific models trained on 3 times as much data. Furthermore, these approaches perform just as well for language modeling pre-trained on 1 30 of Wikipedia. We release our pre-trained models and code as open source. 1 Introduction Sentiment analysis is a well-studied task in the field of natural language processing and informa- tion retrieval (Sadegh et al., 2012; Hussein, 2018). In the past few years, researchers have made sig- nificant progress from models that make use of deep learning techniques.(Kim, 2014; Lai et al., 2015; Chen et al., 2017; Lin et al., 2017). How- ever, while there has been significant progress in sentiment analysis for English, not much effort has been invested in analyzing Japanese due to its sparse nature and the dependency on large datasets required by deep learning. Japanese script con- tains no whitespace, and sentences may be am- biguous such that there are multiple ways to split characters into words, each with a completely dif- ferent meaning. To see if existing research can make progress in Japanese, we make use of re- cent transfer learning models such as ELMo (Pe- ters et al., 2018), ULMFiT (Howard and Ruder, 2018), and BERT (Devlin et al., 2018) to each pre- train a language model which can then be used to perform downstream tasks. We test the models on binary and multi-class classification. Figure 1: Transfer learning-based text classification. First, we train the LM on a large corpus. Then, we fine- tune it on a target corpus. Finally, we train the classifier using labeled examples. The training process involves three stages as il- lustrated in Figure 1. The basic idea is similar to how fine-tuning ImageNet (Deng et al., 2009) helps many computer vision tasks (Huh et al., 2016). However, this model does not require la- beled data for pre-training. Instead, we pre-train a language model in unsupervised manner and then fine-tune it on a domain-specific dataset to efficiently classify using much less data. This is highly desired since there is a lack of large labeled datasets in practice. 2 Contributions The following are the primary contributions of this paper: • We experiment …

Content truncated to 500 words