Automatically Extracting Variant-Normalization Pairs for Japanese Text Normalization

Proceedings of the The 8th International Joint Conference on Natural Language Processing, pages 937–946, Taipei, Taiwan, November 27 – December 1, 2017 c⃝2017 AFNLP Automatically Extracting Variant-Normalization Pairs for Japanese Text Normalization Itsumi Saito Kyosuke Nishida Kugatsu Sadamitsu∗ Kuniko Saito Junji Tomita NTT Media Intelligence Laboratories {saito.itsumi, nishida.kyosuke}@lab.ntt.co.jp {saito.kuniko, tomita.junji}@lab.ntt.co.jp k.sadamitsu.ic@future.co.jp Abstract Social media texts, such as tweets from Twitter, contain many types of non- standard tokens, and the number of nor- malization approaches for handling such noisy text has been increasing. We present a method for automatically extracting pairs of a variant word and its normal form from unsegmented text on the basis of a pair-wise similarity approach. We incor- porated the acquired variant-normalization pairs into Japanese morphological analy- sis. The experimental results show that our method can extract widely covered vari- ants from large Twitter data and improve the recall of normalization without degrad- ing the overall accuracy of Japanese mor- phological analysis. 1 Introduction Social media texts contain many non-standard tokens (lexical variants), e.g., by lengthen- ing (“goooood” for “good”) or abbreviating them (“tmrw” for “tomorrow”). Current lan- guage processing systems often fail to ana- lyze such non-standard tokens, so normaliz- ing them into standard tokens as a prepro- cess is promising for analyzing such noisy texts robustly (Cook and Stevenson, 2009; Han et al., 2012; Li and Liu, 2012, 2014). The normalization task mainly consists of two components. One is detecting variant words and generating normaliza- tion candidates. The other is constructing a word lattice from possible normalization candidates and decoding to select the best normalized word se- quences. Early work on normalization focused on supervised approaches using labeled text, e.g., an approach based on a statistical machine translation ∗Present affiliation: Future Architect,Inc. (Aw et al., 2006; Pennell and Liu, 2011). How- ever, social network service (SNS) text has a dy- namic nature, and large SNS text is costly to an- notate. Recent work has been focused on unsuper- vised approaches. For example, Han et al. (2012) proposed generating variant-normalization pairs automatically on the basis of distributional simi- larity and string similarity. Hassan and Menezes (2013) developed the approach by using a graph- based approach. Yang and Eisenstein (2013) in- troduced a highly accurate unsupervised normal- ization model. As just described, unsupervised methods have been developed for English normal- ization tasks. Japanese SNS text also contains variant words, and several normalization methods have been pro- posed (Sasano et al., 2013; Kaji and Kitsuregawa, 2014; Saito et al., 2014). The basic framework of Japanese normalization is quite similar to that of English normalization. However, the problem is more complicated in Japanese normalization be- cause Japanese words are not segmented using explicit delimiters, so we have to estimate word segmentation simultaneously in the decoding step. Variant words are also more difficult to extract automatically in Japanese than in explicitly seg- mented languages such as English. Unlike En- glish normalization, the approaches for generating normalization candidates in Japanese are based on manually created rules or supervised training us- ing …

Content truncated to 500 words