User-Generated Text Corpus for Evaluating Japanese Morphological Analysis and Lexical Normalization

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5532–5541 June 6–11, 2021. ©2021 Association for Computational Linguistics 5532 User-Generated Text Corpus for Evaluating Japanese Morphological Analysis and Lexical Normalization Shohei Higashiyama1,2, Masao Utiyama1, Taro Watanabe2, Eiichiro Sumita1 1National Institute of Information and Communications Technology, Kyoto, Japan 2Nara Institute of Science and Technology, Nara, Japan {shohei.higashiyama, mutiyama, eiichiro.sumita}@nict.go.jp, taro@is.naist.jp Abstract Morphological analysis (MA) and lexical nor- malization (LN) are both important tasks for Japanese user-generated text (UGT). To evalu- ate and compare different MA/LN systems, we have constructed a publicly available Japanese UGT corpus. Our corpus comprises 929 sen- tences annotated with morphological and nor- malization information, along with category information we classified for frequent UGT- specific phenomena. Experiments on the cor- pus demonstrated the low performance of ex- isting MA/LN methods for non-general words and non-standard forms, indicating that the corpus would be a challenging benchmark for further research on UGT. 1 Introduction Japanese morphological analysis (MA) is a fun- damental and important task that involves word segmentation, part-of-speech (POS) tagging and lemmatization because the Japanese language has no explicit word delimiters. Although MA meth- ods for well-formed text (Kudo et al., 2004; Neu- big et al., 2011) have been actively developed tak- ing advantage of the existing annotated corpora of newswire domains, they perform poorly on user- generated text (UGT), such as social media posts and blogs. Additionally, because of the frequent occurrence of informal words, lexical normaliza- tion (LN), which identifies standard word forms, is another important task in UGT. Several stud- ies have been devoted to both tasks in Japanese UGT (Sasano et al., 2013; Kaji and Kitsuregawa, 2014; Saito et al., 2014, 2017) to achieve the robust performance for noisy text. Previous researchers have evaluated their own systems using in-house data created by individual researchers, and thus it is difficult to compare the performance of dif- ferent systems and discuss what issues remain in these two tasks. Therefore, publicly available data is necessary for a fair evaluation of MA and LN performance on Japanese UGT. In this paper, we present the blog and Q&A site normalization corpus (BQNC),1 which is a public Japanese UGT corpus annotated with morphologi- cal and normalization information. We have con- structed the corpus under the following policies: (1) available and restorable; (2) compatible with the segmentation standard and POS tags used in the existing representative corpora; and (3) enabling a detailed evaluation of UGT-specific problems. For the first requirement, we extracted and used the raw sentences in the blog and Q&A site registers compiled by (the non-core data of) the Balanced Corpus of Contemporary Written Japanese (BCCWJ) (Maekawa et al., 2014), in which the original sentences are preserved.2 For the second requirement, we followed the short unit word (SUW) criterion of the National Institute for Japanese Language and Linguistics (NINJAL), which is used in various NINJAL’s corpora, includ- ing manually annotated sentences in the BCCWJ. For the third requirement, …

Content truncated to 500 words