OpenKorPOS: Democratizing Korean Tokenization with Voting-Based Open Corpus Annotation

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4975–4983 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4975 OpenKorPOS: Democratizing Korean Tokenization with Voting-Based Open Corpus Annotation Sangwhan Moon†∗, Won Ik Cho‡, Hyejoo Han♯, Naoaki Okazaki†, Nam Soo Kim‡ Tokyo Institute of Technology†, Google LLC∗, Seoul National University‡, Odd Concepts Inc.♯ sangwhan@iki.fi, wicho@hi.snu.ac.kr, hyejoo@oddconcepts.kr, okazaki@c.titech.ac.jp, nkim@snu.ac.kr Abstract Korean is a language with complex morphology that uses spaces at larger-than-word boundaries, unlike other East-Asian languages. While morpheme-based text generation can provide significant semantic advantages compared to commonly used character-level approaches, Korean morphological analyzers only provide a sequence of morpheme-level tokens, losing information in the tokenization process. Two crucial issues are the loss of spacing information and subcharacter level morpheme normalization, both of which make the tokenization result challenging to reconstruct the original input string, deterring the application to generative tasks. As this problem originates from the conventional scheme used when creating a POS tagging corpus, we propose an improvement to the existing scheme, which makes it friendlier to generative tasks. On top of that, we suggest a fully-automatic annotation of a corpus by leveraging public analyzers. We vote the surface and POS from the outcome and fill the sequence with the selected morphemes, yielding tokenization with a decent quality that incorporates space information. Our scheme is verified via an evaluation done on an external corpus, and subsequently, it is adapted to Korean Wikipedia to construct an open, permissive resource. We compare morphological analyzer performance trained on our corpus with existing methods, then perform an extrinsic evaluation on a downstream task. Keywords: Korean NLP, POS tagging, Tokenization, Voting-based annotation 1. Introduction The morphology and script of the Korean language are different from those of Indo-European languages or other East-Asian languages such as Japanese and Chinese (Stratos, 2017; Park et al., 2018). In particu- lar, Korean uses spacing to increase readability, but not necessarily at word boundaries. The agglutinative prop- erties of Korean result in space tokenized boundaries larger than a word, but smaller than a sentence. This particular unit is called Eojeol, which is a property that is not shared with other languages. Additionally, while there is a well-defined standard for spacing, the rules are complicated. Prior art suggests that elaborated text processing through morpheme-level analysis is regarded as partic- ularly important in text generation (Kim et al., 2016). In the context of Korean, generally, a single toolkit tends to provide morpheme-level tokenization, mor- phological analysis and normalization, along with part- of-speech (POS) tagging. For these reasons, the differ- ent functions are closely related and commonly trained from a single corpus in an end-to-end manner. This property is inherited from the canonical Sejong Cor- pus’ format (Kim, 2006), which has been exploited to build and train these tools. However, the standard corpus tagging protocol1 has seen very few updates since it was initially proposed, and omits crucial information to reconstruct the tok- enized results back to their original form, the most ob- …

Content truncated to 500 words