An Empirical Study of Tokenization Strategies for Various Korean NLP Tasks

Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 133–142 December 4 - 7, 2020. c⃝2020 Association for Computational Linguistics 133 An Empirical Study of Tokenization Strategies for Various Korean NLP Tasks Kyubyong Park1,∗ Joohong Lee2,∗ Seongbo Jang2,3,∗ Dawoon Jung2,∗ 1Kakao Brain, 2Pingpong AI Research, Scatter Lab, 3Pohang University of Science and Technology kyubyong.park@kakaobrain.com {joohong,dawoon}@scatterlab.co.kr jang.sb@postech.ac.kr Abstract Typically, tokenization is the very first step in most text processing works. As a token serves as an atomic unit that embeds the contextual in- formation of text, how to define a token plays a decisive role in the performance of a model. Even though Byte Pair Encoding (BPE) has been considered the de facto standard tokeniza- tion method due to its simplicity and universal- ity, it still remains unclear whether BPE works best across all languages and tasks. In this pa- per, we test several tokenization strategies in order to answer our primary research question, that is, “What is the best tokenization strategy for Korean NLP tasks?” Experimental results demonstrate that a hybrid approach of morphological segmentation fol- lowed by BPE works best in Korean to/from English machine translation and natural lan- guage understanding tasks such as KorNLI, KorSTS, NSMC, and PAWS-X. As an excep- tion, for KorQuAD, the Korean extension of SQuAD, BPE segmentation turns out to be the most effective. Our code and pre-trained models are pub- licly available at https://github.com/ kakaobrain/kortok. 1 Introduction Tokenization is the very first step in most text pro- cessing works. Not surprisingly, tremendous aca- demic efforts have been made to find the best tok- enization method for various NLP tasks. For the past few years, Byte Pair Encoding (BPE) (Gage, 1994) has been considered the de facto standard tokenization technique since it was reintroduced by Sennrich et al. (2016a). Besides the fact that BPE turns out to be very effective in the machine transla- tion task, another important reason BPE has gained ∗*Equal contribution. such popularity is that BPE is a data-driven sta- tistical algorithm so it is independent of language. However, it is still not clear whether BPE works best across all languages, irrespective of tasks. In this paper we study various tokenization strate- gies for Korean, a language which is morpholog- ically by far richer than English. Concretely, we empirically examine what is the best tokenization strategy for Korean to English / English to Ko- rean machine translation tasks, and natural lan- guage understanding (NLU) tasks—machine read- ing comprehension (MRC), natural language in- ference (NLI), semantic textual similarity (STS), sentiment analysis, and paraphrase identification. We are particularly interested in how complemen- tary BPE and linguistically motivated segmentation are. 2 Background 2.1 MeCab-ko: A Korean Morphological Analyzer MeCab (Kudo, 2006) is an open-source morpholog- ical analyzer based on Conditional Random Fields (CRFs). It is originally designed for Japanese, but also serves generic purposes so it can be applied to other languages. MeCab-ko1, a Korean extension of …

Content truncated to 500 words