Korean Language Modeling via Syntactic Guide

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2841–2849 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2841 Korean Language Modeling via Syntactic Guide Hyeondey Kim1, Seonhoon Kim2, Inho Kang2, Nojun Kwak3, and Pascale Fung1 1The Hong Kong University of Science and Technology 2Naver Search 3Seoul National University hdkimaa@connect.ust.hk, seonhoon.kim@navercorp.com, once.ihkang@navercorp.com, nojunk@snu.ac.kr, pascale@ece.ust.hk Abstract While pre-trained language models play a vital role in modern language processing tasks, but not every language can benefit from them. Most existing research on pre-trained language models focuses primarily on widely-used languages such as English, Chinese, and Indo-European languages. Additionally, such schemes usually require extensive computational resources alongside a large amount of data, which is infeasible for less-widely used languages. We aim to address this research niche by building a language model that understands the linguistic phenomena in the target language which can be trained with low-resources. In this paper, we discuss Korean language modeling, specifically methods for language representation and pre-training methods. With our Korean-specific language representation, we are able to build more powerful models for Korean understanding, even with fewer resources. The paper proposes chunk-wise reconstruction of the Korean language based on a widely used transformer architecture and bidirectional language representation. We also introduce morphological features such as Part-of-Speech (PoS) into the language understanding by leveraging such information during the pre-training. Our experiment results prove that the proposed methods improve the model performance of the investigated Korean language understanding tasks. Keywords: Neural language representation models, Semi-supervised, weakly-supervised and unsupervised learning, Part-of-Speech Tagging 1. Introduction Recent progress in machine learning have enabled neu- ral language models to move beyond traditional natu- ral language processing tasks such as sentiment analy- sis and pos-tagging. Modern language processing sys- tems are now equipped to handle complex tasks such as question answering (Rajpurkar et al., 2016), dialogue systems (Sun et al., 2019) and fact-checking (Thorne et al., 2018) that all require sophisticated language un- derstanding capabilities. The pre-trained language model (Devlin et al., 2018; Lewis et al., 2020) made significant breakthroughs in natural language processing. In most natural language processing tasks, contextual language representations trained from massive unsupervised learning with enor- mous plain texts achieve state-of-the-art performance. However, most of the computational linguistics re- search is focused on English. In order to build a lan- guage model for less commonly studied languages like Korean, it is necessary to focus on the target language’s linguistics characteristics. Unfortunately, the Korean language has very different linguistic structures from the other languages; Korean is classified as a language isolate. As a result, language modeling is extremely challenging in Korean. The concept of a language model can be explained as an algorithm that assigns probability values to words or sentences. Language models are typically trained by predicting the next token based on given context (Roark et al., 2007). However, the technique cannot be applied to languages with SOV order like Korean and Japanese. In a language with such structure, most vital informa- tion …

Content truncated to 500 words