Parameter-Efficient Korean Character-Level Language Modeling

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2350–2356 May 2-6, 2023 ©2023 Association for Computational Linguistics Parameter-Efficient Korean Character-Level Language Modeling Marco Cognetta Tokyo Institute of Technology cognetta.marco@gmail.com Sangwhan Moon Tokyo Institute of Technology sangwhan@iki.fi Lawrence Wolf-Sonkin Google wolfsonkin@google.com Naoaki Okazaki Tokyo Institute of Technology okazaki@c.titech.ac.jp Abstract Character-level language modeling has been shown empirically to perform well on highly agglutinative or morphologically rich lan- guages while using only a small fraction of the parameters required by (sub)word models. Korean fits nicely into this framework, except that, like other CJK languages, it has a very large character vocabulary of 11,172 unique syllables. However, unlike Japanese Kanji and Chinese Hanzi, each Korean syllable can be uniquely factored into a small set of subcharac- ters, called jamo. We explore a "three-hot" scheme, where we ex- ploit the decomposability of Korean characters to model at the syllable level but using only jamo-level representations. We find that our three-hot embedding and decoding scheme al- leviates the two major issues with prior syllable- and jamo-level models. Namely, it requires fewer than 1% of the embedding parameters of a syllable model, and it does not require tripling the sequence length, as with jamo mod- els. In addition, it addresses a theoretical flaw in a prior three-hot modeling scheme. Our experiments show that, even when reduc- ing the number of embedding parameters by > 99.6% (from 11.4M to just 36k), our model suffers no loss in translation quality compared to the baseline syllable model. 1 Introduction Subword modeling has been used for Korean to re- duce the required vocabulary size due to its aggluti- native nature and morphological richness. Several works have characterized the many subword tok- enization strategies for Korean (Park et al., 2020; Moon and Okazaki, 2020). (Sub)character modeling has been employed for translation in Asian languages (Nguyen et al., 2017; Ngo et al., 2019; Yu et al., 2017). Likewise, it has found use in generic Korean NLP tasks (Cho et al., 2019; Choi et al., 2017). Further, Stratos (2017) investigated exploiting the hierarchical nature of Korean characters for modeling. However, nearly all of these works are restricted to the input side only, either because the downstream task is a classi- fication task or it is a translation task where Korean is the source language, and so syllable generation is unnecessary. An exception is Song et al. (2018), where the authors propose a "multi-hot" scheme that appears on both the embedding and decoding sides of a sequence-to-sequence denoising autoen- coder applied to Korean spelling correction. The baseline approaches of syllable- and jamo- level modeling each have downsides. Syllable mod- els must have embeddings for all 11,172 syllables to provide full coverage, which requires an enor- mous number of parameters. On the other hand, jamo-level modeling requires fewer than 70 embed- dings, but the sequence lengths become 3x longer, which greatly slows inference in attention-based models. Here, we propose a three-hot model1 that addresses both of these issues, …

Content truncated to 500 words