Handling Out-Of-Vocabulary Problem in Hangeul Word Embeddings

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 3213–3221 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 3213 Handling Out-Of-Vocabulary Problem in Hangeul Word Embeddings Ohjoon Kwon1† Dohyun Kim2† Soo-Ryeon Lee3 Junyoung Choi3 SangKeun Lee2,3 1Samsung Research, Republic of Korea 2Department of Computer Science and Engineering, Korea University, Seoul, Republic of Korea 3Department of Artificial Intelligence, Korea University, Seoul, Republic of Korea ohjoon.kwon@samsung.com {dhkim1028, bonny05616, dnduddl2015, yalphy}@korea.ac.kr Abstract Word embedding is considered an essential factor in improving the performance of vari- ous Natural Language Processing (NLP) mod- els. However, it is hardly applicable in real- world datasets as word embedding is generally studied with a well-refined corpus. Notably, in Hangeul (Korean writing system), which has a unique writing system, various kinds of Out-Of-Vocabulary (OOV) appear from typos. In this paper, we propose a robust Hangeul word embedding model against typos, while maintaining high performance. The proposed model utilizes a Convolutional Neural Net- work (CNN) architecture with a channel atten- tion mechanism that learns to infer the origi- nal word embeddings. The model train with a dataset that consists of a mix of typos and cor- rect words. To demonstrate the effectiveness of the proposed model, we conduct three kinds of intrinsic and extrinsic tasks. While the existing embedding models fail to maintain stable per- formance as the noise level increases, the pro- posed model shows stable performance. 1 Introduction Word embedding refers to the process of generating vectors that contain semantic and syntactic infor- mation of languages. Machines can understand the meaning of words through the word embeddings. Several embedding methodologies have been pro- posed to improve various Natural Language Pro- cessing (NLP) models’ performance, such as ma- chine translation, sentence classification, and text generation (Mikolov et al., 2013; Pennington et al., 2014; Bojanowski et al., 2017). Word2vec (Mikolov et al., 2013) generates word vectors using the contextual data of words based †Equal contribution on the distributional hypothesis. Word2vec assigns a unique vector to each word, and has proved its effectiveness by improving various systems’ perfor- mance. Similarly, GloVe (Pennington et al., 2014) that uses co-occurrence information of each word found throughout the corpus, generates word vec- tors that contain semantic and syntactic character- istics. However, Word2vec and GloVe suffer from be- ing dependent on the training corpus. In particu- lar, they do not include the word embeddings that did not appear in the training process. To tackle this OOV problem, replacing unseen words with a unique token “UNK” is widely used as a solution. This method can allow the model to deal with OOV words. However, this countermeasure reduces the performance of the system. This is because several unseen words are recognized as having the same meaning. Hence, the importance of each word is ignored and recognized as noise. This OOV problem amplifies when dealing with real-world datasets that contain a large number of typos and intended slang that has not been included in the training phase. In particular, …

Content truncated to 500 words