Grapheme-level Awareness in Word Embeddings for Morphologically Rich Languages

Grapheme-level Awareness in Word Embeddings for Morphologically Rich Languages Suzi Park, Hyopil Shin Department of Linguistics, Seoul National University Gwanakro 1, Gwanak-gu, Seoul, South Korea {mam3b,hpshin}@snu.ac.kr Abstract Learning word vectors from character level is an effective method to improve word embeddings for morphologically rich languages. However, most of these techniques have been applied to languages that are inflectional and written in Roman alphabets. In this paper, we investigate languages that are agglutinative and represented by non-alphabetic scripts, choosing Korean as a case study. We present a grapheme-level coding procedure for neural word embedding that utilizes word-internal features that are composed of syllable characters (Character CNN). Observing that our grapheme-level model is more capable of representing functional and semantic similarities, grouping allomorphs, and disambiguating homographs than syllable-level and word-level models, we recognize the importance of knowledge on the morphological typology and diversity of writing systems. Keywords: Character-level CNN, Word Embedding, Grapheme-level embedding 1. Introduction Semantic word representations are essential to natural lan- guage processing. Most embedding models have learned the meaning of words from their contexts. These techniques work for English, in which word forms rarely change ac- cording to their contexts. Though, in morphologically rich languages, inflectional forms of the same base are consid- ered and learned as distinct words. Even if the base is frequent in a large corpus, each word forms rarely occurs, which leads to data sparsity. This weakness has been over- come to a certain extent by composing representation of subword units such as morphemes or characters. Character- based models can be generalized in a sense that they do not require a pre-trained morphological analyzer, and they enable to calculate vector representations even for out-of- vocabulary words. However, vector representations of characters present the same problem as those of words, due to a trade-off between vocabulary size and token frequency. Alphabet systems, like Roman alphabets, contain a comparatively small num- ber of symbols that occur frequently enough in a corpus. On the contrary, syllabaries have a more extensive inventory of syllable characters each of whose frequency is lower, mak- ing an analogy with morphologically rich languages. Our objective in this paper is to address the issue of data sparsity, which we argue is caused by not only the morpho- logical system of a language and but also its writing sys- tem. We begin by remarking on the morphological typology and diversity of writing systems. We then report two phe- nomena of languages that are agglutinative, represented by non-alphabetic scripts, or both, Korean as a case study. We find that they cannot be captured by the default method to code Korean text on the syllable-character level. From this, we propose a character-internal or grapheme-level coding procedure for neural word embedding that utilizes word- internal features that are composed of characters. We end by observing that our grapheme-level model is capable of representing functional and semantic similarities, grouping allomorphs, and disambiguating homographs. We conclude that it can improve existing character-level approaches to Korean NLP tasks. 2. Challenges in …

Content truncated to 500 words