Improving Character-Aware Neural Language Model by Warming up Character Encoder under Skip-gram Architecture

Proceedings of Recent Advances in Natural Language Processing, pages 421–427 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_048 421 Improving Character-Aware Neural Language Model by Warming Up Character Encoder under Skip-gram Architecture Yukun Feng1, Chenlong Hu1, Hidetaka Kamigaito1, Hiroya Takamura2 and Manabu Okumura1 1Tokyo Institute of Technology 2National Institute of Advanced Industrial Science and Technology (AIST) {yukun,huchenlong,kamigaito}@lr.pi.titech.ac.jp oku@pi.titech.ac.jp takamura.hiroya@aist.go.jp Abstract Character-aware neural language models can capture the relationship between words by ex- ploiting character-level information and are particularly effective for languages with rich morphology. However, these models are usu- ally biased towards information from surface forms. To alleviate this problem, we pro- pose a simple and effective method to improve a character-aware neural language model by forcing a character encoder to produce word- based embeddings under Skip-gram architec- ture in a warm-up step without extra train- ing data. We empirically show that the re- sulting character-aware neural language model achieves obvious improvements of perplexity scores on typologically diverse languages, that contain many low-frequency or unseen words. 1 Introduction Neural language models (NLM) usually maintain a fixed vocabulary and map each word to a contin- uous representation. These models cannot handle new words and are not effective for languages with rich morphology. One solution is to use smaller units, such as bytes, characters, or word pieces learned from word tokens (Wu et al., 2016; Sen- nrich et al., 2016). However, this approach has to process longer sequences than word-level alter- natives and may increase modeling and computa- tional challenges (Cherry et al., 2018). This pa- per focuses on another way based on word-level models in which a character encoder is used on top of characters of each word to calculate the word representation. They are often referred to as character-aware NLMs (CNLMs) (Ling et al., 2015; Kim et al., 2016; Vania and Lopez, 2017; Gerz et al., 2018; Assylbekov and Takhanov, 2018; Feng et al., 2019). However, the character encoders in CNLMs often show over-representation of or- thography rather than semantic meaning in the re- sulting word embedding despite the fact that train- ing word-based NLMs usually helps learn such semantic meaning (Kim et al., 2016; Vania and Lopez, 2017; Assylbekov and Takhanov, 2018). For example, in CNLMs, the nearest neighbors of the word ‘his’ with cosine similarity are ‘hhs’ and ‘this’ while ‘my’ is far from the nearest neighbors. To alleviate the over-representation issue in CNLMs, we propose to directly force the charac- ter encoder to produce the word-based embedding in a warm-up step before the training of CNLMs starts. Specifically, the character encoder encodes an input word, and the encoded embedding will be forced to be close to the embeddings of its sur- rounding words and far from the word embeddings of negative samples. Unlike the dynamically con- structed embedding of the input word, the embed- dings of surrounding words and negative samples are word-based, and thus these embeddings will not be biased to surface forms. The above method is similar to the architecture of the Skip-gram model (Mikolov et al., 2013) with …

Content truncated to 500 words