Syllable-aware Neural Language Models: A Failure to Beat Character-aware Ones

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1866–1872 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Syllable-aware Neural Language Models: A Failure to Beat Character-aware Ones Zhenisbek Assylbekov School of Science and Technology Nazarbayev University zhassylbekov@nu.edu.kz Rustem Takhanov School of Science and Technology Nazarbayev University rustem.takhanov@nu.edu.kz Bagdat Myrzakhmetov National Laboratory Astana Nazarbayev University bagdat.myrzakhmetov@nu.edu.kz Jonathan N. Washington Linguistics Department Swarthmore College jonathan.washington@swarthmore.edu Abstract Syllabification does not seem to im- prove word-level RNN language model- ing quality when compared to character- based segmentation. However, our best syllable-aware language model, achieving performance comparable to the competi- tive character-aware model, has 18%–33% fewer parameters and is trained 1.2–2.2 times faster. 1 Introduction Recent advances in neural language modeling (NLM) are connected with character-aware mod- els (Kim et al., 2016; Ling et al., 2015b; Verwimp et al., 2017). This is a promising approach, and we propose the following direction related to it: We would like to make sure that in the pursuit of the most fine-grained representations one has not missed possible intermediate ways of segmenta- tion, e.g., by syllables. Syllables, in our opinion, are better supported as linguistic units of language than single characters. In most languages, words can be naturally split into syllables: ES: el par-la-men-to a-po-yó la en-mien-da RU: пар-ла-мент под-дер-жал по-прав-ку (EN: the parliament supported the amendment) Based on this observation, we attempted to de- termine whether syllable-aware NLM has any ad- vantages over character-aware NLM. We exper- imented with a variety of models but could not find any evidence to support this hypothesis: split- ting words into syllables does not seem to improve the language modeling quality when compared to splitting into characters. However, there are some positive findings: while our best syllable-aware language model achieves performance comparable to the competitive character-aware model, it has 18%–33% fewer parameters and is 1.2–2.2 times faster to train. 2 Related Work Much research has been done on subword-level and subword-aware1 neural language modeling when subwords are characters (Ling et al., 2015b; Kim et al., 2016; Verwimp et al., 2017) or mor- phemes (Botha and Blunsom, 2014; Qiu et al., 2014; Cotterell and Schütze, 2015). However, not much work has been done on syllable-level or syllable-aware NLM. Mikolov et al. (2012) show that subword-level language models outperform character-level ones.2 They keep the most frequent words untouched and split all other words into syllable-like units. Our approach differs mainly in the following aspects: we make predictions at the word level, use a more linguistically sound syllab- ification algorithm, and consider a variety of more advanced neural architectures. We have recently come across a concurrent paper (Vania and Lopez, 2017) where the au- thors systematically compare different subword units (characters, character trigrams, BPE (Sen- nrich et al., 2016), morphemes) and different rep- resentation models (CNN, Bi-LSTM, summation) on languages with various morphological typol- ogy. However, they do not consider syllables, and they experiment with relatively small models on small data sets (0.6M–1.4M tokens). 3 Syllable-aware word embeddings …

Content truncated to 500 words