Class-based LSTM Russian Language Model with Linguistic Information

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 2470–2474 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 2470 Class-based LSTM Russian Language Model with Linguistic Information Irina Kipyatkova1,2, Alexey Karpov1 1 St. Petersburg Institute for Informatics and Automation of the Russian Academy of Sciences (SPIIRAS), 2 St. Petersburg State University of Aerospace Instrumentation (SUAI) St. Petersburg, Russia {kipyatkova, karpov}@iias.spb.su Abstract In the paper, we present class-based LSTM Russian language models (LMs) with classes generated with the use of both word frequency and linguistic information data, obtained with the help of the “VisualSynan” software from the AOT project. We have created LSTM LMs with various numbers of classes and compared them with word-based LM and class-based LM with word2vec class generation in terms of perplexity, training time, and WER. In addition, we performed a linear interpolation of LSTM language models with the baseline 3-gram language model. The LSTM language models were used for very large vocabulary continuous Russian speech recognition at an N-best list rescoring stage. We achieved significant progress in training time reduction with only slight degradation in recognition accuracy comparing to the word-based LM. In addition, our LM with classes generated using linguistic information outperformed LM with classes generated using word2vec. We achieved WER of 14.94 % at our own speech corpus of continuous Russian speech that is 15 % relative reduction with respect to the baseline 3-gram model. Keywords: speech recognition, class-based language model, long short-term memory, Russian speech 1. Introduction One of the main state-of-the art approach to language modeling is application of neural networks (NNs). For language modeling, the usage of recurrent NNs (RNNs) is preferable because this type of NNs can store the whole context preceding the given word in contrast to feed- forward NNs which store a context of restricted length. One type of recurrent NNs is a long short-term memory (LSTM) network which contains special units called memory blocks. Each memory block is composed of a memory cell, which stores the temporal state of the network, and multiplicative units named gates (an input gate, an output gate, and a forget gate) controlling the information flow (Hochreiter and Schmidhuber, 1997). LSTM-based language models (LMs) can be used in speech recognition systems at N-best or lattice rescoring stage. It was shown in many papers that such models outperform standard n-gram models in term of both perplexity and word error rate (WER) (Sundermeyer et al., 2015; Kumar et al., 2017). Detail review of researches on application of RNN-based LNs for speech recognition is presented in (Kipyatkova and Karpov, 2016). There are only few researches on application of NNs for Russian language modeling. RNN LM for Russian was firstly used in (Vazhenina and Markov, 2013). RNN LM was trained on the text corpus containing 40M words with vocabulary size of about 100K words. An interpolation of the obtained model with the baseline 3-gram and factored LMs was carried out. Obtained LM was used for rescoring of 500-best …

Content truncated to 500 words