Reproducing and Regularizing the SCRN Model

Proceedings of the 27th International Conference on Computational Linguistics, pages 1705–1716 Santa Fe, New Mexico, USA, August 20-26, 2018. 1705 Reproducing and Regularizing the SCRN Model Olzhas Kabdolov, Zhenisbek Assylbekov, Rustem Takhanov School of Science and Technology, Nazarbayev University {olzhas.kabdolov, zhassylbekov, rustem.takhanov}@nu.edu.kz Abstract We reproduce the Structurally Constrained Recurrent Network (SCRN) model, and then regular- ize it using the existing widespread techniques, such as naïve dropout, variational dropout, and weight tying. We show that when regularized and optimized appropriately the SCRN model can achieve performance comparable with the ubiquitous LSTM model in language modeling task on English data, while outperforming it on non-English data. Title and Abstract in Russian Воспроизведение и регуляризация SCRN модели Мы воспроизводим структурно ограниченную рекуррентную сеть (SCRN), а затем добавляем регуляризацию, используя существующие широко распространенные методы, такие как исключение (дропаут), вариационное исключение и связка параметров. Мы показываем, что при правильной регуляризации и оптимизации показатели SCRN сопоставимы с показателями вездесущей LSTM в задаче языкового моделирования на английских текстах, а также превосходят их на неанглийских данных. 1 Introduction Recurrent neural networks (RNN) have demonstrated tremendous success in sequence modeling in gen- eral and in language modeling in particular. The most basic RNN (Elman, 1990) suffers from the problem of vanishing and exploding gradients (Bengio et al., 1994) and is hard to train efficiently. One of the most widespread and efficient alternatives to the basic RNN is the Long-Short Term Memory (LSTM) model (Hochreiter and Schmidhuber, 1997), which effectively addresses the problem of vanishing gradients. However, LSTM is a fairly complex model with excessive number of parameters and its inner function- ality is not obvious. This complexity has motivated some of the researchers to find more apparent and less complex alternatives. One of such alternative models is a Structurally Constrained Recurrent Net- work (SCRN) proposed by Mikolov et al. (2015). They encouraged some of the hidden units to change their state slowly by making part of the recurrent weight matrix close to identity, thus forming a kind of longer term memory and showed that their SCRN model can outperform the simple RNN and achieve the performance comparable with the LSTM under no regularization and small parameter budget. It is natural to try to regularize the SCRN model under larger budgets: Will it approach the performance of LSTM? Our experiments show that (1) under naïve dropout the SCRN demonstrates performance close to that of the LSTM, but (2) under variational dropout and weight tying the LSTM demonstrates better performance. 2 Related Work There has been several attempts on simplifying the ubiquitous LSTM model while not losing in perfor- mance. E.g., Ororbia II et al. (2017) introduced Delta-RNN architecture for which SCRN serves as a This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http:// creativecommons.org/licenses/by/4.0/. 1706 predecessor. They showed that, when regularized using naïve dropout (Zaremba et al., 2014), Delta- RNN performs comparably to LSTM and GRU (Chung et al., 2014). However, they did not compare to regularized SCRN, and they did not consider more …

Content truncated to 500 words