Revisiting the Hierarchical Multiscale LSTM

Proceedings of the 27th International Conference on Computational Linguistics, pages 3215–3227 Santa Fe, New Mexico, USA, August 20-26, 2018. 3215 Revisiting the Hierarchical Multiscale LSTM ´Akos K´ad´ar Tilburg University a.kadar@uvt.nl Marc-Alexandre Cˆot´e Microsoft Research Montreal macote@microsoft.com Grzegorz Chrupa�la Tilburg University g.chrupala@uvt.nl Afra Alishahi Tilburg University a.alishahi@uvt.nl Abstract Hierarchical Multiscale LSTM (Chung et al., 2016a) is a state-of-the-art language model that learns interpretable structure from character-level input. Such models can pro- vide fertile ground for (cognitive) computational linguistics studies. However, the high complexity of the architecture, training procedure and implementations might hinder its applicability. We provide a detailed reproduction and ablation study of the architecture, shedding light on some of the potential caveats of re-purposing complex deep-learning architectures. We further show that simplifying certain aspects of the architecture can in fact improve its performance. We also investigate the linguistic units (segments) learned by various levels of the model, and argue that their quality does not correlate with the overall performance of the model on language modeling. 1 Introduction Verifying and reproducing claims published in scientific articles is an essential part of building a solid foundation for future research. As such, reproduction has a long history in many scientific fields (Willett et al., 1985; Venables et al., 1993; Waltemath et al., 2011). Recent large-scale studies, however, raise concerns about the reproducibility in a variety of areas and the potential effect of this crisis (Baker, 2016). A reproduction study by the Open Science Collaboration (2015) estimates that only 40% of research in psychology is reproducible, while Begley and Ellis (2012) end up confirming only 11% of preclinical cancer studies. The latter work makes an important link between the low reproducibility rates and the notoriously low impact of preclinical cancer research on clinical practice (Hutchinson and Kirk, 2011). Our work is motivated by a similar concern, specifically the applicability of complex deep learning architectures for computational (cognitive) linguistics research. State of the art sys- tems often employ complex architectures which integrate various design features and use many optimization techniques. Because the focus is on boosting the final performance on a given task, often little effort is put into understanding where the power of the system comes from. This makes these models much harder to adapt for new tasks or domains. In our view, it is essential not only to be able to reproduce reported results, but also to understand the contribution of various design features through systematic ablation experiments. The higher performance brought by modern neural network architectures often comes at the cost of our understanding of the representations and structural information the system learns. However, for models to be generalizable to new domains, it is important to move towards ana- lyzing such structural representations, and investigating their impact on the final performance of the model. In the current study, we examine the reproducibility of a language model with the ability to learn explicit linguistic structure: the Hierarchical Multiscale Recurrent Neural Network (HMLSTM) model. This architecture was introduced by Chung et al. (2016a) and …

Content truncated to 500 words