Closing Brackets with Recurrent Neural Networks

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 232–239 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 232 Closing Brackets with Recurrent Neural Networks Natalia Skachkova∗, Thomas A. Trost∗, Adam Kusmirek and Dietrich Klakow Spoken Language Systems Saarland University Saarland Informatics Campus 66123 Saarbr¨ucken, Germany {s9naskac@stud,{thomas.trost,akusmirek,dietrich.klakow}@lsv}.uni-saarland.de Abstract Many natural and formal languages contain words or symbols that require a matching counterpart for making an expression well- formed. The combination of opening and clos- ing brackets is a typical example of such a construction. Due to their commonness, the ability to follow such rules is important for language modeling. Currently, recurrent neu- ral networks (RNNs) are extensively used for this task. We investigate whether they are capable of learning the rules of opening and closing brackets by applying them to syn- thetic Dyck languages that consist of differ- ent types of brackets. We provide an anal- ysis of the statistical properties of these lan- guages as a baseline and show strengths and limits of Elman-RNNs, GRUs and LSTMs in experiments on random samples of these lan- guages. In terms of perplexity and prediction accuracy, the RNNs get close to the theoretical baseline in most cases. 1 Introduction Brackets are a challenge for language models. They regularly appear in texts, they typically pro- duce long-range dependencies, and a failure to properly close them is readily recognized by a hu- man evaluator as a severe error (Shen et al., 2017). Beyond the syntactical level, many natural lan- guages exhibit brackets-like structures. For ex- ample, the German language is infamous for its convoluted sentences with verb-particle construc- tions, in which words from the beginning have to be properly closed at the end (Dewell, 2011; M¨uller et al., 2015). In this paper we present a dedicated study of the capability of Elman-RNNs, GRUs and LSTMs to model expressions with brackets and properly ∗Both authors contributed equally. close them. Towards this end, we conduct exper- iments on Dyck languages, which consist of bal- anced bracket expressions. 1.1 Related Work Synthetic datasets and formal languages have long been used for checking the ability of RNNs to capture a particular feature. For example, Elman (1990), Das et al. (1992), or Gers and Schmidhu- ber (2001) already did such investigations. Recent work in this direction was done by Weiss et al. (2017, 2018). More specifically, the interplay of RNNs with certain grammatical constructs, brackets and Dyck languages has been the subject of several studies. Karpathy et al. (2016) show that RNNs are capa- ble of capturing bracket structures on real-world datasets. Linzen et al. (2016) study the applica- tion of LSTMs to certain grammatical phenomena. RNNs and their variants have been used for recog- nizing Dyck words (Kalinke and Lehmann, 1998; Deleu and Dureau, 2016). Li et al. (2017) evaluate their nonlinear weighted finite automata model on a Dyck language. Most recently, Bernardy (2018) conducted a very similar study to ours on Dyck languages with a slightly different focus. 1.2 …

Content truncated to 500 words