LSTMs Compose—and Learn—Bottom-Up

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2797–2809 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2797 LSTMS Compose—and Learn—Bottom-Up Naomi Saphra University of Edinburgh n.saphra@ed.ac.uk Adam Lopez University of Edinburgh alopez@inf.ed.ac.uk Abstract Recent work in NLP shows that LSTM lan- guage models capture hierarchical structure in language data. In contrast to existing work, we consider the learning process that leads to their compositional behavior. For a closer look at how an LSTM’s sequential representations are composed hierarchically, we present a related measure of Decompositional Interdependence (DI) between word meanings in an LSTM, based on their gate interactions. We connect this measure to syntax with experiments on English language data, where DI is higher on pairs of words with lower syntactic distance. To explore the inductive biases that cause these compositional representations to arise during training, we conduct simple experiments on synthetic data. These synthetic experiments support a specific hypothesis about how hi- erarchical structures are discovered over the course of training: that LSTM constituent rep- resentations are learned bottom-up, relying on effective representations of their shorter chil- dren, rather than learning the longer-range re- lations independently from children. 1 Introduction For years the LSTM dominated language architec- tures. It remains a popular architecture in NLP, and unlike Transformer-based models, it can be trained on small corpora (Tran et al., 2018).1 Ab- nar et al. (2020) even found that the recurrent in- ductive biases behind the LSTM’s success are so essential that distilling from them can improve the performance of fully attentional models. However, the reasons behind the LSTM’s effectiveness in language domains remain poorly understood. 1As evidence of the ongoing popularity of LSTMs in NLP, a Google Scholar search restricted to aclweb.org since 2019 finds 191 citations to the original LSTM paper (Hochre- iter and Schmidhuber, 1997) and 242 citations to the original Transformer paper (Vaswani et al., 2017). A Transformer can encode syntax using at- tention (Hewitt and Manning, 2019), and some LSTM variants explicitly encode syntax (Bowman et al., 2016; Dyer et al., 2016). So, the success of these models is partly explained by their abil- ity to model syntactic relationships when predict- ing a word. By contrast, an LSTM simply scans a sentence from left to right, accumulating mean- ing into a hidden representation one word at a time, and using that representation to summarize the entire preceding sequence when predicting the next word. Yet we have extensive evidence that trained LSTMs are also sensitive to syntax. For example, they can recall more history in natural language data than in similarly Zipfian-distributed n-gram data, implying that they exploit linguis- tic structure in long-distance dependencies (Liu et al., 2018). Their internal representations appear to encode constituency (Blevins et al., 2018; Hup- kes and Zuidema, 2018) and syntactic agreement (Lakretz et al., 2019; Gulordava et al., 2018). In this paper, we consider how such representations are learned, and what kind of inductive bias sup- ports them. To understand how LSTMs exploit syntax, we …

Content truncated to 500 words