Implicit n-grams Induced by Recurrence

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1624 - 1639 July 10-15, 2022 ©2022 Association for Computational Linguistics Implicit N-grams Induced by Recurrence Xiaobing Sun and Wei Lu StatNLP Research Group Singapore University of Technology and Design xiaobing_sun@mymail.sutd.edu.sg, luwei@sutd.edu.sg Abstract Although self-attention based models such as Transformers have achieved remarkable suc- cesses on natural language processing (NLP) tasks, recent studies reveal that they have limitations on modeling sequential transfor- mations (Hahn, 2020), which may prompt re-examinations of recurrent neural networks (RNNs) that demonstrated impressive results on handling sequential data. Despite many prior attempts to interpret RNNs, their internal mechanisms have not been fully understood, and the question on how exactly they capture sequential features remains largely unclear. In this work, we present a study that shows there actually exist some explainable components that reside within the hidden states, which are reminiscent of the classical n-grams features. We evaluated such extracted explainable fea- tures from trained RNNs on downstream sen- timent analysis tasks and found they could be used to model interesting linguistic phenomena such as negation and intensification. Further- more, we examined the efficacy of using such n-gram components alone as encoders on tasks such as sentiment analysis and language model- ing, revealing they could be playing important roles in contributing to the overall performance of RNNs. We hope our findings could add in- terpretability to RNN architectures, and also provide inspirations for proposing new archi- tectures for sequential data. 1 Introduction Modern recurrent neural networks (RNNs), includ- ing Long Short-Term Memory (LSTM) (Hochre- iter and Schmidhuber, 1997) and Gated Recurrent Units (GRU) (Cho et al., 2014), have demonstrated impressive results on tasks involving sequential data. They have proven to be capable of modeling formal languages (Weiss et al., 2018; Merrill, 2019; Merrill et al., 2020) and capturing structural fea- tures (Li et al., 2015a,b, 2016; Linzen et al., 2016; Belinkov et al., 2017; Liu et al., 2019) on NLP tasks. Although Transformers (Vaswani et al., 2017) have achieved remarkable performances on Figure 1: An RNN hidden state may encode a linear combination of all the n-grams ending at the current time step. NLP tasks such as machine translation, it is argued that they may have limitations on modeling hierar- chical structure (Tran et al., 2018; Hahn, 2020) and cannot handle functions requiring sequential pro- cessing of input well (Dehghani et al., 2019; Hao et al., 2019; Bhattamishra et al., 2020; Yao et al., 2021). Furthermore, a recent work shows that com- bining recurrence and attention (Lei, 2021) can result in strong modeling capacity. Another recent work incorporating recurrent cells into Transform- ers (Hutchins et al., 2022) substantially improved performance on language modeling involving very long sequences, prompting re-investigations of RNNs. On the other hand, it was observed in prior work that RNNs were able to capture linguistic phenomena such as negation and intensification (Li et al., 2016), but the question why they could achieve so still …

Content truncated to 500 words