Probing sentence embeddings for structure-dependent tense

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 334–336 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 334 Probing sentence embeddings for structure-dependent tense Geoff Bacon Department of Linguistics UC Berkeley bacon@berkeley.edu Terry Regier Department of Linguistics and Cognitive Science Program UC Berkeley terry.regier@berkeley.edu 1 Introduction Learning universal sentence representations which accurately model sentential semantic content is a current goal of natural language processing re- search (Subramanian et al., 2018; Conneau et al., 2017; Wieting et al., 2016; Kiros et al., 2015). A prominent and successful approach is to train re- current neural networks (RNNs) to encode sen- tences into fixed length vectors (Conneau et al., 2018; Nie et al., 2017). Many core linguistic phenomena that one would like to model in uni- versal sentence representations depend on syn- tactic structure (Chomsky, 1965; Everaert et al., 2015). Despite the fact that RNNs do not have explicit syntactic structural representations, there is some evidence that RNNs can approximate such structure-dependent phenomena under cer- tain conditions (Gulordava et al., 2018; McCoy et al., 2018; Linzen et al., 2016; Bowman et al., 2015), in addition to their widespread success in practical tasks. In this work, we assess RNNs’ ability to learn the structure-dependent phenomenon of main clause tense. To test whether sentence represen- tations derived from RNNs capture main clause tense, we attempt to predict the tense from the rep- resentation. This approach is called probing, and was introduced by Ettinger et al. (2016) and sub- sequently used by Adi et al. (2017) and others. Conneau et al. (2018) probed English sentence representations from various RNN architectures for main clause tense and concluded that these architectures, along with a bag-of-vectors (BoV) baseline, capture tense very well (84-91% accu- racy). However, this result was based on a test set in which the tense category (i.e. past or present) to be predicted was the most common tense cate- gory in the sentence for 95.2% of sentences. The high performance of the BoV model on this test set is not entirely surprising, given that K¨ohn (2015, 2016) showed a wide variety of word embed- ding models capture tense at the word level very well. The high performance of the RNN mod- els is not strong evidence that they are sensitive to the structure-dependence of main clause tense. As suggested by Linzen et al. (2016), these models may be learning a flawed heuristic that only works in grammatically simple examples. Our goal is to determine whether RNNs learn to perform structure-dependent computation or whether they merely learn practical heuristics. To do this, we extend the experimental setup of Adi et al. (2017), which has a two step nature. First, we train autoencoders for English, Span- ish, French and Italian where both the encoder and decoder are either Simple Recurrent Networks (SRNs, Elman, 1990) or Long Short-Term Mem- ory networks (LSTMs, Hochreiter and Schmidhu- ber, 1997). Second, we use the trained encoder to obtain sentence representations and probe those …

Content truncated to 500 words