Revisiting Recurrent Networks for Paraphrastic Sentence Embeddings

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 2078–2088 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1190 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 2078–2088 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1190 Revisiting Recurrent Networks for Paraphrastic Sentence Embeddings John Wieting Kevin Gimpel Toyota Technological Institute at Chicago, Chicago, IL, 60637, USA {jwieting,kgimpel}@ttic.edu Abstract We consider the problem of learn- ing general-purpose, paraphrastic sen- tence embeddings, revisiting the setting of Wieting et al. (2016b). While they found LSTM recurrent networks to un- derperform word averaging, we present several developments that together pro- duce the opposite conclusion. These in- clude training on sentence pairs rather than phrase pairs, averaging states to repre- sent sequences, and regularizing aggres- sively. These improve LSTMs in both transfer learning and supervised settings. We also introduce a new recurrent archi- tecture, the GATED RECURRENT AVER- AGING NETWORK, that is inspired by av- eraging and LSTMs while outperforming them both. We analyze our learned mod- els, finding evidence of preferences for particular parts of speech and dependency relations. 1 1 Introduction Modeling sentential compositionality is a funda- mental aspect of natural language semantics. Re- searchers have proposed a broad range of com- positional functional architectures (Mitchell and Lapata, 2008; Socher et al., 2011; Kalchbrenner et al., 2014) and evaluated them on a large vari- ety of applications. Our goal is to learn a general- purpose sentence embedding function that can be used unmodified for measuring semantic textual similarity (STS) (Agirre et al., 2012) and can also serve as a useful initialization for downstream tasks. We wish to learn this embedding function 1Trained models and code are available at http:// ttic.uchicago.edu/˜wieting. such that sentences with high semantic similar- ity have high cosine similarity in the embedding space. In particular, we focus on the setting of Wieting et al. (2016b), in which models are trained on noisy paraphrase pairs and evaluated on both STS and supervised semantic tasks. Surprisingly, Wieting et al. found that sim- ple embedding functions—those based on aver- aging word vectors—outperform more powerful architectures based on long short-term memory (LSTM) (Hochreiter and Schmidhuber, 1997). In this paper, we revisit their experimental setting and present several techniques that together im- prove the performance of the LSTM to be superior to word averaging. We first change data sources: rather than train on noisy phrase pairs from the Paraphrase Database (PPDB; Ganitkevitch et al., 2013), we use noisy sentence pairs obtained automatically by aligning Simple English to standard English Wikipedia (Coster and Kauchak, 2011). Even though this data was intended for use by text sim- plification systems, we find it to be efficient and ef- fective for learning sentence embeddings, outper- forming much larger sets of examples from PPDB. We then show how we can modify and regular- ize the LSTM to further improve its performance. The main modification is to simply average the hidden …

Content truncated to 500 words