Learning Paraphrastic Sentence Embeddings from Back-Translated Bitext

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 274–285 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Learning Paraphrastic Sentence Embeddings from Back-Translated Bitext John Wieting1 Jonathan Mallinson2 Kevin Gimpel1 1Toyota Technological Institute at Chicago, Chicago, IL, 60637, USA 2School of Informatics, University of Edinburgh, Edinburgh, UK {jwieting,kgimpel}@ttic.edu, j.mallinson@ed.ac.uk Abstract We consider the problem of learning general-purpose, paraphrastic sentence embeddings in the setting of Wieting et al. (2016b). We use neural machine trans- lation to generate sentential paraphrases via back-translation of bilingual sentence pairs. We evaluate the paraphrase pairs by their ability to serve as training data for learning paraphrastic sentence embed- dings. We find that the data quality is stronger than prior work based on bitext and on par with manually-written English paraphrase pairs, with the advantage that our approach can scale up to generate large training sets for many languages and do- mains. We experiment with several lan- guage pairs and data sources, and develop a variety of data filtering techniques. In the process, we explore how neural machine translation output differs from human- written sentences, finding clear differences in length, the amount of repetition, and the use of rare words.1 1 Introduction Pretrained word embeddings have received a great deal of attention from the research community, but there is much less work on developing pretrained embeddings for sentences. Here we target sen- tence embeddings that are “paraphrastic” in the sense that two sentences with similar meanings are close in the embedding space. Wieting et al. (2016b) developed paraphrastic sentence embed- dings that are useful for semantic textual similar- ity tasks and can also be used as initialization for supervised semantic tasks. 1Generated paraphrases and code are available at http: //ttic.uchicago.edu/˜wieting. R: We understand that has already commenced, but there is a long way to go. T: This situation has already commenced, but much still needs to be done. R: The restaurant is closed on Sundays. No breakfast is available on Sunday mornings. T: The restaurant stays closed Sundays so no breakfast is served these days. R: Improved central bank policy is another huge factor. T: Another crucial factor is the improved policy of the central banks. Table 1: Illustrative examples of references (R) paired with back-translations (T). To learn their sentence embeddings, Wieting et al. used the Paraphrase Database (PPDB) (Gan- itkevitch et al., 2013). PPDB contains a large set of paraphrastic textual fragments extracted auto- matically from bilingual text (“bitext”), which is readily available for languages and domains. Ver- sions of PPDB have been released for several lan- guages (Ganitkevitch and Callison-Burch, 2014). However, more recent work has shown that the fragmental nature of PPDB’s pairs can be problematic, especially for recurrent net- works (Wieting and Gimpel, 2017). Better per- formance can be achieved with a smaller set of sentence pairs derived from aligning Simple En- glish and standard English Wikipedia (Coster and Kauchak, 2011). While effective, this type of data is inherently limited in size and scope, and …

Content truncated to 500 words