Simple and Effective Paraphrastic Similarity from Parallel Translations

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4602–4608 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4602 Simple and Effective Paraphrastic Similarity from Parallel Translations John Wieting1, Kevin Gimpel2, Graham Neubig1, and Taylor Berg-Kirkpatrick3 1Carnegie Mellon University, Pittsburgh, PA, 15213, USA 2Toyota Technological Institute at Chicago, Chicago, IL, 60637, USA 3University of California San Diego, San Diego, CA, 92093, USA {jwieting,gneubig}@cs.cmu.edu, kgimpel@ttic.edu, tberg@eng.ucsd.edu Abstract We present a model and methodology for learning paraphrastic sentence embeddings directly from bitext, removing the time- consuming intermediate step of creating para- phrase corpora. Further, we show that the re- sulting model can be applied to cross-lingual tasks where it both outperforms and is orders of magnitude faster than more complex state- of-the-art baselines.1 1 Introduction Measuring sentence similarity is a core task in semantics (Cer et al., 2017), and prior work has achieved strong results by training similarity mod- els on datasets of paraphrase pairs (Dolan et al., 2004). However, such datasets are not produced naturally at scale and therefore must be created ei- ther through costly manual annotation or by lever- aging natural annotation in specific domains, like Simple English Wikipedia (Coster and Kauchak, 2011) or Twitter (Lan et al., 2017). One of the most promising approaches for in- ducing paraphrase datasets is via manipulation of large bilingual corpora. Examples include bilin- gual pivoting over phrases (Callison-Burch et al., 2006; Ganitkevitch et al., 2013), and automatic translation of one side of the bitext (Wieting et al., 2017; Wieting and Gimpel, 2018; Hu et al., 2019). However, this is costly – Wieting and Gimpel (2018) report their large-scale database of sen- tential paraphrases required 10,000 GPU hours to generate. In this paper, we propose a method that trains highly performant sentence embeddings (Pham et al., 2015; Hill et al., 2016; Pagliardini et al., 2017; McCann et al., 2017; Conneau et al., 2017) directly on bitext, obviating these intermediate 1Code and data to replicate results are available at https://www.cs.cmu.edu/˜jwieting. steps and avoiding the noise and error propaga- tion from automatic dataset preparation methods. This approach eases data collection, since bitext occurs naturally more often than paraphrase data and, further, has the additional benefit of creat- ing cross-lingual representations that are useful for tasks such as mining or filtering parallel data and cross-lingual retrieval. Most previous work for cross-lingual repre- sentations has focused on models based on en- coders from neural machine translation (Espana- Bonet et al., 2017; Schwenk and Douze, 2017; Schwenk, 2018) or deep architectures using a con- trastive loss (Gr´egoire and Langlais, 2018; Guo et al., 2018; Chidambaram et al., 2018). How- ever, the paraphrastic sentence embedding liter- ature has observed that simple models such as pooling word embeddings generalize significantly better than complex architectures (Wieting et al., 2016b). Here, we find a similar effect in the bilingual setting. We propose a simple model that not only produces state-of-the-art monolin- gual and bilingual sentence representations, but also encode sentences hundreds of times …

Content truncated to 500 words