Unsupervised Paraphrasing without Translation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6033–6039 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6033 Unsupervised Paraphrasing without Translation Aurko Roy Google Research aurkor@google.com David Grangier Google Research grangier@google.com Abstract Paraphrasing exemplifies the ability to abstract semantic content from surface forms. Recent work on automatic paraphrasing is dominated by methods leveraging Machine Translation (MT) as an intermediate step. This contrasts with humans, who can paraphrase without be- ing bilingual. This work proposes to learn paraphrasing models from an unlabeled mono- lingual corpus only. To that end, we propose a residual variant of vector-quantized variational auto-encoder. We compare with MT-based approaches on paraphrase identification, generation, and training augmentation. Monolingual para- phrasing outperforms unsupervised translation in all settings. Comparisons with supervised translation are more mixed: monolingual para- phrasing is interesting for identification and augmentation; supervised translation is supe- rior for generation. 1 Introduction Many methods have been developed to generate paraphrases automatically (Madnani and J. Dorr, 2010). Approaches relying on Machine Transla- tion (MT) have proven popular due to the scarcity of labeled paraphrase pairs (Callison-Burch, 2007; Mallinson et al., 2017; Iyyer et al., 2018). Recent progress in MT with neural methods (Bahdanau et al., 2014; Vaswani et al., 2017) has popularized this latter strategy. Conceptually, translation is ap- pealing since it abstracts semantic content from its linguistic realization. For instance, assigning the same source sentence to multiple translators will result in a rich set of semantically close sen- tences (Callison-Burch, 2007). At the same time, bilingualism does not seem necessary to humans to generate paraphrases. This work evaluates if data in two languages is necessary for paraphrasing. We consider three settings: supervised translation (parallel bilin- gual data is used), unsupervised translation (non- parallel corpora in two languages are used) and monolingual (only unlabeled data in the para- phrasing language is used). Our comparison devises comparable encoder-decoder neural net- works for all three settings. While the litera- ture on supervised (Bahdanau et al., 2014; Cho et al., 2014; Vaswani et al., 2017) and unsuper- vised translation (Lample et al., 2018a; Artetxe et al., 2018; Lample et al., 2018b) offer solu- tions for the bilingual settings, monolingual neural paraphrase generation has not received the same attention. We consider discrete and continuous auto- encoders in an unlabeled monolingual setting, and contribute improvements in that context. We in- troduce a model based on Vector-Quantized Auto- Encoders, VQ-VAE (van den Oord et al., 2017), for generating paraphrases in a purely monolin- gual setting. Our model introduces residual con- nections parallel to the quantized bottleneck. This lets us interpolate from classical continuous auto- encoder (Vincent et al., 2010) to VQ-VAE. Com- pared to VQ-VAE, our architecture offers a better control over the decoder entropy and eases opti- mization. Compared to continuous auto-encoder, our method permits the generation of diverse, but semantically close sentences from an input sen- tence. We compare paraphrasing models over intrin- sic and extrinsic metrics. Our intrinsic evalua- tion evaluates paraphrase …

Content truncated to 500 words