Syntactically-Informed Unsupervised Paraphrasing with Non-Parallel Data

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2594–2604 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 2594 Syntactically-Informed Unsupervised Paraphrasing with Non-Parallel Data Erguang Yang1∗, Mingtong Liu1 , Deyi Xiong2∗, Yujie Zhang1, Yao Meng3 , Changjian Hu3 , Jinan Xu1 , Yufeng Chen1 1School of Computer and Information Technology, Beijing Jiaotong University, Beijing, China 2College of Intelligence and Computing, Tianjin University, Tianjin, China 3Lenovo Research AI Lab, Beijing, China {egyang,mingtongliu,yjzhang,jaxu,chenyf}@bjtu.edu.cn, dyxiong@tju.edu.cn, {yaomeng1,hucj1}@lenovo.com Abstract Previous works on syntactically controlled paraphrase generation heavily rely on large- scale parallel paraphrase data that are not eas- ily available for many languages and domains. In this paper, we take this research direc- tion to the extreme and investigate whether it is possible to learn syntactically controlled paraphrase generation with non-parallel data. We propose a syntactically-informed unsuper- vised paraphrasing model based on conditional variational auto-encoder (VAE) which can gen- erate texts in a specified syntactic structure. Particularly, we design a two-stage learning method to effectively train the model using non-parallel data. The conditional VAE is trained to reconstruct the input sentence ac- cording to the given input and its syntactic structure. Furthermore, to improve the syn- tactic controllability and semantic consistency of the pre-trained conditional VAE, we fine- tune it using syntax controlling and cycle re- construction learning objectives, and employ Gumbel-Softmax to combine these new learn- ing objectives. Experiment results demon- strate that the proposed model trained only on non-parallel data is capable of generating diverse paraphrases with specified structures. Additionally, we further validate the effective- ness of our method for generating syntactically adversarial examples on a sentiment analysis task. Source codes are available at https: //github.com/lanse-sir/sup. 1 Introduction Paraphrases are texts or passages conveying the same meaning but with different surface realization. Paraphrase generation (PG) is a key technology of automatically generating a restatement for a given text, which has the potential use in many down- stream tasks, such as question answering (Dong et al., 2017), machine translation (Zhou et al., 2019) and text summarization (Zhao et al., 2018). ∗Corresponding Authors. Recent years have witnessed that learning con- trollable paraphrase generation (CPG) with spec- ified styles is attracting intense research interests, e.g., satisfying particular syntactic templates (Iyyer et al., 2018) or exemplars (Chen et al., 2019; Ku- mar et al., 2020). As CPG can produce diverse paraphrases by exposing syntactic control, it can be also employed for adversarial example genera- tion (Iyyer et al., 2018). Existing syntactically controlled paraphrase net- works (Iyyer et al., 2018) rely on large paraphrase parallel data for training. Unfortunately, paraphrase parallel corpora are not easily available for many languages, and are expensive to build. Conversely, non-parallel data is much easier to find, and many languages with limited parallel data still possess a huge amount of non-parallel data. In this paper, we propose a Syntactically- informed Unsupervised Paraphrasing (SUP) frame- work based on conditional variational auto-encoder (VAE) to generate syntactic paraphrases with spec- ified syntactic skeletons, which does not require any parallel paraphrase data. …

Content truncated to 500 words