Query and Output: Generating Words by Querying Distributed Word Representations for Paraphrase Generation

Proceedings of NAACL-HLT 2018, pages 196–206 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Query and Output: Generating Words by Querying Distributed Word Representations for Paraphrase Generation Shuming Ma1, Xu Sun1,2, Wei Li1, Sujian Li1, Wenjie Li3, Xuancheng Ren1 1MOE Key Lab of Computational Linguistics, School of EECS, Peking University 2Deep Learning Lab, Beijing Institute of Big Data Research, Peking University 3Department of Computing, The Hong Kong Polytechnic University {shumingma, xusun, liweitj47, lisujian, renxc}@pku.edu.cn cswjli@comp.polyu.edu.hk Abstract Most recent approaches use the sequence- to-sequence model for paraphrase genera- tion. The existing sequence-to-sequence model tends to memorize the words and the patterns in the training dataset instead of learn- ing the meaning of the words. Therefore, the generated sentences are often grammati- cally correct but semantically improper. In this work, we introduce a novel model based on the encoder-decoder framework, called Word Embedding Attention Network (WEAN). Our proposed model generates the words by query- ing distributed word representations (i.e. neu- ral word embeddings), hoping to capturing the meaning of the according words. Following previous work, we evaluate our model on two paraphrase-oriented tasks, namely text sim- plification and short text abstractive summa- rization. Experimental results show that our model outperforms the sequence-to-sequence baseline by the BLEU score of 6.3 and 5.5 on two English text simplification datasets, and the ROUGE-2 F1 score of 5.7 on a Chi- nese summarization dataset. Moreover, our model achieves state-of-the-art performances on these three benchmark datasets.1 1 Introduction Paraphrase is a restatement of the meaning of a text using other words. Many natural language generation tasks are paraphrase-orientated, such as text simplification and short text summariza- tion. Text simplification is to make the text easier to read and understand, especially for poor read- ers, while short text summarization is to generate a brief sentence to describe the short texts (e.g. posts on the social media). Most recent approaches use sequence-to-sequence model for paraphrase gen- eration (Prakash et al., 2016; Cao et al., 2017). It 1The code is available at https://github.com/ lancopku/WEAN compresses the source text information into dense vectors with the neural encoder, and the neural decoder generates the target text using the com- pressed vectors. Although neural network models achieve suc- cess in paraphrase generation, there are still two major problems. One of the problem is that the ex- isting sequence-to-sequence model tends to mem- orize the words and the patterns in the training dataset instead of the meaning of the words. The main reason is that the word generator (i.e. the output layer of the decoder) does not model the semantic information. The word generator, which consists of a linear transformation and a softmax operation, converts the Recurrent Neural Network (RNN) output from a small dimension (e.g. 500) to a much larger dimension (e.g. 50,000 words in the vocabulary), where each dimension repre- sents the score of each word. The latent assump- tion of the word generator is that each word is in- dependent and the score is irrelevant to …

Content truncated to 500 words