Learning Semantic Sentence Embeddings using Sequential Pair-wise Discriminator

Proceedings of the 27th International Conference on Computational Linguistics, pages 2715–2729 Santa Fe, New Mexico, USA, August 20-26, 2018. 2715 Learning Semantic Sentence Embeddings using Pair-wise Discriminator Badri N. Patro∗ Vinod K. Kurmi∗ Sandeep Kumar∗ Indian Institute of Technology, KanpurVinay P. Namboodiri {badri,vinodkk,sandepkr,vinaypn}@iitk.ac.in Abstract In this paper, we propose a method for obtaining sentence-level embeddings. While the problem of securing word-level embeddings is very well studied, we propose a novel method for obtaining sentence-level embeddings. This is obtained by a simple method in the context of solving the paraphrase generation task. If we use a sequential encoder-decoder model for generating paraphrase, we would like the generated paraphrase to be semantically close to the original sentence. One way to ensure this is by adding constraints for true paraphrase embeddings to be close and unrelated paraphrase candidate sentence embeddings to be far. This is ensured by using a sequential pair-wise discriminator that shares weights with the encoder that is trained with a suitable loss function. Our loss function penalizes paraphrase sentence embedding distances from being too large. This loss is used in combination with a sequential encoder-decoder network. We also validated our method by evaluating the obtained embeddings for a sentiment analysis task. The proposed method results in semantic embeddings and outperforms the state-of-the-art on the paraphrase generation and sentiment analysis task on standard datasets. These results are also shown to be statistically significant. 1 Introduction The problem of obtaining a semantic embedding for a sentence that ensures that the related sentences are closer and unrelated sentences are farther lies at the core of understanding languages. This would be relevant for a wide variety of machine reading comprehension and related tasks such as sentiment analysis. Towards this problem, we propose a supervised method that uses a sequential encoder-decoder framework for paraphrase generation. The task of generating paraphrases is closely related to the task of obtaining semantic sentence embeddings. In our approach, we aim to ensure that the generated paraphrase embedding should be close to the true corresponding sentence and far from unrelated sentences. The embeddings so obtained help us to obtain state-of-the-art results for paraphrase generation task. Our model consists of a sequential encoder-decoder that is further trained using a pairwise discrim- inator. The encoder-decoder architecture has been widely used for machine translation and machine comprehension tasks. In general, the model ensures a ‘local’ loss that is incurred for each recurrent unit cell. It only ensures that a particular word token is present at an appropriate place. This, however, does not imply that the whole sentence is correctly generated. To ensure that the whole sentence is correctly encoded, we make further use of a pair-wise discriminator that encodes the whole sentence and obtains an embedding for it. We further ensure that this is close to the desired ground-truth embeddings while being far from other (sentences in the corpus) embeddings. This model thus provides a ‘global’ loss that ensures the sentence embedding as a whole is close to other semantically related sentence embeddings. …

Content truncated to 500 words