Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670–680 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Supervised Learning of Universal Sentence Representations from Natural Language Inference Data Alexis Conneau Facebook AI Research aconneau@fb.com Douwe Kiela Facebook AI Research dkiela@fb.com Holger Schwenk Facebook AI Research schwenk@fb.com Lo¨ıc Barrault LIUM, Universit´e Le Mans loic.barrault@univ-lemans.fr Antoine Bordes Facebook AI Research abordes@fb.com Abstract Many modern NLP systems rely on word embeddings, previously trained in an un- supervised manner on large corpora, as base features. Efforts to obtain embed- dings for larger chunks of text, such as sentences, have however not been so suc- cessful. Several attempts at learning unsu- pervised representations of sentences have not reached satisfactory enough perfor- mance to be widely adopted. In this paper, we show how universal sentence represen- tations trained using the supervised data of the Stanford Natural Language Inference datasets can consistently outperform unsu- pervised methods like SkipThought vec- tors (Kiros et al., 2015) on a wide range of transfer tasks. Much like how com- puter vision uses ImageNet to obtain fea- tures, which can then be transferred to other tasks, our work tends to indicate the suitability of natural language inference for transfer learning to other NLP tasks. Our encoder is publicly available1. 1 Introduction Distributed representations of words (or word em- beddings) (Bengio et al., 2003; Collobert et al., 2011; Mikolov et al., 2013; Pennington et al., 2014) have shown to provide useful features for various tasks in natural language processing and computer vision. While there seems to be a con- sensus concerning the usefulness of word embed- dings and how to learn them, this is not yet clear with regard to representations that carry the mean- ing of a full sentence. That is, how to capture the 1https://www.github.com/ facebookresearch/InferSent relationships among multiple words and phrases in a single vector remains an question to be solved. In this paper, we study the task of learning uni- versal representations of sentences, i.e., a sentence encoder model that is trained on a large corpus and subsequently transferred to other tasks. Two questions need to be solved in order to build such an encoder, namely: what is the preferable neu- ral network architecture; and how and on what task should such a network be trained. Follow- ing existing work on learning word embeddings, most current approaches consider learning sen- tence encoders in an unsupervised manner like SkipThought (Kiros et al., 2015) or FastSent (Hill et al., 2016). Here, we investigate whether su- pervised learning can be leveraged instead, tak- ing inspiration from previous results in computer vision, where many models are pretrained on the ImageNet (Deng et al., 2009) before being trans- ferred. We compare sentence embeddings trained on various supervised tasks, and show that sen- tence embeddings generated from models trained on a natural language inference (NLI) task reach the best results in terms of transfer accuracy. We hypothesize that the suitability of NLI as a train- ing task is …

Content truncated to 500 words