SentEval: An Evaluation Toolkit for Universal Sentence Representations

SentEval: An Evaluation Toolkit for Universal Sentence Representations Alexis Conneau∗ and Douwe Kiela Facebook Artificial Intelligence Research {aconneau,dkiela}@fb.com Abstract We introduce SentEval, a toolkit for evaluating the quality of universal sentence representations. SentEval encompasses a variety of tasks, including binary and multi-class classification, natural language inference and sentence similarity. The set of tasks was selected based on what appears to be the community consensus regarding the appropriate evaluations for universal sentence representations. The toolkit comes with scripts to download and preprocess datasets, and an easy interface to evaluate sentence encoders. The aim is to provide a fairer, less cumbersome and more centralized way for evaluating sentence representations. Keywords: representation learning, evaluation 1. Introduction Following the recent word embedding upheaval, one of NLP’s next challenges has become the hunt for univer- sal general-purpose sentence representations. What distin- guishes these representations, or embeddings, is that they are not necessarily trained to perform well on one specific task. Rather, their value lies in their transferability, i.e., their ability to capture information that can be of use in any kind of system or pipeline, on a variety of tasks. Word embeddings are particularly useful in cases where there is limited training data, leading to sparsity and poor vocabulary coverage, which in turn lead to poor generaliza- tion capabilities. Similarly, sentence embeddings (which are often built on top of word embeddings) can be used to further increase generalization capabilities, composing unseen combinations of words and encoding grammatical constructions that are not present in the task-specific train- ing data. Hence, high-quality universal sentence represen- tations are highly desirable for a variety of downstream NLP tasks. The evaluation of general-purpose word and sentence em- beddings has been problematic (Chiu et al., 2016; Faruqui et al., 2016), leading to much discussion about the best way to go about it1. On the one hand, people have measured performance on intrinsic evaluations, e.g. of human judg- ments of word or sentence similarity ratings (Agirre et al., 2012; Hill et al., 2016b) or of word associations (Vuli´c et al., 2017). On the other hand, it has been argued that the focus should be on downstream tasks where these repre- sentations would actually be applied (Ettinger et al., 2016; Nayak et al., 2016). In the case of sentence representa- tions, there is a wide variety of evaluations available, many from before the “embedding era”, that can be used to as- sess representational quality on that particular task. Over the years, something of a consensus has been established, mostly based on the evaluations in seminal papers such as SkipThought (Kiros et al., 2015), concerning what evalu- ations to use. Recent works in which various alternative sentence encoders are compared use a similar set of tasks ∗LIUM, Universit´e Le Mans 1See also recent workshops on evaluating representations for NLP, e.g. RepEval: https://repeval2017.github.io/ (Hill et al., 2016a; Conneau et al., 2017). Implementing pipelines for this large set of evaluations, each with its own peculiarities, is cumbersome and in- duces unnecessary wheel reinventions. Another well- known …

Content truncated to 500 words