Composition of Sentence Embeddings: Lessons from Statistical Relational Learning

Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM), pages 33–43 Minneapolis, June 6–7, 2019. c⃝2019 Association for Computational Linguistics 33 Composition of Sentence Embeddings: Lessons from Statistical Relational Learning Damien Sileo3,1, Tim Van De Cruys2, Camille Pradel1, and Philippe Muller3 1Synapse D´eveloppement, Toulouse, France 2IRIT, CNRS, France 3IRIT, University of Toulouse, France damien.sileo@synapse-fr.com Abstract Various NLP problems – such as the predic- tion of sentence similarity, entailment, and dis- course relations – are all instances of the same general task: the modeling of semantic rela- tions between a pair of textual elements. A popular model for such problems is to embed sentences into fixed size vectors, and use com- position functions (e.g. concatenation or sum) of those vectors as features for the prediction. At the same time, composition of embeddings has been a main focus within the field of Statis- tical Relational Learning (SRL) whose goal is to predict relations between entities (typically from knowledge base triples). In this article, we show that previous work on relation pre- diction between texts implicitly uses composi- tions from baseline SRL models. We show that such compositions are not expressive enough for several tasks (e.g. natural language infer- ence). We build on recent SRL models to ad- dress textual relational problems, showing that they are more expressive, and can alleviate is- sues from simpler compositions. The resulting models significantly improve the state of the art in both transferable sentence representation learning and relation prediction. 1 Introduction Predicting relations between textual units is a widespread task, essential for discourse analy- sis, dialog systems, information retrieval, or para- phrase detection. Since relation prediction often requires a form of understanding, it can also be used as a proxy to learn transferable sentence rep- resentations. Several tasks that are useful to build sentence representations are derived directly from text structure, without human annotation: sen- tence order prediction (Logeswaran et al., 2016; Jernite et al., 2017), the prediction of previous and subsequent sentences (Kiros et al., 2015; Jer- nite et al., 2017), or the prediction of explicit dis- course markers between sentence pairs (Nie et al., 2017; Jernite et al., 2017). Human labeled re- lations between sentences can also be used for that purpose, e.g. inferential relations (Conneau et al., 2017). While most work on sentence simi- larity estimation, entailment detection, answer se- lection, or discourse relation prediction seemingly uses task-specific models, they all involve predict- ing whether a relation R holds between two sen- tences s1 and s2. This genericity has been noticed in the literature before (Baudiˇs et al., 2016) and it has been leveraged for the evaluation of sentence embeddings within the SentEval framework (Con- neau et al., 2017). A straightforward way to predict the probability of (s1, R, s2) being true is to represent s1 and s2 with d-dimensional embeddings h1 and h2, and to compute sentence pair features f(h1, h2), where f is a composition function (e.g. concatenation, product, ...). A softmax classifier gθ can learn to predict R …

Content truncated to 500 words