Mining Discourse Markers for Unsupervised Sentence Representation Learning

Proceedings of NAACL-HLT 2019, pages 3477–3486 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3477 Mining Discourse Markers for Unsupervised Sentence Representation Learning Damien Sileo3,1, Tim Van De Cruys2, Camille Pradel1, and Philippe Muller3 1Synapse D´eveloppement, Toulouse, France 2IRIT, CNRS, France 3IRIT, University of Toulouse, France damien.sileo@synapse-fr.com Abstract Current state of the art systems in NLP heav- ily rely on manually annotated datasets, which are expensive to construct. Very little work ad- equately exploits unannotated data – such as discourse markers between sentences – mainly because of data sparseness and ineffective ex- traction methods. In the present work, we pro- pose a method to automatically discover sen- tence pairs with relevant discourse markers, and apply it to massive amounts of data. Our resulting dataset contains 174 discourse mark- ers with at least 10K examples each, even for rare markers such as coincidentally or amaz- ingly. We use the resulting data as supervision for learning transferable sentence embeddings. In addition, we show that even though sen- tence representation learning through predic- tion of discourse markers yields state of the art results across different transfer tasks, it is not clear that our models made use of the semantic relation between sentences, thus leaving room for further improvements. Our datasets are publicly available 1 1 Introduction An important challenge within the domain of nat- ural language processing is the construction of ad- equate semantic representations for textual units – from words over sentences to whole documents. Recently, numerous approaches have been pro- posed for the construction of vector-based repre- sentations for larger textual units, especially sen- tences. One of the most popular frameworks aims to induce sentence embeddings as an intermedi- ate representation for predicting relations between sentence pairs. For instance, similarity judge- ments (paraphrases) or inference relations have been used as prediction tasks, and the resulting embeddings perform well in practice, even when 1https://github.com/ synapse-developpement/Discovery the representations are transfered to other seman- tic tasks (Conneau et al., 2017). However, the kind of annotated data that is needed for such super- vised approaches is costly to obtain, prone to bias, and arguably fairly limited with regard to the kind of semantic information captured, as they single out a narrow aspect of the entire semantic content. Unsupervised approaches have also been pro- posed, based on sentence distributions in large cor- pora in relation to their discourse context. For in- stance, Kiros et al. (2015) construct sentence rep- resentations by trying to reconstruct neighbouring sentences, which allows them to take into account different contextual aspects of sentence meaning. In the same vein, Logeswaran et al. (2016) propose to predict if two sentences are consecutive, even though such local coherence can be straightfor- wardly predicted with relatively shallow features (Barzilay and Lapata, 2008). A more elaborate setting is the prediction of the semantic or rhetor- ical relation between two sentences, as is the goal of discourse parsing. A number of annotated cor- pora exist, such as RST-DT (Carlson et al., 2001) …

Content truncated to 500 words