DisSent: Learning Sentence Representations from Explicit Discourse Relations

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4497–4510 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4497 DisSent: Learning Sentence Representations from Explicit Discourse Relations Allen Nie∗1 Erin D. Bennett∗2 Noah D. Goodman1,2 1Department of Computer Science 2Department of Psychology Stanford University anie@cs.stanford.edu {erindb,ngoodman}@stanford.edu Abstract Learning effective representations of sen- tences is one of the core missions of natu- ral language understanding. Existing models either train on a vast amount of text, or re- quire costly, manually curated sentence rela- tion datasets. We show that with dependency parsing and rule-based rubrics, we can curate a high quality sentence relation task by lever- aging explicit discourse relations. We show that our curated dataset provides an excellent signal for learning vector representations of sentence meaning, representing relations that can only be determined when the meanings of two sentences are combined. We demon- strate that the automatically curated corpus al- lows a bidirectional LSTM sentence encoder to yield high quality sentence embeddings and can serve as a supervised fine-tuning dataset for larger models such as BERT. Our fixed sen- tence embeddings achieve high performance on a variety of transfer tasks, including Sen- tEval, and we achieve state-of-the-art results on Penn Discourse Treebank’s implicit rela- tion prediction task. 1 Introduction Developing general models to represent the mean- ing of a sentence is a key task in natural language understanding. The applications of such general- purpose representations of sentence meaning are many — paraphrase detection, summarization, knowledge-base population, question-answering, automatic message forwarding, and metaphoric language, to name a few. We propose to leverage a high-level relationship between sentences that is both frequently and sys- tematically marked in natural language: the dis- course relations between sentences. Human writ- ers naturally use a small set of very common tran- sition words between sentences (or sentence-like ∗ equal contribution phrases) to identify the relations between adjacent ideas. These words, such as because, but, and, which mark the conceptual relationship between two sentences, have been widely studied in lin- guistics, both formally and computationally, and have many different names. We use the name “dis- course markers”. Learning flexible meaning representations re- quires a sufficiently demanding, yet tractable, training task. Discourse markers annotate deep conceptual relations between sentences. Learning to predict them may thus represent a strong train- ing task for sentence meanings. This task is an in- teresting intermediary between two recent trends. On the one hand, models like InferSent (Conneau et al., 2017) are trained to predict entailment—a strong conceptual relation that must be hand an- notated. On the other hand, models like BERT (Devlin et al., 2018) are trained to predict random missing words in very large corpora (see Table 1 for the data requirements of the models we com- pare). Discourse marker prediction may permit learning from relatively little data, like entailment, but can rely on naturally occurring data rather than hand annotation, like word-prediction. We thus propose the DisSent task, which uses the Discourse …

Content truncated to 500 words