Next Sentence Prediction helps Implicit Discourse Relation Classification within and across Domains

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5790–5796, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5790 Next Sentence Prediction helps Implicit Discourse Relation Classification within and across Domains Wei Shi† and Vera Demberg†,‡ †Dept. of Language Science and Technology ‡Dept. of Mathematics and Computer Science, Saarland University Saarland Informatics Campus, 66123 Saarbr¨ucken, Germany {w.shi, vera}@coli.uni-saarland.de Abstract Implicit discourse relation classification is one of the most difficult tasks in discourse parsing. Previous studies have generally focused on ex- tracting better representations of the relational arguments. In order to solve the task, it is how- ever additionally necessary to capture what events are expected to cause or follow each other. Current discourse relation classifiers fall short in this respect. We here show that this shortcoming can be effectively addressed by using the bidirectional encoder representation from transformers (BERT) proposed by Devlin et al. (2019), which were trained on a next- sentence prediction task, and thus encode a representation of likely next sentences. The BERT-based model outperforms the current state of the art in 11-way classification by 8% points on the standard PDTB dataset. Our ex- periments also demonstrate that the model can be successfully ported to other domains: on the BioDRB dataset, the model outperforms the state of the art system around 15% points. 1 Introduction Discourse relation classification has been shown to be beneficial to multiple down-stream NLP tasks such as machine translation (Li et al., 2014), ques- tion answering (Jansen et al., 2014) and summa- rization (Yoshida et al., 2014). Following the re- lease of the Penn Discourse Tree Bank (Prasad et al., 2008, PDTB), discourse relation classifica- tion has received a lot of attention from the NLP community, including two CoNLL shared tasks (Xue et al., 2015, 2016). Discourse relations in texts are sometimes marked with an explicit connective (e.g., but, be- cause, however), but these explicit signals are of- ten absent. With explicit connectives acting as informative cues, it is relatively easy to classify the discourse relation with high accuracy (93.09% on four-way classification in (Pitler et al., 2008)). When there is no connective, classification has to rely on semantic information from the rela- tional arguments. This task is very challenging, with state-of-the-art systems achieving accuracy of only 45% to 48% on 11-way classification. Consider example 1: (1) [The joint venture with Mr. Lang wasn’t a good one.]Arg1 [The venture, formed in 1986, was supposed to be Time’s low-cost, safe entry into women’s magazines.]Arg2 implicit Comp.Concess.expectation relation from PDTB: wsj 1903 In order to correctly classify the relation, it is necessary to understand that Arg1 raises the ex- pectation that the next discourse segment may pro- vide an explanation for why the venture wasn’t good (e.g., that it was risky), and Arg2 contrasts with this discourse expectation. More generally, this means that a successful discourse relation classification model would have to be able to learn typical temporal event …

Content truncated to 500 words