Implicit Discourse Relation Classification: We Need to Talk about Evaluation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5404–5414 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 5404 Implicit Discourse Relation Classification: We Need to Talk about Evaluation Najoung Kim∗ Department of Cognitive Science Johns Hopkins University n.kim@jhu.edu Song Feng, Chulaka Gunasekara, Luis A. Lastras IBM Research AI {sfeng@us, chulaka.gunasekara@, lastrasl@us}.ibm.com Abstract Implicit relation classification on Penn Dis- course TreeBank (PDTB) 2.0 is a common benchmark task for evaluating the understand- ing of discourse relations. However, the lack of consistency in preprocessing and evaluation poses challenges to fair comparison of results in the literature. In this work, we highlight these inconsistencies and propose an improved evaluation protocol. Paired with this proto- col, we report strong baseline results from pre- trained sentence encoders, which set the new state-of-the-art for PDTB 2.0. Furthermore, this work is the first to explore fine-grained re- lation classification on PDTB 3.0. We expect our work to serve as a point of comparison for future work, and also as an initiative to discuss models of larger context and possible data aug- mentations for downstream transferability. 1 Introduction Understanding discourse relations in natural lan- guage text is crucial to end tasks involving larger context, such as question-answering (Jansen et al., 2014) and conversational systems grounded on doc- uments (Saeidi et al., 2018; Feng et al., 2020). One way to characterize discourse is through relations between two spans or arguments (ARG1/ARG2) as in the Penn Discourse TreeBank (PDTB) (Prasad et al., 2008, 2019). For instance: [Arg1 I live in this world,] [Arg2 assuming that there is no morality, God or police.] (wsj_0790) Label: EXPANSION.MANNER.ARG2-AS-MANNER The literature has focused on implicit discourse re- lations from PDTB 2.0 (Pitler et al., 2009; Lin et al., 2009), on which deep learning has yielded substan- tial performance gains (Chen et al., 2016; Liu and Li, 2016; Lan et al., 2017; Qin et al., 2017; Bai and ∗Work done while at IBM Research. Zhao, 2018; Nguyen et al., 2019, i.a.). However, in- consistencies in preprocessing and evaluation such as different label sets (Rutherford et al., 2017) pose challenges to fair comparison of results and to ana- lyzing the impact of new models. In this paper, we revisit prior work to explicate the inconsistencies and propose an improved evaluation protocol to promote experimental rigor in future work. Paired with this guideline, we present a set of strong base- lines from pretrained sentence encoders on both PDTB 2.0 and 3.0 that set the state-of-the-art. We furthermore reflect on the results and discuss fu- ture directions. We summarize our contributions as follows: • We highlight preprocessing and evaluation in- consistencies in works using PDTB 2.0 for implicit discourse relation classification. We expect our work to serve as a comprehensive guide to common practices in the literature. • We lay out an improved evaluation protocol using section-based cross-validation that pre- serves document-level structure. • We report state-of-the-art results on both top- level and second-level implicit discourse rela- tion classification on PDTB …

Content truncated to 500 words