Inferring Discourse Relations from PDTB-style Discourse Labels for Argumentative Revision Classification

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2615–2624, Osaka, Japan, December 11-17 2016. Inferring Discourse Relations from PDTB-style Discourse Labels for Argumentative Revision Classification Fan Zhang Diane Litman Katherine Forbes Riley University of Pittsburgh Pittsburgh, PA, 15260 {zhangfan,litman}@cs.pitt.edu, katherineforbesriley@gmail.com Abstract Penn Discourse Treebank (PDTB)-style annotation focuses on labeling local discourse relations between text spans and typically ignores larger discourse contexts. In this paper we propose two approaches to infer discourse relations in a paragraph-level context from annotated PDTB labels. We investigate the utility of inferring such discourse information using the task of revision classi- fication. Experimental results demonstrate that the inferred information can significantly improve classification performance compared to baselines, not only when PDTB annotation comes from humans but also from automatic parsers. 1 Introduction Widely used in discourse research, Penn Discourse Treebank (PDTB)-style annotation (Prasad et al., 2008) adopts a lexically grounded approach by anchoring discourse relations according to discourse connectives. In a typical PDTB annotation process, an annotator first locates discourse connectives (explicit or implicit) then annotates text spans as their arguments. While the process of manual PDTB annotation has been demonstrated to yield reliable results (Alsaif and Markert, 2011; Danlos et al., 2012; Zhou and Xue, 2015; Zeyrek et al., 2013), it yields more shallow annotation when compared to another widely-used discourse scheme, namely Rhetorical Structure Theory (RST) (Mann and Thompson, 1988; Carlson et al., 2002). This is because when using RST a text is represented as a hierarchical discourse tree, while when using PDTB the relations exist only locally (typically between sentences or clauses). The lack of discourse information across larger contexts potentially limits the utility of PDTB-style labels. Feng et al. (2014) found that when applied to the tasks of sentence ordering and essay scoring, an RST-style discourse parser outperformed a PDTB-style parser. Performance on both tasks was also likely impacted by parsing errors. To address both the local nature of PDTB-style annotations as well as the errors introduced by state-of-the-art discourse parsers, we propose to first build paragraph-level discourse structures from annotated PDTB labels, then to infer discourse relations based on these structures. We hypothesize that features extracted from inferred relations will improve performance in downstream applications, compared to features extracted from only original annotations. To verify our hypotheses, we choose the task of argumentative revision classification (Zhang and Lit- man, 2015; Zhang and Litman, 2016), which aims to identify the purpose of an author’s revisions during argumentative writing. This task first detects the differences between two drafts of a paper at the sentence level, then labels each revision using one of five categories: Claim/Ideas, Warrant/Reasoning/Backing, Evidence, General Content and Surface. While Zhang and Litman (2016) demonstrated decent classi- fication performance without using discourse structure, an error analysis of their results suggests that discourse relations might improve performance (e.g., their current system has trouble differentiating between changing reasoning (Warrant/Reasoning/Backing) versus smoothing transitions (General Con- tent)). One of the corpora used in their work has recently been annotated …

Content truncated to 500 words