How much progress have we made on RST discourse parsing? A replication study of recent results on the RST-DT

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1319–1324 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics How much progress have we made on RST discourse parsing? A replication study of recent results on the RST-DT Mathieu Morey and Philippe Muller and Nicholas Asher IRIT, Universit´e Toulouse & CNRS, Univ. Paul Sabatier {mathieu.morey, philippe.muller, nicholas.asher}@irit.fr Abstract This article evaluates purported progress over the past years in RST discourse pars- ing. Several studies report a relative er- ror reduction of 24 to 51% on all met- rics that authors attribute to the introduc- tion of distributed representations of dis- course units. We replicate the standard evaluation of 9 parsers, 5 of which use distributed representations, from 8 stud- ies published between 2013 and 2017, us- ing their predictions on the test set of the RST-DT. Our main finding is that most re- cently reported increases in RST discourse parser performance are an artefact of dif- ferences in implementations of the eval- uation procedure. We evaluate all these parsers with the standard Parseval proce- dure to provide a more accurate picture of the actual RST discourse parsers per- formance in standard evaluation settings. Under this more stringent procedure, the gains attributable to distributed representa- tions represent at most a 16% relative error reduction on fully-labelled structures. 1 Introduction While several theories of discourse structure for text exist, discourse parsing work has largely concentrated on Rhetorical Structure Theory (RST) (Mann and Thompson, 1988) and the RST Discourse Treebank (RST-DT) (Carlson et al., 2003), which is the largest corpus of texts anno- tated with full discourse structures. The RST-DT, annotated in the style of RST, consists of 385 news articles from the Penn Treebank, split into a train- ing and test sets of 347 and 38 documents.The standard evaluation procedure for RST discourse parsing, RST-Parseval, proposed by Marcu (2000), adapts the Parseval procedure for syntactic pars- ing (Black et al., 1991). RST-Parseval computes scores on discourse structures with no label (S for Span) or labelled with nuclearity (N), relation (R) or both (F for Full). The semantic nature of discourse relations makes discourse parsing a dif- ficult task. However, the recent introduction of distributed representations of discourse units has seemingly led to significant improvements, with a claimed relative error reduction of 51% on fully labelled structures. As part of a broader study of methods and evaluation metrics for discourse pars- ing, we collected predictions from nine RST dis- course parsers and reimplemented RST-Parseval. In section 2, we present these RST parsers and re- port their published scores on RST-Parseval. In section 3, we replicate their evaluation and show that most of the heterogeneity in performance across RST parsers arises from differences in their evaluation procedures. In section 4, we replace RST-Parseval with the standard Parseval proce- dure and obtain a more accurate picture of the ac- tual performance of RST parsers. 2 A sample of RST discourse parsers Almost all RST discourse parsers are evaluated on the …

Content truncated to 500 words