DiscoGeM: A Crowdsourced Corpus of Genre-Mixed Implicit Discourse Relations

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 3281–3290 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 3281 DiscoGeM: A Crowdsourced Corpus of Genre-Mixed Implicit Discourse Relations Merel C.J. Scholman, Tianai Dong, Frances Yung, Vera Demberg Saarland University Campus C7.2, 66123, Saarbr¨ucken {m.c.j.scholman,tdong,frances,vera}@coli.uni-saarland.de Abstract We present DiscoGeM, a crowdsourced corpus of 6,505 implicit discourse relations from three genres: political speech, literature, and encyclopedic texts. Each instance was annotated by 10 crowd workers. Various label aggregation methods were explored to evaluate how to obtain a label that best captures the meaning inferred by the crowd annotators. The results show that a significant proportion of discourse relations in DiscoGeM are ambiguous and can express multiple relation senses. Probability distribution labels better capture these interpretations than single labels. Further, the results emphasize that text genre crucially affects the distribution of discourse relations, suggesting that genre should be included as a factor in automatic relation classification. We make available the newly created DiscoGeM corpus, as well as the dataset with all annotator-level labels. Both the corpus and the dataset can facilitate a multitude of applications and research purposes, for example to function as training data to improve the performance of automatic discourse relation parsers, as well as facilitate research into non-connective signals of discourse relations. Keywords: discourse annotations, implicit relations, genre, crowdsourcing, label aggregation 1. Introduction Discourse relations are semantic links between text segments (Hobbs, 1979; Sanders et al., 1992). common distinction is made between explicit relations,A which are marked by a discourse connective such as “because” or “however”, and implicit relations, which do not contain a specific discourse connective. Un- derstanding the discourse relations that hold between segments in natural language is crucial to many NLP applications, such as text generation, dialogue under- standing, and question-answering systems. Shallow discourse parsers aim to predict the sense of the discourse relation. While parsers show good perfor- mance on explicit relations (Pitler and Nenkova, 2009; Lin et al., 2014; Knaebel and Stede, 2020), perfor- mance on labelling implicit relations is significantly lacking, with the current state-of-the-art achieving an F1 around 64% on a four-way classification (Ji et al., 2016; Lan et al., 2017; Liang et al., 2020; Shi and Demberg, 2019a). Moreover, it has been shown that the performance of parsers degrades strongly if used on a different domain (Scholman et al., 2021). There- fore, creating additional resources for implicit relations in different genres is necessary. In addition, resources on implicit relations can facilitate research investigat- ing how implicit relations are signalled. However, obtaining manually annotated data is a costly and time-consuming process. To contribute to this data collection challenge, we used a crowdsourcing method- ology to create a corpus of 6,505 annotated inter- sentential implicit discourse relations. We included data from three genres to foster cross-genre studies: political speech taken from the Europarl corpus, liter- ary texts taken from 20 novels, and encyclopedic texts stemming from Wikipedia. The relations are annotated with the same annotation …

Content truncated to 500 words