On the Creation of a Corpus for Coherence Evaluation of Discursive Units

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1067–1072 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1067 On the Creation of a Corpus for Coherence Evaluation of Discursive Units Elham Mohammadi, Timothe Beiko, and Leila Kosseim Computational Linguistics at Concordia (CLaC) Laboratory Department of Computer Science and Software Engineering Concordia University, Montr´eal, Qu´ebec, Canada elham.mohammadi@concordia.ca t.beiko23@gmail.com leila.kosseim@concordia.ca Abstract In this paper, we report on our experiments towards the creation of a corpus for coherence evaluation. Most corpora for textual coherence evaluation are composed of randomly shuffled sentences that focus on sentence ordering, regardless of whether the sentences were originally related by a discourse relation. To the best of our knowledge, no publicly available corpus has been designed specifically for the evaluation of coherence of known discursive units. In this paper, we focus on coherence modeling at the intra-discursive level and describe our approach to build a corpus of incoherent pairs of sentences. We experimented with a variety of corruption strategies to create synthetic incoherent pairs of discourse arguments from coherent ones. Using discourse argument pairs from the Penn Discourse Tree Bank (Prasad et al., 2008), we generate incoherent discourse argument pairs, by swapping either their discourse connective or a discourse argument. To evaluate how incoherent the generated corpora are, we use a convolutional neural network to try to distinguish the original pairs from the corrupted ones. Results of the classifier as well as a manual inspection of the corpora show that generating such corpora is still a challenge as the generated instances are clearly not “incoherent enough”, indicating that more effort should be spent on developing more robust ways of generating incoherent corpora. Keywords: Discourse Analysis, Discourse Coherence, Corpus Creation 1. Introduction A common assumption in natural language analysis is that the input text is coherent. However, this premise may not always hold, especially in the case of automatically gen- erated texts or texts written by humans with lower lan- guage skills or with health issues affecting language. In these cases, the automatic evaluation of textual coherence can help towards improving the quality of automatically- generated text or detecting authors with specific linguistic deficiencies (Abdalla et al., 2018). In order to perform automatic coherence evaluation, a cor- pus including both coherent and incoherent samples is needed. Coherent texts, are easy to find; however incoher- ent texts are not. Most corpora for textual coherence eval- uation are synthetic data sets composed of randomly shuf- fled sentences (Lapata and Barzilay, 2005; Li and Jurafsky, 2017a; Logeswaran et al., 2018) which are commonly used for sentence ordering tasks (Logeswaran et al., 2018; Cui et al., 2018; Gong et al., 2016; Chen et al., 2016). How- ever, these corpora do not consider if the original pairs of sentences are related by a discourse relation or not; hence, the difficulty of the sentence ordering task may vary sig- nificantly. To our knowledge, no publicly available corpus exists for coherence evaluation of known discursive units …

Content truncated to 500 words