DDisCo: A Discourse Coherence Dataset for Danish

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2440–2445 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2440 DDisCo: A Discourse Coherence Dataset for Danish Linea Flansmose1∗, Oliver Kinch2, Anders Jess Pedersen2, Oph´elie Lacroix3∗ 1Aarhus University, Jens Chr. Skous Vej 2, 8000 Aarhus 2Alexandra Institute, Rued Langgaards Vej 7, 2300 Copenhagen 3Wunderman Thompson MAP, Glentevej 61, 2400 Copenhagen linea.flansmose@gmail.com {oliver.kinch, anders.j.pedersen}@alexandra.dk ophelie.lacroix@wundermanthompson.com Abstract To date, there has been no resource for studying discourse coherence on real-world Danish texts. Discourse coherence has mostly been approached with the assumption that incoherent texts can be represented by coherent texts in which sentences have been shuffled. However, incoherent real-world texts rarely resemble that. We thus present DDisCo, a dataset including text from the Danish Wikipedia and Reddit annotated for discourse coherence. We choose to annotate real-world texts instead of relying on artificially incoherent text for training and testing models. Then, we evaluate the performance of several methods, including neural networks, on the dataset. Keywords: discourse coherence, dataset, low-resource 1. Introduction Coherence in text refers to the internal structure of a text as well as the overall meaning and purpose of the text in its context. This means that there are many fac- tors to look at when deciding the overall coherence of a given text, some of which are beyond the text itself. Coherence is often described as what makes the differ- ence between a unified text and a random selection of sentences or clauses (Halliday and Hasan, 2014). How- ever, when people try and fail to speak or write coher- ently it rarely resembles a completely random selec- tion of sentences. According to Halliday and Hasan (2014) it is the very definition of a text that it is coher- ent. That is, all texts written by a human should have at least some degree of coherence — but some texts can be more coherent than others. In this study, we address two main issues in the field of discourse coherence: 1) Previous studies have mainly focused on English text and, 2) developed models using randomly shuffled texts instead of real-world annotated data. In general, most current studies of discourse coherence modeling use highly edited and coherent data and then use a sentence ordering task to test the models. In the sentence ordering task, the model must decide whether a given text is a coherent text or an originally coherent text where the sentences have been shuffled randomly. At this point many models perform very well on this task, however, this test does not accurately reflect how humans actually write. No-one who tries but fails to write a coherent text will put sentences in a random order. Thus, these models don’t measure the quality of coherence in a text but rather to what degree it re- sembles a human-written text or a randomly organized ∗Research conducted at the Alexandra Institute. text. For this reason there is a need to investigate how a model would …

Content truncated to 500 words