Evaluating Document Coherence Modeling

Evaluating Document Coherence Modeling Aili Shen♣, Meladel Mistica♣, Bahar Salehi♣, Hang Li♦, Timothy Baldwin♣, Jianzhong Qi♣ ♣ The University of Melbourne, Australia ♦ AI Lab at ByteDance, China {aili.shen, misticam, tbaldwin, jianzhong.qi}@unimelb.edu.au baharsalehi@gmail.com, lihang.lh@bytedance.com Abstract While pretrained language models (LMs) have driven impressive gains over morpho- syntactic and semantic tasks, their ability to model discourse and pragmatic phenomena is less clear. As a step towards a better under- standing of their discourse modeling capabili- ties, we propose a sentence intrusion detection task. We examine the performance of a broad range of pretrained LMs on this detection task for English. Lacking a dataset for the task, we introduce INSteD, a novel intruder sentence detection dataset, containing 170,000+ doc- uments constructed from English Wikipedia and CNN news articles. Our experiments show that pretrained LMs perform impressively in in-domain evaluation, but experience a substantial drop in the cross-domain setting, indicating limited generalization capacity. Further results over a novel linguistic probe dataset show that there is substantial room for improvement, especially in the cross- domain setting. 1 Introduction Rhetorical relations refer to the transition of one sentence to the next in a span of text (Mann and Thompson, 1988; Asher and Lascarides, 2003). They are important as a discourse device that contributes to the overall coherence, understand- ing, and flow of the text. These relations span a tremendous breadth of types, including contrast, elaboration, narration, and justification. These connections allow us to communicate coopera- tively in understanding one another (Grice, 2002; Wilson and Sperber, 2004). The ability to under- stand such coherence (and conversely detect incoherence) is potentially beneficial for down- stream tasks, such as storytelling (Fan et al., 2019; Hu et al., 2020b), recipe generation (Chandu et al., 2019), document-level text generation (Park and Kim, 2015; Holtzman et al., 2018), and essay scoring (Tay et al., 2018; Li et al., 2018). However, there is little work on document coherence understanding, especially examining the capacity of pretrained language models (LMs) to model the coherence of longer documents. To address this gap, we examine the capacity of pretrained language models to capture document coherence, focused around two research questions: (1) do models truly capture the intrinsic properties of document coherence? and (2) what types of doc- ument incoherence can/can’t these models detect? We propose the sentence intrusion detection task: (1) to determine whether a document con- tains an intruder sentence (coarse-grained level); and (2) to identify the span of any intruder sen- tence (fine-grained level). We restrict the scope of the intruder text to a single sentence, noting that in practice, the incoherent text could span multiple sentences, or alternatively be sub-sentential. Existing datasets in document coherence mea- surement (Chen et al., 2019; Clercq et al., 2014; Lai and Tetreault, 2018; Mim et al., 2019; Pitler and Nenkova, 2008; Tien Nguyen and Joty, 2017) are unsuitable for our task: They are either pro- hibitively small, or do not specify the span of in- coherent text. For example, in the dataset of Lai and Tetreault (2018), …

Content truncated to 500 words