Entity-based Neural Local Coherence Modeling

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7787 - 7805 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Entity-based Neural Local Coherence Modeling Sungho Jeon and Michael Strube Heidelberg Institute for Theoretical Studies gGmbH {sungho.jeon, michael.strube}@h-its.org Abstract In this paper, we propose an entity-based neu- ral local coherence model which is linguis- tically more sound than previously proposed neural coherence models. Recent neural co- herence models encode the input document using large-scale pretrained language models. Hence their basis for computing local coher- ence are words and even sub-words. An anal- ysis of their output shows that these models frequently compute coherence on the basis of connections between (sub-)words which, from a linguistic perspective, should not play a role. Still, these models achieve state-of-the-art per- formance in several end applications. In con- trast to these models, we compute coherence on the basis of entities by constraining the in- put to noun phrases and proper names. This provides us with an explicit representation of the most important items in sentences leading to the notion of focus. This brings our model linguistically in line with pre-neural models of computing coherence. It also gives us better insight into the behaviour of the model thus leading to better explainability. Our approach is also in accord with a recent study (O’Connor and Andreas, 2021), which shows that most usable information is captured by nouns and verbs in transformer-based language models. We evaluate our model on three downstream tasks showing that it is not only linguistically more sound than previous models but also that it outperforms them in end applications1. 1 Introduction Coherence describes the semantic relation between elements of a text. It recognizes how well a text is organized to convey the information to the reader effectively. Modeling coherence can be beneficial to any system which needs to process a text. 1Our code is available at: https://github.com/ sdeva14/acl22-entity-neural-local-cohe. Example Sentence 1 Mr. Specter, seeming exasperated, said in an inter- view Thursday. Focus candidates captured by XLNet “_said”, “_in”, “day”, “_interview”, “_,”, “er”, “_an”, “th”, “s”, “_exasperated”, ..., “spect” Example Sentence 2 At the same time, unadvertised products may have almost identical ingredients but less name- recognition. Focus candidates captured by XLNet “_name”, “ition”, “_products”, “-”, ”_un”, “_may”, “_less”, “_ingredients”, “_have”, ..., “_same” Table 1: The pretrained language model, XLNet Yang et al. (2019), captures undesirable (sub-)words as focus (Jeon and Strube, 2020). The sub-words are sorted by their attention scores in descending order. In the first example, “Thursday” is split into four: “th”, “ur”, “s”, and “day”. In the second example, some sub-words, such as “ition”, might be beneficial in their vector space but the model might exploit spurious information. Recent neural coherence models (Mesgar and Strube, 2018; Moon et al., 2019) encode the input document using large-scale pretrained language models (Peters et al., 2018). These neural models compute local coherence, semantic relations be- tween items in adjacent sentences, on the basis of words and even …

Content truncated to 500 words