Incremental Neural Lexical Coherence Modeling

Proceedings of the 28th International Conference on Computational Linguistics, pages 6752–6758 Barcelona, Spain (Online), December 8-13, 2020 6752 Incremental Neural Lexical Coherence Modeling Sungho Jeon and Michael Strube Heidelberg Institute for Theoretical Studies gGmbH {sungho.jeon, michael.strube}@h-its.org Abstract Pretrained language models, neural models pretrained on massive amounts of data, have estab- lished the state of the art in a range of NLP tasks. They are based on a modern machine-learning technique, the Transformer which relates all items simultaneously to capture semantic relations in sequences. However, it differs from what humans do. Humans read sentences one-by-one, incrementally. Can neural models benefit by interpreting texts incrementally as humans do? We investigate this question in coherence modeling. We propose a coherence model which interprets sentences incrementally to capture lexical relations between them. We compare the state of the art in each task, simple neural models relying on a pretrained language model, and our model in two downstream tasks. Our findings suggest that interpreting texts incrementally as humans could be useful to design more advanced models. 1 Introduction Coherence describes the semantic relation between elements of a text. It distinguishes a text as either a unified whole or a collection of unrelated sentences. Lexical coherence represents the cohesive effect achieved by lexical relations (Halliday and Hasan, 1976). Earlier work mainly focuses on capturing lexical relations using external resources (Morris and Hirst, 1991). Mesgar and Strube (2016) introduce a graph model, the latest model for lexical coherence, to represent lexical relations between sentences on the graph. It encodes sentences as nodes and lexical relations between sentences as edges. This model, nevertheless, considers lexical items independently. Recent neural models adopt a modern machine learning-based technique (Liu and Lapata, 2019; Gupta and Durrett, 2019), the Transformer (Vaswani et al., 2017). It relates all items simultaneously to capture semantic relations in sequences. More recently, large-scale pretrained language models, Transformer- based models pretrained on the massive amounts of text, have led to significant improvements in a range of NLP tasks (Devlin et al., 2019). However, the Transformer processes texts in a way which is different from the way humans do it. Psy- cholinguistic experiments show that humans read texts incrementally (Marslen-Wilson, 1975; Kamide et al., 2003; Gibson and Warren, 2004). K¨ohn (2018) claim that NLP systems which follow this theory should interpret texts incrementally, too. Do neural models benefit from both pretrained language models and incremental sentence processing? To investigate this question, we propose a coherence model which interprets sentences incrementally to capture lexical relations. For the ongoing sentence being read, our model first captures a semantic centroid vector which represents the centroid of preceding sentences. The centroid vector is computed as averaged representations of sentences. The model then measures semantic similarity between the centroid vector and the current sentence. Our model iterates this procedure for all sentences. This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http:// creativecommons.org/licenses/by/4.0/. 6753 Large-scale pretrained language model word1 h Model Output word2 wordn h h word3 h sent1 …

Content truncated to 500 words