Multilevel Text Alignment with Cross-Document Attention

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5012–5025, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5012 Multilevel Text Alignment with Cross-Document Attention Xuhui Zhou♥ Nikolaos Pappas♣ Noah A. Smith♣♦ ♥Department of Linguistics, University of Washington ♣Paul G. Allen School of Computer Science & Engineering, University of Washington ♦Allen Institute for Artificial Intelligence xuhuizh@uw.edu,{npappas,nasmith}@cs.washington.edu Abstract Text alignment finds application in tasks such as citation recommendation and plagiarism de- tection. Existing alignment methods operate at a single, predefined level and cannot learn to align texts at, for example, sentence and document levels. We propose a new learning approach that equips previously established hi- erarchical attention encoders for representing documents with a cross-document attention component, enabling structural comparisons across different levels (document-to-document and sentence-to-document). Our component is weakly supervised from document pairs and can align at multiple levels. Our evaluation on predicting document-to-document relation- ships and sentence-to-document relationships on the tasks of citation recommendation and plagiarism detection shows that our approach outperforms previously established hierarchi- cal, attention encoders based on recurrent and transformer contextualization that are unaware of structural correspondence between docu- ments. 1 Introduction Aligning texts and understanding their relation- ships is a common problem for NLP tasks such as citation recommendation (Bhagavatula et al., 2018; Jiang et al., 2019), comparable document mining (He et al., 2010; Peng et al., 2016; Bhagavatula et al., 2018; Guo et al., 2019), parallel sentence mining (Shi et al., 2006; Ture and Lin, 2012; Guo et al., 2018), plagiarism detection (Barr´on-Cede˜no et al., 2010; Forner et al., 2013; Ferrero et al., 2017), paraphrase identification (Wan et al., 2006; Das and Smith, 2009; Wang et al., 2016), and textual entailment (Dagan and Glickman, 2004; Androut- sopoulos and Malakasiotis, 2010; Zhao et al., 2016). Longer texts make the problem more challenging due to the potential complexity of the underlying correspondence. Here, we develop a model to ad- Figure 1: A motivating example of aligning scientific documents at different levels. We consider citation rec- ommendation (whether A cites B) and citation local- ization (which sentence in A cites) at the same time. The confidence of our model for citation localization is represented by the degree of blueness. dress this problem and demonstrate its applicability on three different tasks which require such under- standing, namely on citation recommendation, ci- tation localization, and plagiarism detection for general web documents. One key component of an NLP system for align- ing documents is the encoding process. Present approaches for comparing documents rely on hi- erarchically structured document encoders such as hierarchical attention networks (HANs; Yang et al., 2016), which independently represent the two documents as fixed-length vectors. The vec- tors are fed to a classifier which makes a decision on the relation between them (Jiang et al., 2019; Guo et al., 2019). However, such methods do not provide insights about or leverage the underlying relationships across documents and are applicable only to a single, predefined level (Jiang et al., 2019; Yang et al., 2020). Importantly, when …

Content truncated to 500 words