Online Coreference Resolution for Dialogue Processing: Improving Mention-Linking on Real-Time Conversations

Proceedings of the 11th Joint Conference on Lexical and Computational Semantics, pages 28 - 43 July 14-15, 2022 ©2022 Association for Computational Linguistics Semantics-aware Attention Improves Neural Machine Translation Aviv Slobodkin Leshem Choshen Omri Abend School of Computer Science and Engineering The Hebrew University of Jerusalem {aviv.slobodkin,leshem.choshen,omri.abend}@mail.huji.ac.il Abstract The integration of syntactic structures into Transformer machine translation has shown positive results, but to our knowledge, no work has attempted to do so with semantic struc- tures. In this work we propose two novel parameter-free methods for injecting seman- tic information into Transformers, both rely on semantics-aware masking of (some of) the at- tention heads. One such method operates on the encoder, through a Scene-Aware Self-Attention (SASA) head. Another on the decoder, through a Scene-Aware Cross-Attention (SACrA) head. We show a consistent improvement over the vanilla Transformer and syntax-aware models for four language pairs. We further show an additional gain when using both semantic and syntactic structures in some language pairs. 1 Introduction It has long been argued that semantic representa- tion can benefit machine translation (Weaver, 1955; Bar-Hillel, 1960). Moreover, RNN-based neural machine translation (NMT) has been shown to ben- efit from the injection of semantic structure (Song et al., 2019; Marcheggiani et al., 2018). Despite these gains, to our knowledge, there have been no attempts to incorporate semantic structure into NMT Transformers (Vaswani et al., 2017). We ad- dress this gap, focusing on the main events in the text, as represented by UCCA (Universal Cogni- tive Conceptual Annotation; Abend and Rappoport, 2013), namely scenes. UCCA is a semantic framework originating from typological and cognitive-linguistic theories (Dixon, 2009, 2010, 2012). Its principal goal is to represent some of the main elements of the se- mantic structure of the sentence while disregarding its syntax. Formally, a UCCA representation of a passage is a directed acyclic graph where leaves correspond to the words of the sentence and nodes correspond to semantic units. The edges are la- beled by the role of their endpoint in the relation corresponding to their starting point (see Fig. 1). One of the motivations for using UCCA is its capa- bility to separate the sentence into "Scenes", which are analogous to events (see Fig. 1). Every such Scene consists of one main relation, which can be either a Process (i.e., an action), denoted by P, or a State (i.e., continuous state), denoted by S. Scenes also contain at least one Participant (i.e., entity), denoted by A. For example, the sentence in Fig. 1a comprises two scenes: the first one has the Process "saw" and two Participants – "I" and "the dog"; the second one has the Process "barked" and a single Participant – "dog". So far, to the best of our knowledge, the only structure-aware work that integrated linguistic knowledge and graph structures into Transform- ers used syntactic structures (Strubell et al., 2018; Bugliarello and Okazaki, 2020; Akoury et al., 2019; Sundararaman et al., 2019; Choshen and Abend, 2021, inter alia). The presented method builds on the method …

Content truncated to 500 words