RE-Matching: A Fine-Grained Semantic Matching Method for Zero-Shot Relation Extraction

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 606–626 July 9-14, 2023 ©2023 Association for Computational Linguistics When Does Translation Require Context? A Data-driven, Multilingual Exploration Patrick Fernandes1,2,3∗ Kayo Yin4∗ Emmy Liu1 André F. T. Martins2,3,5 Graham Neubig1 1Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA 2Instituto Superior Técnico & LUMLIS (Lisbon ELLIS Unit), Lisbon, Portugal 3Instituto de Telecomunicações, Lisbon, Portugal 4University of California, Berkeley 5Unbabel, Lisbon, Portugal pfernand@cs.cmu.edu kayoyin@berkeley.edu Abstract Although proper handling of discourse signif- icantly contributes to the quality of machine translation (MT), these improvements are not adequately measured in common translation quality metrics. Recent works in context-aware MT attempt to target a small set of discourse phenomena during evaluation, however not in a fully systematic way. In this paper, we develop the Multilingual Discourse-Aware (MUDA) benchmark, a series of taggers that identify and evaluate model performance on discourse phe- nomena in any given dataset. The choice of phenomena is inspired by a novel methodology to systematically identify translations requiring context. We confirm the difficulty of previously studied phenomena while uncovering others that were previously unaddressed. We find that common context-aware MT models make only marginal improvements over context-agnostic models, which suggests these models do not handle these ambiguities effectively. We re- lease code and data for 14 language pairs to encourage the MT community to focus on ac- curately capturing discourse phenomena.1 1 Introduction In order to properly translate discourse phenom- ena including anaphoric pronouns, lexical cohe- sion, and discourse markers, a machine translation (MT) model must use information from previous utterances (Guillou et al., 2018; Läubli et al., 2018; Toral et al., 2018). However, while generating proper translations of these phenomena is important for comprehen- sion, they represent a small portion of words in natural language. Therefore, common metrics such as BLEU (Papineni et al., 2002) cannot be used to judge the quality of discourse translation. ∗ Equal contribution 1Code available at https://github.com/CoderPat/MuDA . See §A for example usages of our released toolkit Dataset Lang. Phenomena Müller et al. (2018) EN → DE Pronouns Bawden et al. (2018) EN → FR Pronouns, Coherence Lexical Consistency Voita et al. (2018) Voita et al. (2019b) EN → RU Pronouns Deixis, Ellipsis Lexical Consistency Jwalapuram et al. (2020) DE → EN FR → EN RU → EN Pronouns, Coherence Lexical Consistency Discourse Connectives Our Work 14 Pairs (§5) Pronouns, Ellipsis Formality Lexical Consistency Verb Forms Table 1: Some representative works on contextual ma- chine translation that perform evaluation on discourse phenomena, contrasted to our work. For a more com- plete review see Maruf et al. (2021). Recent work on neural machine translation (NMT) models that attempt to incorporate extra- sentential context (Tiedemann and Scherrer, 2017; Miculicich et al., 2018; Maruf and Haffari, 2018, inter alia) often perform targeted evaluation of cer- tain discourse phenomena, mostly focusing on el- lipsis, formality (Voita et al., 2019b,a), and pro- noun translation (Müller et al., 2018; Bawden et al., 2018; Lopes et al., …

Content truncated to 500 words