Discourse-Centric Evaluation of Document-level Machine Translation with a New Densely Annotated Parallel Corpus of Novels

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7853–7872 July 9-14, 2023 ©2023 Association for Computational Linguistics Discourse-Centric Evaluation of Document-level Machine Translation with a New Densely Annotated Parallel Corpus of Novels Yuchen Eleanor Jiangζ Tianyu Liuζ Shuming Maγ Dongdong Zhangγ Mrinmaya Sachanζ Ryan Cotterellζ ζETH Zürich γMicrosoft Research Asia {yuchen.jiang,tianyu.liu,ryan.cotterell,mrinmaya.sachan}@inf.ethz.ch {shuming.ma,dongdong.zhang}@microsoft.com Abstract Several recent papers claim to have achieved human parity at sentence-level machine translation (MT)—especially between high- resource language pairs. In response, the MT community has, in part, shifted its focus to document-level translation. Translating docu- ments requires a deeper understanding of the structure and meaning of text, which is often captured by various kinds of discourse phe- nomena such as consistency, coherence, and cohesion. However, this renders conventional sentence-level MT evaluation benchmarks inadequate for evaluating the performance of context-aware MT systems. This paper presents a new dataset with rich discourse annotations, built upon the large-scale parallel corpus BWB introduced in Jiang et al. (2022a). The new BWB annotation introduces four extra evaluation aspects, i.e., entity, terminology, coreference, and quotation, covering 15,095 entity mentions in both languages. Using these annotations, we systematically investigate the similarities and differences between the discourse structures of source and target languages, and the challenges they pose to MT. We discover that MT outputs differ fundamentally from human translations in terms of their latent discourse structures. This gives us a new perspective on the challenges and opportunities in document-level MT. We make our resource publicly available to spur future research in document-level MT and its generalization to other language translation tasks. https://github.com/EleanorJiang/ BlonDe/tree/main/BWB 1 Introduction The field of machine translation (MT) has made tremendous strides in recent years thanks to neural machine translation (NMT) models that can utilize massive quantities of parallel training data (Vaswani et al., 2017; Junczys-Dowmunt et al., 2018, inter alia). Unfortunately, most parallel cor- pora are aligned only at the sentence level, and document-level translation has remained limited to small-scale studies (Ansari et al., 2020; Lison et al., 2018; Koehn, 2005; Tiedemann, 2012). There are, however, inherent differences be- tween sentence-level translation and document- level translation, as documents consist of com- plex discourse structures that go beyond the mere concatenation of individual sentences. Three key discourse features are particularly important in document-level translation. First, the translations of named entities have to be consistent. For ex- ample, in Fig. 1, the same terminologies are not consistently referred to with the same translations (i.e., Weibo vs. micro-blog, Qiao Lian vs. Joe vs. Joe love), and as a result, the sentence-level MT system fails to capture discourse dependencies across sentences. Second, the coreference relation- ship in the source language needs to be preserved. In particular, the relations between entities and their pronominal anaphora should be preserved, as well as the transition of discourse centers1. In Fig. 1 and Fig. 8, coreference chains are color-coded, vi- sually demonstrating the preservation of anaphoric referential relations and the transition chains of cen- ters. Third, the conversational structure, …

Content truncated to 500 words