WMT20 Document-Level Markable Error Exploration

Proceedings of the 5th Conference on Machine Translation (WMT), pages 371–380 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 371 WMT20 Document-Level Markable Error Exploration Vil´em Zouhar Tereza Vojtˇechov´a Ondˇrej Bojar Charles University, Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics Malostransk´e n´amˇest´ı 25, 118 00 Prague, Czech Republic {zouhar, vojtechova, bojar}@ufal.mff.cuni.cz Abstract Even though sentence-centric metrics are used widely in machine translation evaluation, document-level performance is at least equally important for professional usage. In this paper, we bring attention to detailed document-level evaluation focused on markables (expressions bearing most of the document meaning) and the negative impact of various markable error phenomena on the translation. For an annotation experiment of two phases, we chose Czech and English documents trans- lated by systems submitted to WMT20 News Translation Task. These documents are from the News, Audit and Lease domains. We show that the quality and also the kind of errors varies significantly among the domains. This systematic variance is in contrast to the auto- matic evaluation results. We inspect which specific markables are prob- lematic for MT systems and conclude with an analysis of the effect of markable error types on the MT performance measured by humans and automatic evaluation tools. 1 Introduction This paper presents the results of our test suite for WMT20 News Translation Task.1 The conclusion of Vojtˇechov´a et al. (2019), a last year’s similar effort, states that expert knowledge is vital for correct and comprehensible translation of professional domains, such as Audits or Lease agreements. Furthermore, even MT systems which make fewer mistakes and score above others in both automatic and manual evaluations are prone to making fatal errors related to markable conflicts, which render the whole document translation unus- able. 1http://www.statmt.org/wmt20/ translation-task.html In this study, we aim to organize and describe a more detailed study with a higher number of an- notators. We show three evaluation approaches: (1) automatic evaluation, (2) fluency and adequacy per document line and (3) detailed markable phe- nomena evaluation. We compare the results of this evaluation across the three domains and try to ex- plain why all of these evaluations do not produce the same ordering of MT systems by performance. This paper is organized accordingly: Section 1.1 defines the term “Markable”, Section 1.2 describes the examined documents and Section 2 introduces the two phases of our annotation experiment and shows the annotator user interface in Section 2.3. In Section 3, we discuss the results from both phases and also automatic evaluation. The main results of this examination are shown in Section 3.5 and spe- cific markable examples are discussed in Section 4. We conclude in Section 5. 1.1 Markable Definition A markable in this context is an occurrence of any technical or non-technical term or expression that satisfies at least one of the following conditions: 1. The term was translated into two or more dif- ferent ways within one document. 2. The term was translated into two or more dif- ferent ways across several translations. …

Content truncated to 500 words