Unbabel-IST at the WMT Chat Translation Shared Task

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 140 - 153 December 7-8, 2022 ©2022 Association for Computational Linguistics Test Set Sampling Affects System Rankings: Expanded Human Evaluation of WMT20 English–Inuktitut Systems Rebecca Knowles and Chi-kiu Lo National Research Council Canada (NRC-CNRC) {rebecca.knowles,chikiu.lo}@nrc-cnrc.gc.ca Abstract We present a collection of expanded human annotations of the WMT20 English–Inuktitut machine translation shared task, covering the Nunavut Hansard portion of the dataset. Addi- tionally, we recompute News rankings to take into account the completed set of human anno- tations and certain irregularities in the annota- tion task construction. We show the effect of these changes on the downstream task of the evaluation of automatic metrics. Finally, we demonstrate that character-level metrics corre- late well with human judgments for the task of automatically evaluating translation into this polysynthetic language. 1 Introduction Translation between Inuktitut1 and English was fea- tured as part of the 2020 News Translation task at WMT (Barrault et al., 2020). The English–Inuktitut machine translation system rankings published in Barrault et al. (2020) were incomplete, due to the delay in the 2020 annotation campaign and because they only cover the out-of-domain portion of the test set. In this work, we present: • an expanded dataset of human annotations that covers the in-domain portion of the test set,2 • an analysis of both the existing (out-of- domain) human annotations and the new (in- domain) annotations, • revised system rankings based on the new an- notations and controlling for irregularities in the original data collection process, • and correlations of automatic MT evaluation metrics with the revised system rankings and the newly collected human annotations. 1We use the term Inuktitut here because the website of the Legislative Assembly of Nunavut lists Inuktitut, Inuinnaqtun, and English (along with French) as the languages spoken in the House (https://assembly.nu.ca/faq#n113) and the version of the Hansard released as training data is the Inuktitut version, written in syllabics. See Appendix A. 2Dataset and code: https://github.com/ nrc-cnrc/Reranking-WMT20-IKU Dataset Segments Hansard-A (H-A) 11404 Hansard-B (H-B) 7801 All Hansard 19205 WMT20-DA (N-1) 8000 WMT20-DA2 (N-2) 12002 WMT20-DACrowd (N-C) 9728 All News 29730 Total 48935 Table 1: Number of segments annotated in each dataset, without any data filtering. We use names corresponding to the dataset files, with short forms in parentheses. Our aim with this work is to release a broader set of human annotations, for continued research on translation between English and Inuktitut, as well as to demonstrate the downstream impacts of irreg- ularities in WMT data collection and publication. In particular, we draw attention to how errors in human annotation setup (not errors in the work of the annotators, but errors in the way that organizers constructed the annotation tasks) and the failure to account for their effects impact both the validity of the rankings themselves and the shared task on automatic metrics which relies on the rankings. In this way, the 2020 WMT task on Inuktitut serves as a case study of a broader issue in the field. We …

Content truncated to 500 words