NUIG-Panlingua-KMI Hindi-Marathi MT Systems for Similar Language Translation Task @ WMT 2020

Proceedings of the 5th Conference on Machine Translation (WMT), pages 76–91 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 76 Findings of the WMT 2020 Shared Task on Machine Translation Robustness Lucia Specia1, Zhenhao Li1, Juan Pino2, Vishrav Chaudhary2, Francisco Guzm´an2 Paul Michel3, Graham Neubig3, Hassan Sajjad4, Nadir Durrani4, Yonatan Belinkov5, Philipp Koehn 2,6, Xian Li2 1Imperial College London, 2Facebook AI, 3Carnegie Mellon University, 4Qatar Computing Research Institute, 5Harvard University and MIT, 6Johns Hopkins University Abstract We report the findings of the second edition of the shared task on improving robustness in Ma- chine Translation (MT). The task aims to test current machine translation systems in their ability to handle challenges facing MT models to be deployed in the real world, including do- main diversity and non-standard texts common in user generated content, especially in social media. We cover two language pairs – English- German and English-Japanese and provide test sets in zero-shot and few-shot variants. Par- ticipating systems are evaluated both automat- ically and manually, with an additional hu- man evaluation for “catastrophic errors”. We received 59 submissions by 11 participating teams from a variety of types of institutions. 1 Introduction In recent years, Machine Translation (MT) sys- tems have seen great progress, with neural models becoming the de-facto methods and even approach- ing human quality in news domain (Hassan et al., 2018). However, like other deep learning mod- els, neural machine translation (NMT) models are found to be sensitive to synthetic and natural noise in input, distributional shift, and adversarial ex- amples (Koehn and Knowles, 2017; Belinkov and Bisk, 2018; Durrani et al., 2019; Anastasopoulos et al., 2019; Michel et al., 2019). From an appli- cation perspective, MT systems need to deal with non-standard, noisy text of the kind which is ubiq- uitous on social media and the internet, yet has different distributional signatures from corpora in common benchmark datasets. Following the first shared task on Machine Trans- lation (MT) Robustness, we now propose a second edition, which aims at testing MT systems’ robust- ness towards domain diversity. Specifically, this year’s task aims to evaluate a general MT system’s performance in the following two scenarios: • Zero-shot: the goal is to evaluate a general MT system’s performance in unseen domains at test time, which are likely to be different from a training domain (e.g. News, Wikipedia). For that, no domain-specific data or information on the test sets is given to participants. • Few-shot: the goal is to test an MT system’s performance if a few in-domain training exam- ples are provided for the target domain. The question we ask is: can the general MT system leverage those training examples to improve performance on this domain while not drop- ping its performance on other domains? We describe the dataset and the task setup in Section 3. The shared-task attracted a total of 23 submissions from 11 teams. The teams employed a variety of methods to improve robustness. A spe- cific challenge was the small size of the in-domain …

Content truncated to 500 words