TMUOU Submission for WMT20 Quality Estimation Shared Task

Proceedings of the 5th Conference on Machine Translation (WMT), pages 1037–1041 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 1037 TMUOU Submission for WMT20 Quality Estimation Shared Task Akifumi Nakamachi Osaka University nakamachi.akifumi@ist.osaka-u.ac.jp Tomoyuki Kajiwara Osaka University kajiwara@ids.osaka-u.ac.jp Hiroki Shimanaka Tokyo Metropolitan University shimanaka-hiroki@ed.tmu.ac.jp Mamoru Komachi Tokyo Metropolitan University komachi@tmu.ac.jp Abstract We introduce the TMUOU1 submission for the WMT20 Quality Estimation Shared Task 1: Sentence-Level Direct Assessment. Our sys- tem is an ensemble model of four regression models based on XLM-RoBERTa with lan- guage tags. We ranked 4th in Pearson and 2nd in MAE and RMSE on a multilingual track. 1 Introduction Quality Estimation (QE) is a task of estimating translation quality without reference sentences (Gandrabur and Foster, 2003; Blatz et al., 2004; Specia et al., 2018). Automatic evaluation metrics based on reference sentences, such as BLEU (Pap- ineni et al., 2002), have contributed to improving translation quality on benchmark datasets. How- ever, in situations where machine translation (MT) is actually used, these metrics are sometimes un- able to assess the translation quality owing to the lack of reference sentences. The development of QE methods that are well correlated with manual evaluations enable users to decide whether to use the translation results as is, post-edit the results, or employ other machine translations. At the Conference on Machine Translation (WMT), there have been conducted several QE- related competitions such as the QE task (Fonseca et al., 2019) for estimating post-edit rate HTER (Snover et al., 2006) and the QE as a Metric task (Ma et al., 2019) for relative evaluations of trans- lation quality. This year, the WMT QE task held a new competition (Specia et al., 2020) on absolute evaluations of translation quality. In task 1, sen- tences are annotated with direct assessment (DA) scores as in the metrics task (Bojar et al., 2017). 1Tokyo Metropolitan University and Osaka University We have been working on the metrics task with an approach that uses pre-trained sentence encoders (Shimanaka et al., 2018, 2019). Shimanaka et al. (2018) employed InferSent (Conneau et al., 2017), Quick-Thought (Logeswaran and Lee, 2018), and Universal Sentence Encoder (Cer et al., 2018) as encoders, and achieved the highest performance in all to-English language pairs of WMT18 metrics shared task (Ma et al., 2018). Subsequently, Shi- manaka et al. (2019) employed BERT (Devlin et al., 2019) as an encoder to further improve the corre- lation with manual evaluations. In this study, we apply similar approaches to the QE task. However, to support both source and target languages, we employ XLM-RoBERTa2 (Conneau et al., 2020), a pre-trained multilingual sentence encoder. 2 WMT20 QE Shared Task 1 In the WMT20 QE task 1 (Sentence-Level Direct Assessment), participants predict translation qual- ity at the sentence level from pairs of source and MT output sentences. This task provides datasets for seven language pairs and sets up a multilingual track for a language-independent approach. 2.1 Datasets Source sentences have been collected from Wikipedia for six language pairs: English– German (En-De), English–Chinese (Eh-Zh), Romanian–English …

Content truncated to 500 words