PheMT: A Phenomenon-wise Dataset for Machine Translation Robustness on User-Generated Contents

Proceedings of the 28th International Conference on Computational Linguistics, pages 5929–5943 Barcelona, Spain (Online), December 8-13, 2020 5929 PheMT: A Phenomenon-wise Dataset for Machine Translation Robustness on User-Generated Contents Ryo Fujii1, Masato Mita2,1, Kaori Abe1, Kazuaki Hanawa2,1, Makoto Morishita3,1, Jun Suzuki1,2, Kentaro Inui1,2 1Tohoku University, 2RIKEN, 3NTT Communication Science Laboratories {r-fujii, abe-k, jun.suzuki}@ecei.tohoku.ac.jp, {masato.mita, kazuaki.hanawa}@riken.jp, makoto.morishita.gr@hco.ntt.co.jp, inui@ecei.tohoku.ac.jp Abstract Neural Machine Translation (NMT) has shown drastic improvement in its quality when translat- ing clean input, such as text from the news domain. However, existing studies suggest that NMT still struggles with certain kinds of input with considerable noise, such as User-Generated Con- tents (UGC) on the Internet. To make better use of NMT for cross-cultural communication, one of the most promising directions is to develop a model that correctly handles these expressions. Though its importance has been recognized, it is still not clear as to what creates the great gap in performance between the translation of clean input and that of UGC. To answer the question, we present a new dataset, PheMT, for evaluating the robustness of MT systems against specific linguistic phenomena in Japanese-English translation. Our experiments with the created dataset revealed that not only our in-house models but even widely used off-the-shelf systems are greatly disturbed by the presence of certain phenomena. 1 Introduction The advancement of Neural Machine Translation (NMT) has brought great improvement in translation quality when translating clean input, such as text from the news domain (Luong et al., 2015; Vaswani et al., 2017), and it was recently claimed that NMT has even achieved human parity in certain language pairs (Hassan et al., 2018; Barrault et al., 2019). Despite its remarkable advancements, the applicability of NMT over User-Generated Contents (UGC), such as social media text, still remains limited (Michel and Neubig, 2018; Berard et al., 2019a). Since UGC are prevailing in our real-life communication, it is undoubtedly one of the challenges we need to overcome to make MT systems invaluable for promoting cross-cultural communication. Recently, with the increasing interest in handling UGC, a shared task was organized to measure how well MT systems adapt to those texts (Li et al., 2019). However, the way in which they evaluate systems is just giving an overall score to a dataset, which is the same as traditional MT evaluation (Figure 1a). The overall score does not provide precise information for understanding what leads to the huge performance gap between the translation of clean input and that of UGC. To find a clue for improving the performance of MT systems on UGC, we need a solid basis for more detailed error analysis. As a first step towards a more refined evaluation of MT systems on UGC, we present a new dataset, PheMT: Phenomenon-wise Dataset for Machine Translation Robustness, designed for phenomenon- wise evaluation in Japanese-English translation (Figure 1b). More specifically, we create a set of datasets, each of which provides a focused evaluation on one of four linguistic phenomena commonly seen on UGC, i.e., Proper Noun, Abbreviated Noun, Colloquial Expression …

Content truncated to 500 words