A Test Set for Discourse Translation from Japanese to English

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 3704–3709 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 3704 A Test Set for Discourse Translation from Japanese to English Masaaki Nagata, Makoto Morishita NTT Communication Science Laboratories, NTT Corporation 2-4 Hikaridai Seika-cho Souraku-gun Kyoto-fu 619-0237 Japan {masaaki.nagata.et, makoto.morishita.gr}@hco.ntt.co.jp Abstract We made a test set for Japanese-to-English discourse translation to evaluate the power of context-aware machine translation. For each discourse phenomenon, we systematically collected examples where the translation of the second sentence depends on the first sentence. Compared with a previous study on test sets for English-to-French discourse translation (Bawden et al., 2018), we needed different approaches to make the data because Japanese has zero pronouns and represents different senses in different characters. We improved the translation accuracy using context-aware neural machine translation, and the improvement mainly reflects the betterment of the translation of zero pronouns. Keywords: discourse, translation, test 1. Introduction Translation accuracy for sentences has been greatly im- proved by neural machine translation (Cho et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2015; Luong et al., 2015; Vaswani et al., 2017). Although the accuracy of sentence-level machine translation is approaching hu- man parity (Hassan et al., 2018), human evaluators prefer the translations of human translators over those of machine translation in document-level evaluations (Samuel L¨aubli, 2018). For further improvement, research using discourse information is gaining attention (Tiedemann and Scherrer, 2017; Bawden et al., 2018; Voita et al., 2018; Maruf and Haffari, 2018; Miculicich et al., 2018; Voita et al., 2019a). The easiest method to use discourse information is to con- catenate both the previous and current sentences with a special token <CONCAT> as a separator and translate them using an ordinary sentence-based translation system (Tiedemann and Scherrer, 2017). This method, which is called 2-to-2, is known to be a reasonably strong baseline for discourse translation (Bawden et al., 2018; Voita et al., 2018). In this context, ordinary sentence-based translation is called 1-to-1. Some recent proposals have focused on multiple-encoders, where the input sentence and the con- text have different encoders. Bawden et al. (2018) used an RNN-based encoder-decoder and Voita et al. (2018) used the Transformer (Vaswani et al., 2017) for multiple- encoders. Others used a cache-based model to exploit document-level information (Wang et al., 2017; Kuang et al., 2018; Tu et al., 2018). One problem in the study of discourse translation is that by only using automatic accuracy measures such as BLEU, we cannot distinguish between what is solved and not solved by the proposed method. Bawden et al. (2018) therefore proposed discourse test sets for English-to-French transla- tion to evaluate the power of generating appropriate sen- tences based on the context information using the frame- work of a contrastive test set (Sennrich, 2017). For dis- course phenomena, they targeted coreference and coher- ence, and manually created a test set based on real examples in bilingual corpora. Voita et al. (2019b) made a contrastive test set for …

Content truncated to 500 words