Evaluation of Machine Translation Performance Across Multiple Genres and Languages

Evaluation of Machine Translation Performance Across Multiple Genres and Languages Marlies van der Wees1, Arianna Bisazza2, Christof Monz1 1Informatics Institute, University of Amsterdam 2Leiden Institute of Advanced Computer Science, Leiden University Abstract In this paper, we present evaluation corpora covering four genres for four language pairs that we harvested from the web in an automated fashion. We use these multi-genre benchmarks to evaluate the impact of genre differences on machine translation (MT). We observe that BLEU score differences between genres can be large and that, for all genres and all language pairs, translation quality improves when using four genre-optimized systems rather than a single genre-agnostic system. Finally, we train and use genre classifiers to route test documents to the most appropriate genre systems. The results of these experiments show that our multi-genre benchmarks can serve to advance research on text genre adaptation for MT. Keywords: Machine translation, parallel benchmarks, text genres, genre adaptation 1. Introduction Text genre differences have shown to affect the output qual- ity of statistical machine translation (SMT) systems: SMT systems trained on one genre often achieve poor perfor- mance when used for translating another genre (Foster and Kuhn, 2007; Matsoukas et al., 2009; Wang et al., 2012, among others). In addition, even if different genres in a test set are both present in equal amounts in the bilingual train- ing data, performance differences between the test genres can be large, mostly due to poor model coverage for cer- tain genres (van der Wees et al., 2015a; van der Wees et al., 2015b). In this paper, we evaluate the impact of genre differences on phrase-based SMT for a diverse set of language pairs, covering both commonly and rarely studied language pairs. For common language pairs, parallel training data is abun- dant but limited to a few genres such as parliamentary and legal proceedings. For low-resource languages the situa- tion is—by definition—much worse, with very few to no bilingual corpora available. To alleviate this problem, we present in this paper novel parallel training and evaluation corpora covering four genres for four language pairs that we automatically harvested from the web. Next, we evaluate the usefulness of the newly collected bilingual resources by exploiting them for genre adapta- tion of SMT systems. Most existing adaptation approaches depend on the availability of provenance information and make the strong assumption that a translation task has known domain, genre or topic that is exploited to adapt the system (Matsoukas et al., 2009; Foster et al., 2010; Bisazza and Federico, 2012; Chen et al., 2013; Chen et al., 2014; Kobus et al., 2016; Sennrich et al., 2016; Freitag and Al- Onaizan, 2016; Chu et al., 2017, among others). While this is a fair assumption in a controlled research setting, it is less realistic in real world applications, such as general-purpose online MT services. In this paper, we provide the SMT system with a test document of unknown origin, and we show that we can use automatic genre classification to guide each test document …

Content truncated to 500 words