Upping the Ante: Towards a Better Benchmark for Chinese-to-English Machine Translation

Upping the Ante: Towards a Better Benchmark for Chinese-to-English Machine Translation Christian Hadiwinoto, Hwee Tou Ng Department of Computer Science National University of Singapore {chrhad, nght}@comp.nus.edu.sg Abstract There are many machine translation (MT) papers that propose novel approaches and show improvements over their self-defined baselines. The experimental setting in each paper often differs from one another. As such, it is hard to determine if a proposed approach is really useful and advances the state of the art. Chinese-to-English translation is a common translation direction in MT papers, although there is not one widely accepted experimental setting in Chinese-to-English MT. Our goal in this paper is to propose a benchmark in evaluation setup for Chinese-to-English machine translation, such that the effectiveness of a new proposed MT approach can be directly compared to previous approaches. Towards this end, we also built a highly competitive state-of-the-art MT system trained on a large-scale training set. Our system outperforms reported results on NIST OpenMT test sets in almost all papers published in major conferences and journals in computational linguistics and artificial intelligence in the past 11 years. We argue that a standardized benchmark on data and performance is important for meaningful comparison. Keywords: machine translation, benchmark, Chinese-to-English 1. Introduction Over the years, there have been many published papers on machine translation (MT), proposing novel ideas by sho- wing improvements over certain baselines. However, a pa- per often has a different experimental setup from the others. These variations include the approach (algorithm) and data- set. Ideally, research and development work on MT should be based on a benchmark of system setup with good per- formance. Otherwise, there is no support in asserting that a proposed method advances the state of the art. Unfortunately, for Chinese-to-English MT, two widely spo- ken human languages and one of the most well-studied language translation directions in MT, there is no widely accepted standard benchmark for evaluation, comprising a standardized training set, development set, and test set. Throughout the past decade, Chinese-to-English translation has been most commonly performed on NIST OpenMT1 test sets, trained on parallel and monolingual corpora from the Linguistic Data Consortium (LDC)2. Our goal in this paper is to propose a benchmark in evalua- tion setup for Chinese-to-English machine translation, such that the effectiveness of a new proposed MT approach can be directly compared to previous approaches. Towards this end, we also built a highly competitive state-of-the-art MT system trained on a large-scale training set. Our system outperforms reported results on NIST OpenMT test sets in almost all papers published in major conferences and jour- nals in computational linguistics and artificial intelligence in the past 11 years. The rest of this paper is organized as follows. Section 2 describes our MT approach. Section 3 elaborates our expe- rimental setup. Section 4 presents our experimental results. Section 5 describes related work. Finally, Section 6 gives 1www.nist.gov/itl/iad/mig/ open-machine-translation-evaluation 2catalog.ldc.upenn.edu the conclusion. 2. Neural Machine Translation We built a neural machine translation (NMT) system based on the encoder-decoder approach with attention …

Content truncated to 500 words