A Challenge Set Approach to Evaluating Machine Translation

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2486–2496 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics A Challenge Set Approach to Evaluating Machine Translation Pierre Isabelle and Colin Cherry National Research Council Canada first.last@nrc-cnrc.gc.ca George Foster Google∗ fosterg@google.com Abstract Neural machine translation represents an exciting leap forward in translation qual- ity. But what longstanding weaknesses does it resolve, and which remain? We ad- dress these questions with a challenge set approach to translation evaluation and er- ror analysis. A challenge set consists of a small set of sentences, each hand-designed to probe a system’s capacity to bridge a particular structural divergence between languages. To exemplify this approach, we present an English–French challenge set, and use it to analyze phrase-based and neural systems. The resulting analy- sis provides not only a more fine-grained picture of the strengths of neural systems, but also insight into which linguistic phe- nomena remain out of reach. 1 Introduction The advent of neural techniques in machine trans- lation (MT) (Kalchbrenner and Blunsom, 2013; Cho et al., 2014; Sutskever et al., 2014) has led to profound improvements in MT quality. For “easy” language pairs such as English/French or English/Spanish in particular, neural (NMT) sys- tems are much closer to human performance than previous statistical techniques (Wu et al., 2016). This puts pressure on automatic evaluation met- rics such as BLEU (Papineni et al., 2002), which exploit surface-matching heuristics that are rela- tively insensitive to subtle differences. As NMT continues to improve, these metrics will inevitably lose their effectiveness. Another challenge posed by NMT systems is their opacity: while it was usually clear which phenomena were ill-handled ∗Work performed while at NRC. Src The repeated calls from his mother should have alerted us. Ref Les appels r´ep´et´es de sa m`ere auraient dˆu nous alerter. Sys Les appels r´ep´et´es de sa m`ere devraient nous avoir alert´es. Is the subject-verb agreement correct (y/n)? Yes Figure 1: Example challenge set question. by previous statistical systems—and why—these questions are more difficult to answer for NMT. We propose a new evaluation methodology cen- tered around a challenge set of difficult examples that are designed using expert linguistic knowl- edge to probe an MT system’s capabilities. This methodology is complementary to the standard practice of randomly selecting a test set from “real text,” which remains necessary in order to predict performance on new text. By concentrating on difficult examples, a challenge set is intended to provide a stronger signal to developers. Although we believe that the general approach is compatible with automatic metrics, we used manual evalua- tion for the work presented here. Our challenge set consists of short sentences that each focus on one particular phenomenon, which makes it easy to collect reliable manual assessments of MT out- put by asking direct yes-no questions. An example is shown in Figure 1. We generated a challenge set for English to French translation by canvassing areas of linguis- tic divergence between the two language pairs, es- pecially …

Content truncated to 500 words