PROTEST: A Test Suite for Evaluating Pronouns in Machine Translation

PROTEST: A Test Suite for Evaluating Pronouns in Machine Translation Liane Guillou and Christian Hardmeier University of Edinburgh, University of Uppsala L.K.Guillou@sms.ed.ac.uk, christian.hardmeier@lingfil.uu.se Abstract We present PROTEST, a test suite for the evaluation of pronoun translation by MT systems. The test suite comprises 250 hand-selected pronoun tokens and an automatic evaluation method which compares the translations of pronouns in MT output with those in the reference translation. Pronoun translations that do not match the reference are referred for manual evaluation. PROTEST is designed to support analysis of system performance at the level of individual pronoun groups, rather than to provide a single aggregate measure over all pronouns. We wish to encourage detailed analyses to highlight issues in the handling of specific linguistic mechanisms by MT systems, thereby contributing to a better understanding of those problems involved in translating pronouns. We present two use cases for PROTEST: a) for measuring improvement/degradation of an incremental system change, and b) for comparing the performance of a group of systems whose design may be largely unrelated. Following the latter use case, we demonstrate the application of PROTEST to the evaluation of the systems submitted to the DiscoMT 2015 shared task on pronoun translation. Keywords: Evaluation, pronouns, machine translation 1. Motivation In most current statistical machine translation (SMT) meth- ods, output words are generated in correspondence with the input words, according to word-alignments found at train- ing time. In addition to word-alignments, only very limited context information is taken into account in the generation process. While the approach works well for content words, it does not for function words, such as pronouns and nega- tion markers, which are critical to meaning (Hardmeier et al., 2015; Hardmeier et al., 2013; Nov´ak et al., 2013; Guil- lou, 2012). Pronouns have different functions, and their use varies be- tween languages. Some pronouns function as referring el- ements, creating a link to an element occurring elsewhere in the discourse. Others are simply to ensure a grammati- cal sentence. For example pleonastic pronouns, such as the “it” in “It is raining” or “il” in “Il pleut”, are used to fill the subject position. In many languages, pronouns are morpho- logically marked for categories such as gender and number, subject to certain agreement constraints that must be sat- isfied according to the rules of the target language. This is mostly a problem for referring pronouns, where generat- ing the correct form requires identifying what the pronoun refers to (anaphora resolution). Evaluation poses a particular problem for researchers inter- ested in pronoun generation in machine translation (MT). Owing to the cost and difficulty of manual evaluation (in- cluding manual post-editing based methods as a means to assess MT quality), MT researchers rely on automatic eval- uation metrics such as BLEU (Papineni et al., 2002) to guide their development efforts. Most automatic metrics as- sume that overlap of the MT output with a human-generated reference translation may be used as a proxy for correct- ness. In the case of anaphoric pronouns, this assumption …

Content truncated to 500 words