Going Beyond AER: An Extensive Analysis of Word Alignments and Their Impact on MT

Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL, pages 9–16, Sydney, July 2006. c⃝2006 Association for Computational Linguistics Going Beyond AER: An Extensive Analysis of Word Alignments and Their Impact on MT Necip Fazil Ayan and Bonnie J. Dorr Institute of Advanced Computer Studies (UMIACS) University of Maryland College Park, MD 20742 {nfa,bonnie}@umiacs.umd.edu Abstract This paper presents an extensive evalua- tion of five different alignments and in- vestigates their impact on the correspond- ing MT system output. We introduce new measures for intrinsic evaluations and examine the distribution of phrases and untranslated words during decoding to identify which characteristics of different alignments affect translation. We show that precision-oriented alignments yield better MT output (translating more words and using longer phrases) than recall- oriented alignments. 1 Introduction Word alignments are a by-product of statistical machine translation (MT) and play a crucial role in MT performance. In recent years, researchers have proposed several algorithms to generate word alignments. However, evaluating word alignments is difficult because even humans have difficulty performing this task. The state-of-the art evaluation metric— alignment error rate (AER)—attempts to balance the precision and recall scores at the level of alignment links (Och and Ney, 2000). Other met- rics assess the impact of alignments externally, e.g., different alignments are tested by comparing the corresponding MT outputs using automated evaluation metrics (e.g., BLEU (Papineni et al., 2002) or METEOR (Banerjee and Lavie, 2005)). However, these studies showed that AER and BLEU do not correlate well (Callison-Burch et al., 2004; Goutte et al., 2004; Ittycheriah and Roukos, 2005). Despite significant AER improvements achieved by several researchers, the improvements in BLEU scores are insignificant or, at best, small. This paper demonstrates the difficulty in assess- ing whether alignment quality makes a difference in MT performance. We describe the impact of certain alignment characteristics on MT perfor- mance but also identify several alignment-related factors that impact MT performance regardless of the quality of the initial alignments. In so doing, we begin to answer long-standing questions about the value of alignment in the context of MT. We first evaluate 5 different word alignments intrinsically, using: (1) community-standard metrics—precision, recall and AER; and (2) a new measure called consistent phrase error rate (CPER). Next, we observe the impact of differ- ent alignments on MT performance. We present BLEU scores on a phrase-based MT system, Pharaoh (Koehn, 2004), using five different align- ments to extract phrases. We investigate the im- pact of different settings for phrase extraction, lex- ical weighting, maximum phrase length and train- ing data. Finally, we present a quantitative analy- sis of which phrases are chosen during the actual decoding process and show how the distribution of the phrases differ from one alignment into another. Our experiments show that precision-oriented alignments yield better phrases for MT than recall- oriented alignments. Specifically, they cover a higher percentage of our test sets and result in fewer untranslated words and selection of longer phrases during decoding. The next section …

Content truncated to 500 words