A Quantitative Analysis of Reordering Phenomena

Proceedings of the Fourth Workshop on Statistical Machine Translation , pages 197–205, Athens, Greece, 30 March – 31 March 2009. c⃝2009 Association for Computational Linguistics A Quantitative Analysis of Reordering Phenomena Alexandra Birch Phil Blunsom Miles Osborne a.c.birch-mayne@sms.ed.ac.uk pblunsom@inf.ed.ac.uk miles@inf.ed.ac.uk University of Edinburgh 10 Crichton Street Edinburgh, EH8 9AB, UK Abstract Reordering is a serious challenge in sta- tistical machine translation. We propose a method for analysing syntactic reorder- ing in parallel corpora and apply it to un- derstanding the differences in the perfor- mance of SMT systems. Results at recent large-scale evaluation campaigns show that synchronous grammar-based statisti- cal machine translation models produce superior results for language pairs such as Chinese to English. However, for language pairs such as Arabic to English, phrase- based approaches continue to be competi- tive. Until now, our understanding of these results has been limited to differences in BLEU scores. Our analysis shows that cur- rent state-of-the-art systems fail to capture the majority of reorderings found in real data. 1 Introduction Reordering is a major challenge in statistical ma- chine translation. Reordering involves permuting the relative word order from source sentence to translation in order to account for systematic dif- ferences between languages. Correct word order is important not only for the fluency of output, it also affects word choice and the overall quality of the translations. In this paper we present an automatic method for characterising syntactic reordering found in a parallel corpus. This approach allows us to analyse reorderings quantitatively, based on their number and span, and qualitatively, based on their relation- ship to the parse tree of one sentence. The methods we introduce are generally applicable, only requir- ing an aligned parallel corpus with a parse over the source or the target side, and can be extended to allow for more than one reference sentence and derivations on both source and target sentences. Using this method, we are able to compare the re- ordering capabilities of two important translation systems: a phrase-based model and a hierarchical model. Phrase-based models (Och and Ney, 2004; Koehn et al., 2003) have been a major paradigm in statistical machine translation in the last few years, showing state-of-the-art performance for many language pairs. They search all possible re- orderings within a restricted window, and their output is guided by the language model and a lexicalised reordering model (Och et al., 2004), both of which are local in scope. However, the lack of structure in phrase-based models makes it very difficult to model long distance movement of words between languages. Synchronous grammar models can encode structural mappings between languages which al- low complex, long distance reordering. Some grammar-based models such as the hierarchical model (Chiang, 2005) and the syntactified target language phrases model (Marcu et al., 2006) have shown better performance than phrase-based mod- els on certain language pairs. To date our understanding of the variation in re- ordering performance between phrase-based and synchronous grammar models has been limited to relative BLEU scores. However, Callison-Burch et al. …

Content truncated to 500 words