Training a Parser for Machine Translation Reordering

Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 183–192, Edinburgh, Scotland, UK, July 27–31, 2011. c⃝2011 Association for Computational Linguistics Training a Parser for Machine Translation Reordering Jason Katz-Brown Slav Petrov Ryan McDonald Franz Och David Talbot Hiroshi Ichikawa Masakazu Seno Hideto Kazawa Google {jasonkb|slav|ryanmcd|och|talbot|ichikawa|seno|kazawa}@google.com Abstract We propose a simple training regime that can improve the extrinsic performance of a parser, given only a corpus of sentences and a way to automatically evaluate the extrinsic quality of a candidate parse. We apply our method to train parsers that excel when used as part of a reordering component in a statistical ma- chine translation system. We use a corpus of weakly-labeled reference reorderings to guide parser training. Our best parsers contribute significant improvements in subjective trans- lation quality while their intrinsic attachment scores typically regress. 1 Introduction The field of syntactic parsing has received a great deal of attention and progress since the creation of the Penn Treebank (Marcus et al., 1993; Collins, 1997; Charniak, 2000; McDonald et al., 2005; Petrov et al., 2006; Nivre, 2008). A common— and valid—criticism, however, is that parsers typi- cally get evaluated only on Section 23 of the Wall Street Journal portion of the Penn Treebank. This is problematic for many reasons. As previously ob- served, this test set comes from a very narrow do- main that does not necessarily reflect parser perfor- mance on text coming from more varied domains (Gildea, 2001), especially web text (Foster, 2010). There is also evidence that after so much repeated testing, parsers are indirectly over-fitting to this set (Petrov and Klein, 2007). Furthermore, parsing was never meant as a stand-alone task, but is rather a means to an end, towards the goal of building sys- tems that can process natural language input. This is not to say that parsers are not used in larger systems. All to the contrary, as parsing technology has become more mature, parsers have become ef- ficient and accurate enough to be useful in many natural language processing systems, most notably in machine translation (Yamada and Knight, 2001; Galley et al., 2004; Xu et al., 2009). While it has been repeatedly shown that using a parser can bring net gains on downstream application quality, it is of- ten unclear how much intrinsic parsing accuracy ac- tually matters. In this paper we try to shed some light on this is- sue by comparing different parsers in the context of machine translation (MT). We present experiments on translation from English to three Subject-Object- Verb (SOV) languages,1 because those require ex- tensive syntactic reordering to produce grammatical translations. We evaluate parse quality on a num- ber of extrinsic metrics, including word reordering accuracy, BLEU score and a human evaluation of fi- nal translation quality. We show that while there is a good correlation between those extrinsic metrics, parsing quality as measured on the Penn Treebank is not a good indicator of the final downstream ap- plication quality. Since the word reordering metric …

Content truncated to 500 words