Estimating Word Alignment Quality for SMT Reordering Tasks

Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 275–286, Baltimore, Maryland USA, June 26–27, 2014. c⃝2014 Association for Computational Linguistics Estimating Word Alignment Quality for SMT Reordering Tasks Sara Stymne J¨org Tiedemann Uppsala UniversityJoakim Nivre Department of Linguistics and Philology firstname.lastname@lingfil.uu.se Abstract Previous studies of the effect of word alignment on translation quality in SMT generally explore link level metrics only and mostly do not show any clear connec- tions between alignment and SMT qual- ity. In this paper, we specifically inves- tigate the impact of word alignment on two pre-reordering tasks in translation, us- ing a wider range of quality indicators than previously done. Experiments on German–English translation show that re- ordering may require alignment models different from those used by the core trans- lation system. Sparse alignments with high precision on the link level, for trans- lation units, and on the subset of cross- ing links, like intersected HMM models, are preferred. Unlike SMT performance the desired alignment characteristics are similar for small and large training data for the pre-reordering tasks. Moreover, we confirm previous research showing that the fuzzy reordering score is a useful and cheap proxy for performance on SMT re- ordering tasks. 1 Introduction Word alignment is a key component in all state-of- the-art statistical machine translation (SMT) sys- tems, and there has been some work exploring the connection between word alignment quality and translation quality (Och and Ney, 2003; Fraser and Marcu, 2007; Lambert et al., 2012). The standard way to evaluate word alignments in this context is by using metrics like alignment error rate (AER) and F-measure on the link level, and the general conclusion appears to be that translation quality benefits from alignments with high recall (rather than precision), at least for large training data. Al- though many other ways of measuring alignment quality have been proposed, such as working on translation units (Ahrenberg et al., 2000; Ayan and Dorr, 2006; Søgaard and Kuhn, 2009) or using link degree and related measures (Ahrenberg, 2010), these methods have not been used to study the re- lation between alignment and translation quality, with the exception of Lambert et al. (2012). Word alignment is also used for many other tasks besides translation, including term bank creation (Merkel and Foo, 2007), cross-lingual annotation projection for part-of-speech tagging (Yarowsky et al., 2001), semantic roles (Pado and Lapata, 2005), pronoun anaphora (Postolache et al., 2006), and cross-lingual clustering (T¨ackstr¨om et al., 2012). Even within SMT itself, there are tasks such as reordering that often make crucial use of word alignments. For instance, source lan- guage reordering commonly relies on rules learnt automatically from word-aligned data (e.g., Xia and McCord (2004)). As far as we know, no one has studied the impact of alignment quality on these additional tasks, and it seems to be tacitly assumed that alignments that are good for transla- tion are also good for other tasks. In this paper we set out to explore the impact of alignment quality on two pre-reordering tasks …

Content truncated to 500 words