Choosing the Right Translation: A Syntactically Informed Classification Approach

Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 1153–1160 Manchester, August 2008 Choosing the Right Translation: A Syntactically Informed Classification Approach Simon Zwarts Centre for Language Technology Macquarie University Sydney, Australia szwarts@ics.mq.edu.au Mark Dras Centre for Language Technology Macquarie University Sydney, Australia madras@ics.mq.edu.au Abstract One style of Multi-Engine Machine Translation architecture involves choos- ing the best of a set of outputs from different systems. Choosing the best translation from an arbitrary set, even in the presence of human references, is a difficult problem; it may prove better to look at mechanisms for making such choices in more restricted contexts. In this paper we take a classification- based approach to choosing between candidates from syntactically informed translations. The idea is that using multiple parsers as part of a classifier could help detect syntactic problems in this context that lead to bad transla- tions; these problems could be detected on either the source side—perhaps sen- tences with difficult or incorrect parses could lead to bad translations—or on the target side—perhaps the output quality could be measured in a more syntactically informed way, looking for syntactic abnormalities. We show that there is no evidence that the source side information is useful. However, a target-side classifier, when used to identify particularly bad trans- lation candidates, can lead to signifi- cant improvements in Bleu score. Im- provements are even greater when com- bined with existing language and align- ment model approaches. c⃝ 2008. Licensed under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported license (http://creativecommons.org/licenses/by-nc- sa/3.0/). Some rights reserved. 1 Introduction It is fairly safe to say that whenever there are multiple approaches to solving a problem in Artificial Intelligence, the idea of trying to find a better solution by combining those ap- proaches has been proposed: blackboard archi- tectures, ensemble methods for machine learn- ing, and so on. In Machine Translation (MT), there is a long tradition of combining multiple machine translations, as through a Multi-Engine MT (MEMT) architecture; the origins of this are generally credited to Frederking and Nirenburg (1994). One way of dividing up such systems is into those that take the whole output from multiple systems and judge between them to select the best candidate, and those that com- bine elements of the outputs to construct a best candidate. Deciding between whole sentence level out- puts looks like a classical classification prob- lem. Of course, deciding between MT out- puts in the general case is a problem that cur- rently has no good solution, and is unlikely to in the near future: Bleu (and similar met- rics) require one or more reference texts to dis- tinguish between candidate outputs with the level of accuracy that they achieve, and even then they are open to substantial criticism (Callison-Burch et al., 2006). However, there are reasons to think that there is some promise in considering this as a classification problem. Corston-Oliver et al. (2001) build a classifier to distinguish between human and machine trans- lations with an 80% accuracy. Several …

Content truncated to 500 words