Quality Estimation of English-French Machine Translation: A Detailed Study of the Role of Syntax

Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 2052–2063, Dublin, Ireland, August 23-29 2014. Quality Estimation of English-French Machine Translation: A Detailed Study of the Role of Syntax Rasoul Kaljahi†‡, Jennifer Foster†, Raphael Rubino†‡, Johann Roturier‡ †NCLT, School of Computing, Dublin City University, Ireland {rkaljahi, jfoster, rrubino}@computing.dcu.ie ‡Symantec Research Labs, Dublin, Ireland johann roturier@symantec.com Abstract We investigate the usefulness of syntactic knowledge in estimating the quality of English-French translations. We find that dependency and constituency tree kernels perform well but the error rate can be further reduced when these are combined with hand-crafted syntactic features. Both types of syntactic features provide information which is complementary to tried-and-tested non- syntactic features. We then compare source and target syntax and find that the use of parse trees of machine translated sentences does not affect the performance of quality estimation nor does the intrinsic accuracy of the parser itself. However, the relatively flat structure of the French Treebank does appear to have an adverse effect, and this is significantly improved by simple transformations of the French trees. Finally, we provide further evidence of the usefulness of these transformations by applying them in a separate task – parser accuracy prediction. 1 Introduction Quality Estimation (QE) for Machine Translation (MT) involves judging the correctness of the output of an MT system given an input and no reference translation (Blatz et al., 2003; Ueffing et al., 2003; Specia et al., 2009). An accurate QE-for-MT system would mean that reliable decisions could be made regarding whether to publish a machine translation as is or to re-direct it to a translator, either for post- editing or to be translated from scratch. The scores produced by a QE system can also be used to choose between translations, in a system combination framework or in n-best list reranking. The work presented here takes place in the context of a wider study, the aim of which is to develop an English-French QE system so that technical support material that is produced on a daily basis by a company’s English- speaking customers can be translated automatically into French and made available with confidence to the company’s French-speaking customer base. It is reasonable to assume that syntactic features are useful in QE for MT as a way of capturing the syntactic complexity of the source sentence, the grammaticality of the target translation and the syntactic symmetry between the source sentence and its translation. This assumption has been borne out by previous research which has demonstrated the usefulness of syntactic features for English-Spanish QE (Hardmeier et al., 2012; Rubino et al., 2012). We focus more closely on understanding the role of syntax by comparing the use of hand-crafted features and tree kernels (Collins and Duffy, 2002; Moschitti, 2006), and by teasing apart the contribution of target and source syntax. We find that both tree kernels and manually engineered features produce statistically significantly better results than a strong set of non-syntactic features provided as a baseline by the organisers …

Content truncated to 500 words