Parser Accuracy in Quality Estimation of Machine Translation: A Tree Kernel Approach

International Joint Conference on Natural Language Processing, pages 1092–1096, Nagoya, Japan, 14-18 October 2013. Parser Accuracy in Quality Estimation of Machine Translation: A Tree Kernel Approach Rasoul Samad Zadeh Kaljahi†‡, Jennifer Foster†, Raphael Rubino†‡, Johann Roturier‡ and Fred Hollowood‡ †NCLT, School of Computing, Dublin City University, Ireland {rkaljahi, jfoster, rrubino}@computing.dcu.ie ‡Symantec Research Labs, Dublin, Ireland {johann roturier, fhollowood}@symantec.com Abstract We report on experiments designed to in- vestigate the role of syntactic features in the task of quality estimation for machine translation, focusing on the effect of parser accuracy. Tree kernels are used to predict the segment-level BLEU score of English- French translations. In order to examine the effect of the accuracy of the parse tree on the accuracy of the quality estimation system, we experiment with various pars- ing systems which differ substantially with respect to their Parseval f-scores. We find that it makes very little difference which system we choose to use in the quality esti- mation task – this effect is particularly ap- parent for source-side English parse trees. 1 Introduction Much research has been carried out on quality es- timation (QE) for machine translation (MT) (Blatz et al., 2003; Ueffing et al., 2003; Specia et al., 2009; Callison-Burch et al., 2012), with the aim of solving the problem of how to accurately assess the quality of a translation without access to a ref- erence translation. Approaches differ with respect to the nature of the quality scores being estimated (binary, 5-point or real-valued scales; human eval- uations versus automatic metrics), the learning al- gorithms used or the feature set chosen to repre- sent the translation pairs. The aspect of the task that we focus on is the feature set, and, in particu- lar, the role of syntactic features. We ask the fol- lowing: To what extent is QE for MT influenced by the quality of the syntactic information provided to it? Does the accuracy of the parsing model used to provide the syntactic features influence the accu- racy of the QE system? We compare two pairs of parsing systems which differ with respect to their Parseval f-scores by around 17 absolute points in a QE system for English-French MT and find that it makes little difference which system we use. 2 Related Work Features extracted from parser output have been used before in QE for MT. Quirk (2004) uses a feature which indicates whether a full parse for a sentence can be found. Gamon et al. (2005) use part-of-speech (POS) tag trigrams, CFG produc- tion rules and features derived from a dependency analysis of the MT output. Specia and Gim´enez (2010) use POS tag language model probabilities of the MT output 3-grams. Hardmeier et al. (2012) combine syntactic tree kernels with surface fea- tures to produce a system which was ranked sec- ond in the WMT 2012 shared task on QE for MT (Callison-Burch et al., 2012). Rubino et al. (2012) explore source syntactic features extracted from the output of a hand-crafted broad-coverage gram- mar/parser and a statistical …

Content truncated to 500 words