Improving Machine Translation Quality Prediction with Syntactic Tree Kernels

Improving Machine Translation Quality Prediction with Syntactic Tree Kernels Christian Hardmeier Uppsala universitet Inst. f¨or lingvistik och filologi SE-751 26 Uppsala christian.hardmeier@lingfil.uu.se Abstract We investigate the problem of predicting the quality of a given Machine Translation (MT) output segment as a binary classifi- cation task. In a study with four different data sets in two text genres and two lan- guage pairs, we show that the performance of a Support Vector Machine (SVM) clas- sifier can be improved by extending the feature set with implicitly defined syn- tactic features in the form of tree ker- nels over syntactic parse trees. Moreover, we demonstrate that syntax tree kernels achieve surprisingly high performance lev- els even without additional features, which makes them suitable as a low-effort initial building block for an MT quality estima- tion system. 1 Introduction Even though automatic high-quality translation for general domains is still far beyond the reach of current Statistical Machine Translation (SMT) systems, recent systems achieve levels of perfor- mance that make them viable for use as core el- ements in commercial translation processes. In certain text genres and with systems trained on sufficient amounts of in-domain data, producing raw translations with an SMT system and having human translators post-edit them can be a more cost-effective way of obtaining production-quality translations than doing fully human translation. One example for this is the domain of TV film sub- titles, where a good SMT system can output trans- lations closely similar to a translation produced by a professional translator for more than 30 % of the subtitles under favourable conditions (Volk et al., 2010). Still, despite the large number of good translations, other subtitles in the SMT output can c⃝ 2011 European Association for Machine Translation. be of very low quality, placing an unnecessary bur- den on the post-editors, who have to take a deci- sion to discard the bad raw translation before trans- lating the subtitle from scratch anyway. In order to reduce the effort post-editors have to spend on acceptance decisions and make subtitle post-editing a more pleasant experience, it would be desirable to predict the quality of a segment automatically given the input, the output and the models of the SMT system, a task that has gone un- der the name of confidence prediction in the litera- ture. While the SMT system itself internally scores alternative translations of each segment to find the best one, raw SMT scores are not sufficient as a confidence measure. As conditional probabilities, they are not comparable across sentences. Fur- thermore, they are not properly normalised by the SMT decoder since performing the required nor- malisation would render decoding intractable. Us- ing a decoder-external confidence prediction mod- ule also makes it possible to use certain features which by their nature are difficult to integrate in the left-to-right beam search framework typical of current phrase-based SMT decoders. Previous research in Machine Translation (MT) confidence estimation has used a variety of dif- ferent features representing characteristics of the input and the output …

Content truncated to 500 words