Tree Kernels for Machine Translation Quality Estimation

Proceedings of the 7th Workshop on Statistical Machine Translation, pages 109–113, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics Tree Kernels for Machine Translation Quality Estimation Christian Hardmeier and Joakim Nivre and J¨org Tiedemann Uppsala University Department of Linguistics and Philology Box 635, 751 26 Uppsala, Sweden firstname.lastname@lingfil.uu.se Abstract This paper describes Uppsala University’s submissions to the Quality Estimation (QE) shared task at WMT 2012. We present a QE system based on Support Vector Machine re- gression, using a number of explicitly defined features extracted from the Machine Transla- tion input, output and models in combination with tree kernels over constituency and de- pendency parse trees for the input and output sentences. We confirm earlier results suggest- ing that tree kernels can be a useful tool for QE system construction especially in the early stages of system design. 1 Introduction The goal of the WMT 2012 Quality Estimation (QE) shared task (Callison-Burch et al., 2012) was to create automatic systems to judge the quality of the translations produced by a Statistical Ma- chine Translation (SMT) system given the input text, the proposed translations and information about the models used by the SMT system. The shared task organisers provided a training set of 1832 sen- tences drawn from earlier WMT Machine Transla- tion test sets, translated from English to Spanish with a phrase-based SMT system, along with the models used and diagnostic output produced by the SMT system as well as manual translation quality annotations on a 1–5 scale for each sentence. Ad- ditionally, a set of 17 baseline features was made available to the participants. Systems were evalu- ated on a test set of 422 sentences annotated in the same way. Uppsala University submitted two systems to this shared task. Our systems were fairly successful and achieved results that were outperformed by only one competing group. They improve over the baseline performance in two ways, building on and extend- ing earlier work by Hardmeier (2011), on which the system description in the following sections is partly based: On the one hand, we enhance the set of 17 baseline features provided by the organisers with another 82 explicitly defined features. On the other hand, we use syntactic tree kernels to extract implicit features from constituency and dependency parse trees over the input sentences and the Machine Translation (MT) output. The experimental results confirm the findings of our earlier work, showing tree kernels to be a valuable tool for rapid prototyp- ing of QE systems. 2 Features Our QE systems used two types of features: On the one hand, we used a set of explicit features that were extracted from the data before running the Ma- chine Learning (ML) component. On the other hand, syntactic parse trees of the MT input and output sentences provided implicit features that were com- puted directly by the ML component using tree ker- nels. 2.1 Explicit features Both of the QE systems we submitted to the shared task used the complete set of 17 …

Content truncated to 500 words