Estimating the Quality of Translated User-Generated Content

International Joint Conference on Natural Language Processing, pages 1167–1173, Nagoya, Japan, 14-18 October 2013. Estimating the Quality of Translated User-Generated Content Raphael Rubino†‡, Jennifer Foster†, Rasoul Samad Zadeh Kaljahi†‡, Johann Roturier‡ and Fred Hollowood‡ †NCLT, School of Computing, Dublin City University, Ireland {rrubino, jfoster, rkaljahi}@computing.dcu.ie ‡Symantec Research Labs, Dublin, Ireland {johann_roturier, fhollowood}@symantec.com Abstract Previous research on quality estimation for machine translation has demonstrated the possibility of predicting the transla- tion quality of well-formed data. We present a first study on estimating the translation quality of user-generated con- tent. Our dataset contains English tech- nical forum comments which were trans- lated into French by three automatic sys- tems. These translations were rated in terms of both comprehensibility and fi- delity by human annotators. Our exper- iments show that tried-and-tested quality estimation features work well on this type of data but that extending this set can be beneficial. We also show that the perfor- mance of particular types of features de- pends on the type of system used to pro- duce the translation. 1 Introduction Quality Estimation (QE) involves judging the cor- rectness of a system output given an input with- out any output reference. Substantial progress has been made on QE for Machine Translation (MT), but research has been mainly conducted on well- formed, edited text (Blatz et al., 2003; Ueffing et al., 2003; Raybaud et al., 2009; Specia et al., 2009). We turn our attention to estimating the quality of user-generated content (UGC) transla- tion – a particularly relevant use of QE since the translation process is likely to be affected by the noisy nature of the input, particularly if the MT system is trained on well-formed text. The source language content is collected from an IT Web forum in English and translated into French by three automatic systems. For each MT system, the produced translation is manually evaluated following two criteria: the translation comprehensibility and fidelity. We evaluate sev- eral feature sets on the UGC dataset including the baseline suggested by the organisers of the WMT 2012 QE for MT shared task (Callison-Burch et al., 2012) and a feature set designed to model typ- ical characteristics of forum text. The novel contributions of the paper are: 1) testing the WMT QE for MT Shared Task base- line feature set on the UGC dataset and demon- strating its portability, 2) introducing new features which contribute to significant performance gains on both QE tasks, and 3) building three different QE systems using three different MT systems and showing that the usefulness of a feature type de- pends on the MT system although better perfor- mance can be achieved by training a QE system on the combined output of the three systems. The paper is organised as follows. Related work on QE for MT is described in Section 2, followed in Section 3 by a description of the dataset. We de- scribe the QE features in Section 4 and present the results of our experiments in Section 5. A discus- sion of …

Content truncated to 500 words