The TALP-UPC Approach to System Selection: Asiya Features and Pairwise Classification Using Random Forests

Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 359–364, Sofia, Bulgaria, August 8-9, 2013 c⃝2013 Association for Computational Linguistics The TALP-UPC Approach to System Selection: ASIYA Features and Pairwise Classification using Random Forests Llu´ıs Formiga1, Meritxell Gonz`alez1, Alberto Barr´on-Cede˜no1,2 Jos´e A. R. Fonollosa1 and Llu´ıs M`arquez1 1 TALP Research Center, Universitat Polit`ecnica de Catalunya, Spain 2 Facultad de Inform´atica, Universidad Polit´ecnica de Madrid, Spain {lluis.formiga,jose.fonollosa}@upc.edu, {mgonzalez,albarron,lluism}@lsi.upc.edu Abstract This paper describes the TALP-UPC par- ticipation in the WMT’13 Shared Task on Quality Estimation (QE). Our partic- ipation is reduced to task 1.2 on System Selection. We used a broad set of fea- tures (86 for German-to-English and 97 for English-to-Spanish) ranging from stan- dard QE features to features based on pseudo-references and semantic similarity. We approached system selection by means of pairwise ranking decisions. For that, we learned Random Forest classifiers es- pecially tailored for the problem. Evalua- tion at development time showed consider- ably good results in a cross-validation ex- periment, with Kendall’s τ values around 0.30. The results on the test set dropped significantly, raising different discussions to be taken into account. 1 Introduction In this paper we discuss the TALP-UPC1 partici- pation in the WMT’13 Shared Task on Quality Es- timation (QE). Our participation is circumscribed to task 1.2, which deals with System Selection. Concretely, we were required to rank up to five al- ternative translations for the same source sentence produced by multiple MT systems, in the absence of any reference translation. We used a broad set of features; mainly avail- able through the last version of the ASIYA toolkit for MT evaluation2 (Gim´enez and M`arquez, 2010). Concretely, we derived 86 features for the German-to-English subtask and 97 features for English-to-Spanish. These features cover different approaches and include standard Quality Estima- tion features, as provided by the above mentioned 1Center for Language and Speech Technologies and Ap- plications (TALP), Technical University of Catalonia (UPC). 2http://asiya.lsi.upc.edu ASIYA toolkit and Quest (Specia et al., 2010), but also a variety of features based on pseudo- references (Soricut and Echihabi, 2010), explicit semantic analysis (Gabrilovich and Markovitch, 2007) and specialized language models. See sec- tion 3 for details. In order to model the ranking problem associ- ated to the system selection task, we adapted it to a classification task of pairwise decisions. We trained Random Forest classifiers (and compared them to SVM classifiers), expanding the work of Formiga et al. (2013), from which a full ranking can be derived and the best system per sentence identified. Evaluation at development time, using cross- validation, showed considerably good and stable results for both language pairs, with correlation values around 0.30 (Kendall τ coefficient) classi- fication accuracies around 52% (pairwise classifi- cation) and 41% (best translation identification). Unfortunately, the results on the test set were sig- nificantly lower. Current research is devoted to ex- plain the behavior of the system at testing time. On the one hand, it seems clear that more research re- garding the assignment of ties is needed in …

Content truncated to 500 words