Strategy-Based Technology for Estimating MT Quality

Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 348–352, Lisboa, Portugal, 17-18 September 2015. c⃝2015 Association for Computational Linguistics. Strategy-Based Technology for Estimating MT Quality Liugang Shang Knowledge Engineering and Human-Computer Interaction center, Shenyang Aerospace University, Shenyang, China 1437260083@qq.com Dongfeng Cai Knowledge Engineering and Human-Computer Interaction center , Shenyang Aerospace University, Shenyang, China caidf@vip.163.com Duo Ji Knowledge Engineering and Human-Computer Interaction center, Shenyang Aerospace University, Shenyang, China jiduo_1@163.com Abstract This paper introduces our SAU-KERC system that achieved F1 score of 0.39 in the world-level quality estimation task in WMT2015. The goal is to assign each trans- lated word a “OK” or “BAD” label indicating translation quality. We adopt the sequence labeling model, conditional random fields (CRF), to predict the labels. Since “BAD” la- bels are rare in the training and development sets, recognition rate of "BAD" is low. To solve this problem, we propose two strategies. One is to replace “OK” label with sub-labels to balance label distribution. The other is to reconstruct the training set to include more "BAD" words. 1 Introduction QE task is proposed to estimate the quality of machine translation without relying on reference translations. It contains three levels -- word, sen- tence, and document and our work focuses on the word-level task. The word-level task was pro- posed in 2013 and was divided into binary classi- fication and multi-class classification. This year only binary classification was considered in WMT2015. OK/BAD: If a word need editing, then it is BAD. It is OK, otherwise. As a confidence estimation problem, methods aim to confidence estimation before 2013. A lot of researchers started to investigate confidence measures for machine translation for nearly a decade (Gandrabur and Foster, 2003; Quirk, 2004; Ueffing et al., 2003). Many different con- fidence measures are investigated in(Blatz et al 2003). They are based on source and target lan- guage models features, n-best list, word-lattices, translation tables, and so on. The authors also present efficient ways of classifying words as ”correct” or “incorrect” by using native Bayes, single- or multi-layer perceptron. (Blatz et al 2003) combines several features and use neural network and naïve Bayes learning algorithms to predict whether a word is ok or bad. (Xiong et al., 2010) combines syntax feature, vocabulary fea- ture and word posterior probability feature, which are extracted based on LG parsing, and use the binary classifier based on Maximum En- tropy Model to predict the label of each word in machine translation(ok or bad). Some good ideas are proposed in word-level QE task of WMT. (Luong et al., 2013) use both internal and external features into a conditional random fields(CRF) model to predict the label for each word in the MT hypothesis. (Wisniew- skiet al., 2014) rely on a random forest classifier and 16 features to predict the label of a word. (Souza et al., 2014) train two classifier models by using bidirectional long short-term memory recurrent neural networks and CRF to complete word level QE Task. In WMT2015, the high ratio of OK labels …

Content truncated to 500 words