Improving Machine Translation Quality Estimation with Neural Network Features

Proceedings of the Conference on Machine Translation (WMT), Volume 2: Shared Task Papers, pages 551–555 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics Improving Machine Translation Quality Estimation with Neural Network Features Zhiming Chen, Yiming Tan, Chenlin Zhang, Qingyu Xiang, Lilin Zhang, Maoxi Li, Mingwen Wang School of Computer Information Engineering, Jiangxi Normal University {qqchenzhiming, tt_yymm, zhangchenlin, qingyuxian, lilinzhang, mosesli, mwwang} @jxnu.edu.cn Abstract Machine translation quality estimation is a challenging task in the WMT evaluation campaign. Feature extrac- tion plays an important role in auto- matic quality estimation, and in this paper, we propose neural network fea- tures, including embedding features and cross-entropy features of source sentences and machine translations, to improve machine translation quality estimation. The sentence embedding features are extracted through global average pooling from word embedding and are trained by the word2vec toolkits, while the sentence cross- entropy features are calculated by the recurrent neural network language model. The experimental results on the development set of WMT17 machine translation quality estimation tasks show that the neural network features gain significant improvements over the baseline. Furthermore, when combin- ing the neural network features and the baseline features, the system perfor- mance obtains further improvement. 1 Introduction Quality estimation (QE) of machine translation estimates the quality of machine translation sys- tem outputs without human references using ma- chine learning methods. It is often divided into two steps: first, it extracts various features from source sentences, translation outputs, and external language resources to describe the translation complexity, fluency and adequacy; and second, it predicts the quality of the translation outputs with the pre-trained machine learning model. Feature extraction is crucial to the performance of QE, and traditional methods, such as QuEst (Specia et al., 2013), extract linguistically motivated fea- tures to improve the correlation between the au- tomatic QE and human assessment. However, ex- tracting linguistically motivated features requires part-of-speech analysis, syntactic analysis, or se- mantic analysis, and these linguistic analyses re- late to the target language types; this considera- tion limits their application in other languages. To address this problem, Shah et al. (2015a, 2016) investigated continuous space language models for sentence-level QE, and Scarton et al. (2016) proposed word embedding features for document- level QE. Inspired by their work, we propose sentence embedding features and cross-entropy features to improve the correlation between automatic QE and human assessment and to investigate how dif- ferent sentence embedding dimensions of source sentences and translation outputs, as well as the size of the training corpus, affect the system per- formance of QE. 2 Related work With the great success of deep learning that has been achieved in digital image processing and au- tomatic speech recognition, deep learning has also made tremendous breakthroughs in natural lan- guage processing, e.g., the proposition of neural network language models (Bengio et al. 2003) and neural machine translation encoder-decoder frameworks (Bahdanau et al. 2014). Therefore, many researchers have proposed deep learning approaches for the QE task. In the word-level QE task, Kreutzer et al. (2015) …

Content truncated to 500 words