Improving Back-Translation with Uncertainty-based Confidence Estimation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 791–802, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 791 Improving Back-Translation with Uncertainty-based Confidence Estimation Shuo Wang†, Yang Liu†⋆∗, Chao Wang⊤, Huanbo Luan†, and Maosong Sun† †Institute for Artificial Intelligence State Key Laboratory of Intelligent Technology and Systems Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology ⋆Beijing Advanced Innovation Center for Language Resources ⊤6ESTATES PTE LTD, Singapore Abstract While back-translation is simple and effec- tive in exploiting abundant monolingual cor- pora to improve low-resource neural machine translation (NMT), the synthetic bilingual cor- pora generated by NMT models trained on limited authentic bilingual data are inevitably noisy. In this work, we propose to quan- tify the confidence of NMT model predictions based on model uncertainty. With word- and sentence-level confidence measures based on uncertainty, it is possible for back-translation to better cope with noise in synthetic bilingual corpora. Experiments on Chinese-English and English-German translation tasks show that uncertainty-based confidence estimation sig- nificantly improves the performance of back- translation. 1 1 Introduction The past several years have witnessed the rapid development of end-to-end neural machine trans- lation (NMT) (Sutskever et al., 2014; Bahdanau et al., 2015; Vaswani et al., 2017), which lever- ages neural networks to map between natural lan- guages. Capable of learning representations from data, NMT has significantly outperformed con- ventional statistical machine translation (SMT) (Koehn et al., 2003) and been widely deployed in large-scale MT systems in the industry (Wu et al., 2016; Hassan et al., 2018). Despite the remarkable success, NMT suffers from the data scarcity problem. For most lan- guage pairs, large-scale, high-quality, and wide- coverage bilingual corpora do not exist. Even for the top handful of resource-rich languages, the major sources of available parallel corpora are of- ∗ Yang Liu is the corresponding author: liuyang2011@ tsinghua.edu.cn. 1The source code is available at https://github. com/THUNLP-MT/UCE4BT Bush hold a talks and Sharon … … bushi yu shalong jvxing le huitan y <latexit sha1_base64="oyl3CDdc4pKXMbwMXulNjWZ+gQY=">AB8XicbVBNS8NAFHypX7V+VT16WSyCp5KoM eiF48VbC2oWy2m3bpZhN2X4QS+i+8eFDEq/Gm/GTZuDtg4sDPvsfMmSKQw6LrfTmldW19o7xZ2dre2d2r7h+0TZxqxlslrHuBNRwKRvoUDJO4nmNAokfwjGN7n/8MS1EbG6x0nC/YgOlQgFo2ilx15EcRSE2WTar9bcujsDWSZe QWpQoNmvfvUGMUsjrpBJakzXcxP0M6pRMmnlV5qeELZmA51JFI278bJZ4Sk6sMiBhrO1TSGbq742MRsZMosBO5gnNopeL/3ndFMrPxMqSZErNv8oTCXBmOTnk4HQnKGcWEKZFjYrYSOqKUNbUsW4C2evEzaZ3XvO7eXdQa10UdZT iCYzgFDy6hAbfQhBYwUPAMr/DmGOfFeXc+5qMlp9g5hD9wPn8A/2iRHw=</latexit> ˆx <latexit sha1_base64="7tb573AnuxEhCsAMHcBVcKq+Ps=">AB+XicbVDLSsNAFJ3UV62vqEs3g0VwVRIVdF l047KCfUATymQ6aYdOJmHmplhC/sSNC0Xc+ifu/BsnbRbaemDgcM693DMnSATX4DjfVmVtfWNzq7pd29nd2z+wD486Ok4VZW0ai1j1AqKZ4JK1gYNgvUQxEgWCdYPJXeF3p0xpHstHmCXMj8hI8pBTAkYa2LY3JpB5EYFxEGZPeT6w607DmQ OvErckdVSiNbC/vGFM04hJoIJo3XedBPyMKOBUsLzmpZolhE7IiPUNlSRi2s/myXN8ZpQhDmNlngQ8V39vZCTSehYFZrKIqJe9QvzP6cQ3vgZl0kKTNLFoTAVGJc1ICHXDEKYmYIoYqbrJiOiSIUTFk1U4K7/OV0rlouJcN5+Gq3rwt6 iE3SKzpGLrlET3aMWaiOKpugZvaI3K7NerHfrYzFascqdY/QH1ucPVq+UHA=</latexit> ˆ✓y!x <latexit sha1_base64="8nrQv4PTgwZpETGQRGNPpnJ4wWA=">ACHicbVDLSsNAFJ34tr6qLt0MFsFVSVTQZdGNSwXbCk0pN9NJM3QmCTM3agn5AH/C X3Cre3fiVnDrlzitWaj1wMDhnHO5d06QSmHQdT+cmdm5+YXFpeXKyura+kZ1c6tlkwz3mSJTPR1AIZLEfMmCpT8OtUcVCB5Oxiejf32DdGJPEVjlLeVTCIRSgYoJV61ZofAeZ+oHIfI45QFL18RH0tBhGC1sktvStsyq27E9Bp4pWkRkpc9Kqfj9hmeIxMgnGdDw3xW4OGgWTvKj4meEpsCEMeMfSGBQ3XzymYLuWaV Pw0TbFyOdqD8nclDGjFRgkwowMn+9sfif18kwPOnmIk4z5DH7XhRmkmJCx83QvtCcoRxZAkwLeytlEWhgaPv7tSVQRcW4v2tYJq0DureYd29PKo1Tst6lsgO2SX7xCPHpEHOyQVpEkbuySN5Is/Og/PivDpv39EZp5zZJr/gvH8BUcyjBA=</latexit> confidence Figure 1: Confidence estimation for back-translation. Back-translation generates a source (e.g., English) sen- tence for a ground-truth target (e.g., Chinese) sentence. Such synthetic sentence pairs are used to train NMT models. As the model prediction (i.e., ˆx) is often noisy, our work aims to quantify the prediction confidence us- ing model uncertainty to alleviate error propagation. ten restricted to government documents or news articles. Therefore, improving NMT under small-data training conditions has attracted extensive atten- tion in recent years (Sennrich et al., 2016a; Cheng et al., 2016; Zoph et al., 2016; Chen et al., 2017; Fadaee et al., 2017; Ren et al., 2018; Lample et al., 2018). Among them, back-translation (Sennrich et al., 2016a) is an important direction. Its ba- sic idea is to use an NMT model trained on lim- ited …

Content truncated to 500 words