WeTS: A Benchmark for Translation Suggestion

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5278 - 5290 December 7-11, 2022 ©2022 Association for Computational Linguistics WeTS: A Benchmark for Translation Suggestion Zhen Yang, Fandong Meng, Yingxue Zhang, Ernan Li, and Jie Zhou Pattern Recognition Center, WeChat AI, Tencent Inc, Beijing, China {zieenyang, fandongmeng, yxuezhang, cardli, withtomzhou}@tencent.com Abstract Translation suggestion (TS), which provides alternatives for specific words or phrases given the entire documents generated by machine translation (MT), has been proven to play a significant role in post-editing (PE). There are two main pitfalls for existing researches in this line. First, most conventional works only focus on the overall performance of PE but ignore the exact performance of TS, which makes the progress of PE sluggish and less explainable; Second, as no publicly available golden dataset exists to support in-depth research for TS, al- most all of the previous works conduct exper- iments on their in-house datasets or the noisy datasets built automatically, which makes their experiments hard to be reproduced and com- pared. To break these limitations mentioned above and spur the research in TS, we create a benchmark dataset, called WeTS, which is a golden corpus annotated by expert translators on four translation directions. Apart from the golden corpus, we also propose several meth- ods to generate synthetic corpora which can be used to improve the performance substan- tially through pre-training. As for the model, we propose the segment-aware self-attention based Transformer for TS. Experimental re- sults show that our approach achieves the best results on all four directions, including English- to-German, German-to-English, Chinese-to- English, and English-to-Chinese. Codes and corpus can be found at https://github.com/ ZhenYangIACAS/WeTS.git. 1 Introduction Computer-aided translation (CAT) (Barrachina et al., 2009; Green et al., 2014; Knowles and Koehn, 2016; Santy et al., 2019) has attained more and more attention for its promising ability in combin- ing the high efficiency of machine translation (MT) (Cho et al., 2014; Bahdanau et al., 2015; Vaswani et al., 2017) and high accuracy of human transla- tion (HT). A typical way for CAT tools to combine MT and HT is PE (Green et al., 2013; Zouhar et al., 2021), where the human translators are asked to provide alternatives for the incorrect word spans in the results generated by MT. To further reduce the post-editing time, researchers propose to apply TS into PE, where TS provides the sub-segment sug- gestions for the annotated incorrect word spans in the results of MT, and their extensive experiments show that TS can substantially reduce translators’ cognitive loads and the post-editing time (Wang et al., 2020; Lee et al., 2021). As there is no explicit and formal definition for TS, we observe that some previous works simi- lar or related to TS have been proposed (Alabau et al., 2014; Santy et al., 2019; Wang et al., 2020; Lee et al., 2021). However, there are two main pitfalls for these works in this line. First, most conventional works only focus on the overall per- formance of PE …

Content truncated to 500 words