GuoFeng: A Benchmark for Zero Pronoun Recovery and Translation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11266 - 11278 December 7-11, 2022 ©2022 Association for Computational Linguistics GuoFeng: A Benchmark for Zero Pronoun Recovery and Translation Longyue Wang1∗ Mingzhou Xu2∗ Derek F. Wong2 Hongye Liu1 Linfeng Song1 Lidia S. Chao2 Shuming Shi1 Zhaopeng Tu1 1Tencent AI Lab 2University of Macau {vinnylywang, hongyeliu, lfsong, shumingshi, zptu}@tencent.com nlp2ct.mzxu@gmail.com, {derekfw, lidiasc}@um.edu.com Abstract The phenomenon of zero pronoun (ZP) has at- tracted increasing interest in the machine trans- lation (MT) community due to its importance and difficulty. However, previous studies gen- erally evaluate the quality of translating ZPs with BLEU scores on MT testsets, which are not expressive or sensitive enough for accu- rate assessment. To bridge the data and eval- uation gaps, we propose a benchmark testset for target evaluation on Chinese-English ZP translation. The human-annotated testset cov- ers five challenging genres, which reveal dif- ferent characteristics of ZPs for comprehen- sive evaluation. We systematically revisit eight advanced models on ZP translation and iden- tify current challenges for future exploration. We release data, code, models and annotation guidelines, which we hope can significantly promote research in this field.1 1 Introduction Zero pronoun (ZP) is a discourse phenomenon that appears frequently in pronoun-dropping (pro-drop) languages such as Chinese and Japanese. Specifi- cally, pronouns are often omitted when they can be pragmatically or grammatically inferred from intra- and inter-sentential contexts (Li and Thompson, 1979). Since recovery of such ZPs generally fails, this poses difficulties for several generation tasks, including dialogue modelling (Su et al., 2019), question answering (Tan et al., 2021) as well as machine translation (MT) (Wang et al., 2018a). Although neural machine translation (NMT) (Bahdanau et al., 2015; Vaswani et al., 2017a; Gehring et al., 2017) has achieved great progress in recent years, translating ZPs from a pro-drop to a non-pro-drop language is a significant and chal- lenging task (Wang et al., 2018a). As shown in ∗Mingzhou Xu and Longyue Wang contributed equally to this work. Work was done when Mingzhou Xu and Hongye Liu were interning at Tencent AI Lab. 1https://github.com/longyuewangdcu/ mZPRT. Inp. 黄娟 ，女，副教授 。 (她) 主要 教授 《商务英语》 课程 。 Out. Huang Juan, female, associate professor. Mainly teach the course Business English. Inp. A: 菲比 很 想 买 台 电视 。 B: 乔伊 不 让 (她) 买 (它) ？ Out. A: Phoebe really wants to buy a TV. B: Joey won’t let you buy? Table 1: Examples of ZP translation from two different genres. Words in brackets are ZPs that are invisible in decoding and underlined words are corresponding antecedents. “Inp.” and “Out.” represent the Chinese input and output of Google Translate, respectively. As seen, the translations are either incomplete or incorrect. Table 1, an advanced NMT system still fails to recall the ZP information, which leads to severe problems in translation outputs: 1) incompleteness where the first case grammatically lacks the subject “She”, and the associated verb should be “teaches”; and 2) incorrectness where the second case is se- mantically incorrect (i.e., …

Content truncated to 500 words