Revisiting Commonsense Reasoning in Machine Translation: Training, Evaluation and Challenge

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 15536–15550 July 9-14, 2023 ©2023 Association for Computational Linguistics Revisiting Commonsense Reasoning in Machine Translation: Training, Evaluation and Challenge Xuebo Liu1∗ Yutong Wang1 Derek F. Wong2 Runzhe Zhan2 Liangxuan Yu2 Min Zhang1 1Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China {liuxuebo,zhangmin2021}@hit.edu.cn, wangyutong200012@gmail.com 2NLP2CT Lab, Department of Computer and Information Science, University of Macau derekfw@um.edu.mo, nlp2ct.{runzhe,liangxuan}@gmail.com Abstract The ability of commonsense reasoning (CR) decides whether a neural machine translation (NMT) model can move beyond pattern recog- nition. Despite the rapid advancement of NMT and the use of pretraining to enhance NMT models, research on CR in NMT is still in its in- fancy, leaving much to be explored in terms of effectively training NMT models with high CR abilities and devising accurate automatic evalu- ation metrics. This paper presents a comprehen- sive study aimed at expanding the understand- ing of CR in NMT. For the training, we confirm the effectiveness of incorporating pretrained knowledge into NMT models and subsequently utilizing these models as robust testbeds for investigating CR in NMT. For the evaluation, we propose a novel entity-aware evaluation method that takes into account both the NMT candidate and important entities in the candi- date, which is more aligned with human judge- ment. Based on the strong testbed and evalua- tion methods, we identify challenges in train- ing NMT models with high CR abilities and suggest directions for further unlabeled data utilization and model design. We hope that our methods and findings will contribute to advanc- ing the research of CR in NMT. Source data, code and scripts are freely available at https: //github.com/YutongWang1216/CR-NMT. 1 Introduction Commonsense reasoning (CR; Davis and Marcus, 2015) is the ability to understand and navigate the world using basic knowledge and understanding that is shared by most people. In the context of neural machine translation (NMT; Bahdanau et al., 2015; Vaswani et al., 2017; Liu et al., 2019, 2020a), CR is important because it allows the model to move beyond simply recognizing patterns in the data and instead make more informed, nuanced translations. Recent studies have witnessed the ∗Corresponding author Contextless Syntactic Ambiguity SRC 捕获的/Hunting 是/is 猎人/hunter 。 REF The hunter is hunting . NMT The hunter was captured . PT-NMT It was the hunter who caught it . Contextual Syntactic Ambiguity SRC 手术/surgery 开刀的/operated 是/is 他父亲/his father，因为/because 他父亲/his father 得了重 病/get serious illness。 REF It was his father who underwent surgery, because his father was seriously ill. NMT The operation was performed by his father, who was seriously ill. PT-NMT The operation was performed on his father, who was seriously ill. Lexical Ambiguity SRC 学校/school 规定/mandates 学生/students 上学/go to school 要/must 背/carry 书包/school bag。 REF The school requires students to carry school bags. NMT Schools require students to recite school bags. PT-NMT The school requires students to carry their school- bags at school. Table 1: Translations of vanilla NMT and pretraining- based NMT (PT-NMT). Highlights …

Content truncated to 500 words