The Box is in the Pen: Evaluating Commonsense Reasoning in Neural Machine Translation

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3662–3672 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3662 The Box is in the Pen: Evaluating Commonsense Reasoning in Neural Machine Translation Jie He†∗, Tao Wang‡∗, Deyi Xiong†, and Qun Liu§ † College of Intelligence and Computing, Tianjin University, Tianjin, China ‡ School of Computer Science and Technology, Soochow University, Suzhou, China § Huawei Noah’s Ark Lab, Hong Kong, China jieh@tju.edu.cn, rgwt1234@gmail.com dyxiong@tju.edu.cn, qun.liu@huawei.com Abstract Does neural machine translation yield transla- tions that are congenial with common sense? In this paper, we present a test suite to evalu- ate the commonsense reasoning capability of neural machine translation. The test suite consists of three test sets, covering lexical and contextless/contextual syntactic ambiguity that requires commonsense knowledge to re- solve. We manually create 1,200 triples, each of which contain a source sentence and two contrastive translations, involving 7 different common sense types. Language models pre- trained on large-scale corpora, such as BERT, GPT-2, achieve a commonsense reasoning ac- curacy of lower than 72% on target transla- tions of this test suite. We conduct extensive experiments on the test suite to evaluate com- monsense reasoning in neural machine trans- lation and investigate factors that have impact on this capability. Our experiments and anal- yses demonstrate that neural machine transla- tion performs poorly on commonsense reason- ing of the three ambiguity types in terms of both reasoning accuracy ( ⩽ 60.1%) and rea- soning consistency (⩽ 31%). We will release our test suite as a machine translation com- monsense reasoning testbed to promote future work in this direction. 1 Introduction Sixty years ago, the pioneering machine transla- tion researcher and linguist Bar-Hillel published his well-known argument on the non-feasibility of general-purpose fully automatic high-quality ma- chine translation (FAHQT) due to the inevitable requirement of world knowledge to help machine translation to infer correct translations for am- biguous words or linguistic structures (Bar-Hillel, 1960a). The example that Bar-Hillel uses as an ∗Equal Contributions. evidence for the need of commonsense knowl- edge in machine translation is “The box is in the pen”, where machine translation is expected to per- form reasoning on the relative sizes of “box” and “pen”. Bar-Hillel also doubts that a machine, even equipped with extra-linguistic knowledge, would be able to reason with such knowledge sponta- neously as human translators do (Bar-Hillel, 1960a; Macklovitch, 1995). Modern natural language processing (NLP) has made tremendous progress, not only in building abundant resources to develop linguistic insights, but also in plenty of methodological practices. On the one hand, machine translation has been sub- stantially advanced with large-scale parallel data and statistical models. Recent results even suggest that the quality of machine-generated translations is approaching professional human translators (Wu et al., 2016; Hassan et al., 2018). On the other hand, a wide variety of efforts have been conducted to either examine the commonsense reasoning capa- bility of neural models in natural language under- standing, establish commonsense reasoning chal- lenges or enhance neural models in …

Content truncated to 500 words