Unsupervised Neural Machine Translation with Universal Grammar

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3249–3264 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3249 Unsupervised Neural Machine Translation with Universal Grammar Zuchao Li1,2,3,†, Masao Utiyama4,∗, Eiichiro Sumita4, and Hai Zhao1,2,3∗ 1Department of Computer Science and Engineering, Shanghai Jiao Tong University 2Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University, Shanghai, China 3MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University 4National Institute of Information and Communications Technology (NICT), Kyoto, Japan charlee@sjtu.edu.cn, {mutiyama, eiichiro.sumita}@nict.go.jp, zhaohai@cs.sjtu.edu.cn Abstract Machine translation usually relies on parallel corpora to provide parallel signals for training. The advent of unsupervised machine trans- lation has brought machine translation away from this reliance, though performance still lags behind traditional supervised machine translation. In unsupervised machine trans- lation, the model seeks symmetric language similarities as a source of weak parallel sig- nal to achieve translation. Chomsky’s Uni- versal Grammar theory postulates that gram- mar is an innate form of knowledge to hu- mans and is governed by universal principles and constraints. Therefore, in this paper, we seek to leverage such shared grammar clues to provide more explicit language parallel sig- nals to enhance the training of unsupervised machine translation models. Through experi- ments on multiple typical language pairs, we demonstrate the effectiveness of our proposed approaches. 1 Introduction Recently, Neural Machine Translation (NMT) (Bahdanau et al., 2014; Sutskever et al., 2014) has been greatly developed and become the dominant paradigm in machine translation. On the one hand, the development of deep neural networks such as Transformer (Vaswani et al., 2017; Li et al., 2021a) has played a significant role in NMT’s improve- ments. On the other hand, large-scale parallel cor- pora like the UN corpus (Ziemski et al., 2016) have also played an important role. Despite the recent success of NMT in stan- dard benchmarks, the need for large-scale parallel corpora has limited the effectiveness of NMT in many language pairs, especially in low-resource language pairs (Koehn and Knowles, 2017). Un- supervised Neural Machine Translation (UNMT) ∗Corresponding author. † This paper was finished when Zuchao Li was a fixed term technical researcher at NICT. This work was supported by Key Projects of National Natural Science Foundation of China (U1836222 and 61733011). ADV Demnächst 1 VMFIN sollen 2 ADV allerdings 3 S NP (4,5) CARD 450 4 NN Lehrlinge 5 VVPP eingestellt 6 VAIN werden 7 VP (4,5) DT The 1 NNP CWA 2 VBZ represents 3 S NP (4,6) CD 40,000 4 NNP Nynex 5 NNS workers 6 VP (3,6) (a) (b) Figure 1: Examples of constituent trees from En- glish Penn Treebank (PTB) and German dataset of SPMRL14 shared task. The dotted box indicates the constituents that can be masked for prediction. (Artetxe et al., 2018b) was proposed to alleviate this issue by completely removing the need for parallel data and training an NMT system in a com- pletely unsupervised manner, relying on nothing but monolingual corpora. Unsupervised machine translation does …

Content truncated to 500 words