Neural Reranking for Dependency Parsing: An Evaluation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 386–396 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 386 Learning Source Phrase Representations for Neural Machine Translation Hongfei Xu1,2 Josef van Genabith1,2∗ Deyi Xiong3 Qiuhui Liu4 Jingyi Zhang2 1Saarland University / Saarland, Germany 2German Research Center for Artificial Intelligence / Saarland, Germany 3Tianjin University / Tianjin, China 4China Mobile Online Services / Henan, China hfxunlp@foxmail.com, Josef.Van Genabith@dfki.de, dyxiong@tju.edu.cn, liuqhano@foxmail.com, Jingyi.Zhang@dfki.de Abstract The Transformer translation model (Vaswani et al., 2017) based on a multi-head attention mechanism can be computed effectively in parallel and has significantly pushed forward the performance of Neural Machine Trans- lation (NMT). Though intuitively the atten- tional network can connect distant words via shorter network paths than RNNs, empirical analysis demonstrates that it still has difficulty in fully capturing long-distance dependencies (Tang et al., 2018). Considering that model- ing phrases instead of words has significantly improved the Statistical Machine Translation (SMT) approach through the use of larger translation blocks (“phrases”) and its reorder- ing ability, modeling NMT at phrase level is an intuitive proposal to help the model capture long-distance relationships. In this paper, we first propose an attentive phrase representation generation mechanism which is able to gener- ate phrase representations from corresponding token representations. In addition, we incorpo- rate the generated phrase representations into the Transformer translation model to enhance its ability to capture long-distance relation- ships. In our experiments, we obtain signifi- cant improvements on the WMT 14 English- German and English-French tasks on top of the strong Transformer baseline, which shows the effectiveness of our approach. Our ap- proach helps Transformer Base models per- form at the level of Transformer Big mod- els, and even significantly better for long sen- tences, but with substantially fewer parameters and training steps. The fact that phrase repre- sentations help even in the big setting further supports our conjecture that they make a valu- able contribution to long-distance relations. 1 Introduction NMT is a new approach to machine translation that has achieved great success in the last a few ∗ Corresponding author. years (Sutskever et al., 2014; Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017). Com- pared to plain SMT (Brown et al., 1993; Koehn et al., 2003; Chiang, 2005), a neural language model decoder (Sutskever et al., 2014) is better at long-distance re-ordering, and attention mech- anisms (Bahdanau et al., 2015; Vaswani et al., 2017) have been proven effective in modeling long- distance dependencies, while these two issues were both challenging for SMT. The Transformer (Vaswani et al., 2017), which has outperformed previous RNN/CNN based trans- lation models (Bahdanau et al., 2015; Gehring et al., 2017), is based on multi-layer multi-head attention networks and can be trained in parallel very effi- ciently. Though attentional networks can connect distant words via shorter network paths than RNNs, empirical results show that its ability in capturing long-range dependencies does not significantly out- perform RNNs, and it is still a problem for …

Content truncated to 500 words