Enhancing the Transformer Decoder with Transition-based Syntax

Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL), pages 384 - 404 December 7-8, 2022 ©2022 Association for Computational Linguistics Enhancing the Transformer Decoder with Transition-based Syntax Leshem Choshen Department of Computer Science Hebrew University of Jerusalem leshem.choshen@mail.huji.ac.il Omri Abend Department of Computer Science Hebrew University of Jerusalem omri.abend@mail.huji.ac.il Abstract Notwithstanding recent advances, syntactic generalization remains a challenge for text de- coders. While some studies showed gains from incorporating source-side symbolic syn- tactic and semantic structure into text genera- tion Transformers, very little work addressed the decoding of such structure. We propose a general approach for tree decoding using a transition-based approach. Examining the challenging test case of incorporating Univer- sal Dependencies syntax into machine trans- lation, we present substantial improvements on test sets that focus on syntactic generaliza- tion, while presenting improved or compara- ble performance on standard MT benchmarks. Further qualitative analysis addresses cases where syntactic generalization in the vanilla Transformer decoder is inadequate and demon- strates the advantages afforded by integrating syntactic information.1 1 Introduction In parallel to the impressive achievements of large neural networks in a variety of NLP fields, more and more work emphasizes the importance of the inductive biases models possess and the types of generalizations they make (Welleck et al., 2021; Csordás et al., 2021; Ontanón et al., 2021). Syntac- tic generalization has been repeatedly identified as a problem in text generation (Linzen and Baroni, 2020; Hu et al., 2020), an issue that we address here. Importantly, language models may fail, some- times unexpectedly, on constructions that can be reliably parsed using standard syntactic parsers. In this work, we propose a method for incorporating syntax into the decoder to assist in mitigating these challenges, focusing on NMT as a test case. The use of (mostly syntactic) structure in ma- chine translation dates back to the early days of the field (Lopez, 2008). While focus has shifted 1Code supplied in github.com/borgr/nematus to string-to-string methods since the introduction of neural methods, considerable work has shown gains from integrating linguistic structure into NMT and text generation technologies. We briefly survey such methods in §7. Incorporating target-side syntax has been less frequently addressed than source-side syntax, pos- sibly due to the additional conceptual and technical complexity it entails, as it requires to jointly gen- erate the translation and its syntactic structure. In addition to linearizing the structure into a string, that allows to easily incorporate source and target structure (Aharoni and Goldberg, 2017b; Nadejde et al., 2017), several works generated the nodes of the syntactic tree using RNNs (G¯u et al., 2018; Wang et al., 2018; Wu et al., 2017). Others have shown gains from multi-task training of a decoder with a syntactic parser (Eriguchi et al., 2016). How- ever, we are not aware of any Transformer-based architecture to support the integration of target-side structure in the form of a tree or a graph. Address- ing this gap, we propose a flexible architecture for integrating graphs into a Transformer decoder. Our approach is based …

Content truncated to 500 words