Hierarchical Phrase-Based Sequence-to-Sequence Learning

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8211 - 8229 December 7-11, 2022 ©2022 Association for Computational Linguistics Hierarchical Phrase-based Sequence-to-Sequence Learning Bailin Wang MIT bailinw@mit.edu Ivan Titov University of Edinburgh University of Amsterdam ititov@inf.ed.ac.uk Jacob Andreas MIT jda@mit.edu Yoon Kim MIT yoonkim@mit.edu Abstract We describe a neural transducer that maintains the flexibility of standard sequence-to-sequence (seq2seq) models while incorporating hierarchi- cal phrases as a source of inductive bias dur- ing training and as explicit constraints during inference. Our approach trains two models: a discriminative parser based on a bracketing transduction grammar whose derivation tree hi- erarchically aligns source and target phrases, and a neural seq2seq model that learns to trans- late the aligned phrases one-by-one. We use the same seq2seq model to translate at all phrase scales, which results in two inference modes: one mode in which the parser is discarded and only the seq2seq component is used at the sequence-level, and another in which the parser is combined with the seq2seq model. Decoding in the latter mode is done with the cube-pruned CKY algorithm, which is more involved but can make use of new translation rules during inference. We formalize our model as a source- conditioned synchronous grammar and develop an efficient variational inference algorithm for training. When applied on top of both randomly initialized and pretrained seq2seq models, we find that both inference modes performs well compared to baselines on small scale machine translation benchmarks. 1 Introduction Despite the improvements in performance and data-efficiency enabled by recent advances in pre- training, standard neural sequence-to-sequence (seq2seq) models can still fail to model the hier- archical structure of sentences and be brittle with respect to to novel syntactic structures (Lake and Baroni, 2018; Kim and Linzen, 2020; Weißenhorn et al., 2022). It has moreover not been clear how to incorporate explicit constraints such as translation rules (e.g., for translating idioms) into these black- box models without changing the underlying archi- tecture. Classic grammar- and automaton-based Code available at: https://github.com/berlino/btg-seq2seq. Figure 1: An overview of our approach. During training a variational synchronous parser hierarchically segments and aligns source-target phrases (top), which provides training pairs for a seq2seq model (bottom). The seq2seq loss provides learning signal to the synchronous parser which subsequently provides signal to (and is regularized by) a monolingual source parser. After training, there are two inference models: one mode in which both parsers are discarded, which results in a regular seq2seq model that has been regularized by latent hierarchical phrase pairs; another mode in which the seq2seq model can be combined with the source parser to derive a neural synchronous grammar. Inference in this mode is done via translation-as-parsing with cube-pruned CKY. approaches are well-suited for capturing hierarchi- cal structure and can readily incorporate new rules, but have trouble with examples that do not conform exactly to the symbolic specifications. This paper describes a neural transducer that maintains the flexibility of neural seq2seq models but still mod- els hierarchical phrase alignments between …

Content truncated to 500 words