An Empirical Study of Generation Order for Machine Translation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5764–5773, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5764 An Empirical Study of Generation Order for Machine Translation William Chan∗ Google Research, Brain Team williamchan@google.com Mitchell Stern∗ University of California, Berkeley mitchell@berkeley.edu Jamie Kiros Google Research, Brain Team kiros@google.com Jakob Uszkoreit Google Research, Brain Team uszkoreit@google.com Abstract In this work, we present an empirical study of generation order for machine translation. Building on recent advances in insertion-based modeling, we first introduce a soft order- reward framework that enables us to train models to follow arbitrary oracle generation policies. We then make use of this frame- work to explore a large variety of generation orders, including uninformed orders, location- based orders, frequency-based orders, content- based orders, and model-based orders. Curi- ously, we find that for the WMT’14 English → German and WMT’18 English → Chinese translation tasks, order does not have a sub- stantial impact on output quality. Moreover, for English → German, we even discover that unintuitive orderings such as alphabetical and shortest-first can match the performance of a standard Transformer, suggesting that tradi- tional left-to-right generation may not be nec- essary to achieve high performance. 1 Introduction Neural sequence models (Sutskever et al., 2014; Cho et al., 2014) have been successfully ap- plied to a broad range of tasks in recent years. While these models typically generate their out- puts using a fixed left-to-right order, there has also been some investigation into non-left-to-right and order-independent generation in pursuit of qual- ity or speed. For example, Vinyals et al. (2015) explored the problem of predicting sets using se- quence models. While this is a domain where generation order should intuitively be unimpor- tant, they nevertheless found it to make a substan- tial difference in practice. Ford et al. (2018) ex- plored treating language modeling as a two-pass process, where words from certain classes are gen- erated first, and the remaining words are filled ∗Equal contribution. in during the second pass. They found that gen- erating function words first followed by content words second yielded the best results. Separately, Gu et al. (2018) and Lee et al. (2018) developed non-autoregressive approaches to machine trans- lation where the entire output can be generated in parallel in constant time. These models do away with order selection altogether but typically lag behind their autoregressive counterparts in trans- lation quality. More recently, a number of novel insertion- based architectures have been developed for se- quence generation (Gu et al., 2019; Stern et al., 2019; Welleck et al., 2019). These frameworks li- cense a diverse set of generation orders, including uniform (Welleck et al., 2019), random (Gu et al., 2019), or balanced binary trees (Stern et al., 2019). Some of them also match the quality of state-of- the-art left-to-right models (Stern et al., 2019). In this paper, we utilize one such framework to ex- plore an extensive collection of generation orders, evaluating them on the WMT’14 English-German and WMT’18 English-Chinese translation tasks. …

Content truncated to 500 words