ODE Transformer: An Ordinary Differential Equation-Inspired Model for Sequence Generation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8335 - 8351 May 22-27, 2022 c⃝2022 Association for Computational Linguistics ODE Transformer: An Ordinary Differential Equation-Inspired Model for Sequence Generation Bei Li1, Quan Du1,2, Tao Zhou1, Yi Jing1, Shuhan Zhou1, Xin Zeng1 Tong Xiao1,2∗, Jingbo Zhu1,2, Xuebo Liu3 and Min Zhang3 1School of Computer Science and Engineering, Northeastern University, Shenyang, China 2NiuTrans Research, Shenyang, China 3Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China libei_neu@outlook.com, {xiaotong,zhujingbo}@mail.neu.edu.cn, {liuxuebo,zhangmin2021}@hit.edu.cn Abstract Residual networks are an Euler discretization of solutions to Ordinary Differential Equa- tions (ODE). This paper explores a deeper re- lationship between Transformer and numeri- cal ODE methods. We first show that a resid- ual block of layers in Transformer can be de- scribed as a higher-order solution to ODE. Inspired by this, we design a new architec- ture, ODE Transformer, which is analogous to the Runge-Kutta method that is well moti- vated in ODE. As a natural extension to Trans- former, ODE Transformer is easy to imple- ment and efficient to use. Experimental results on the large-scale machine translation, abstrac- tive summarization, and grammar error cor- rection tasks demonstrate the high genericity of ODE Transformer. It can gain large im- provements in model performance over strong baselines (e.g., 30.77 and 44.11 BLEU scores on the WMT’14 English-German and English- French benchmarks) at a slight cost in infer- ence efficiency. 1 Introduction Residual networks have been used with a great success as a standard method of easing information flow in multi-layer neural models (He et al., 2016; Vaswani et al., 2017). Given an input yt, models of this kind define the output of a layer t to be: yt+1 = yt + F(yt, θt) (1) where F(·, ·) is the function of the layer and θt is its parameter. Interestingly, recent work in machine learning (Weinan, 2017; Lu et al., 2018; Haber et al., 2018; Chang et al., 2018; Ruthotto and Haber, 2019) points out that Eq. (1) is an Euler discretiza- tion of the Ordinary Differential Equation (ODE), like this: dy(t) dt = F(y(t), θ(t)) (2) ∗Corresponding author. 6 ODE blocks with 1st-order solutions θ1 θ2 θ3 θ4 θ5 θ6 3 ODE blocks with 2nd-order solutions θ1 θ2 θ3 : F(·, θt) : ODE block Figure 1: Models with different ODE blocks. where y(t) and θ(t) are continuous with respect to t. In this way, we can call Eq. (1) an ODE block. This finding offers a new way of explaining resid- ual networks in the view of numerical algorithms. Then, one can think of a multi-layer network as applying the Euler method (i.e., Eq. (1)) to solve Eq. (2) subject to the initial conditions y(0) = y0 and θ(0) = θ0. The solution of Eq. (2) has a sufficiently low error bound (call it a stable solution) only if θ(t) changes slow along t (Haber and Ruthotto, 2017; Chen et al., 2018). But this assumption does not always hold for …

Content truncated to 500 words