Leveraging Relaxed Equilibrium by Lazy Transition for Sequence Modeling

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2904 - 2924 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Leveraging Relaxed Equilibrium by Lazy Transition for Sequence Modeling Xi Ai College of Computer Science Chongqing University barid.x.ai@gmail.com Bin Fang College of Computer Science Chongqing University fb@cqu.edu.cn Abstract In sequence modeling, certain tokens are usu- ally less ambiguous than others, and represen- tations of these tokens require fewer refine- ments for disambiguation. However, given the nature of attention-based models like Trans- former and UT (universal transformer), all to- kens are equally processed towards depth. In- spired by the equilibrium phenomenon, we present a lazy transition, a mechanism to ad- just the significance of iterative refinements for each token representation. Our lazy transition is deployed on top of UT to build LT (lazy transformer), where all tokens are processed unequally towards depth. Eventually, LT is en- couraged to oscillate around a relaxed equilib- rium. Our experiments show that LT outper- forms baseline models on several tasks of ma- chine translation, pre-training, Learning to Ex- ecute, and LAMBADA. 1 Introduction Attention-based models like Transformer (Vaswani et al., 2017) underly two core concepts: layer and refinement. With layer-stacked structure, the model entirely relies on attention mechanisms (Parikh et al., 2016; Lin et al., 2017) to refine token representations (e.g.,vectors) layer-by-layer from context-informed representations by using residual connections (He et al., 2016; Ebski et al., 2018). However, layer is not necessary. UT (universal transformer) (Dehghani et al., 2019) and its vari- ants (Bai et al., 2019; Lan et al., 2020) iteratively run single-layer but wide Transformer for sequence modeling that token representations are refined at each step by attending to the sequence represen- tation of the previous step, showing higher per- formance than layer-stacked Transformer does on NMT (Dehghani et al., 2019), pre-training (Lan et al., 2020), language modeling (Bai et al., 2019), and other tasks of varying complexity (Dehghani et al., 2019) with the same number of parameters. The computational bound of recurrence is not the number of tokens in the sequence (like RNN) or the number of stacked layers in the model but is the maximum number of refinements made to to- ken representations, e.g., a pre-defined maximum number of iteration steps. We follow this line but further ponder refine- ment. Concretely, the model refines token represen- tations iteratively that tokens are equally processed towards depth for disambiguation. Essentially, dif- ferent token representations are refined at a step equally and concurrently. Thus, regardless of the ambiguous state, a step shows the same importance for all the tokens. However, certain tokens are less ambiguous than others in sequence modeling, es- pecially in NMT. It raises two questions: 1) do deep representations have to be learned for less- ambiguous tokens; 2) do tokens must consider the contributions or importance of a step equally? Meanwhile, our work derives its motivation from the combination of UT and ACT (adaptive compu- tation time) (Graves, 2016) that can dynamically estimate …

Content truncated to 500 words