Towards Incremental Transformers: An Empirical Analysis of Transformer Models for Incremental NLU

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1178–1189 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 1178 Towards Incremental Transformers: An Empirical Analysis of Transformer Models for Incremental NLU Patrick Kahardipraja Brielen Madureira David Schlangen Computational Linguistics, Department of Linguistics University of Potsdam, Germany {kahardipraja,madureiralasota,david.schlangen}@uni-potsdam.de Abstract Incremental processing allows interactive sys- tems to respond based on partial inputs, which is a desirable property e.g. in dialogue agents. The currently popular Transformer architec- ture inherently processes sequences as a whole, abstracting away the notion of time. Recent work attempts to apply Transformers incre- mentally via restart-incrementality by repeat- edly feeding, to an unchanged model, increas- ingly longer input prefixes to produce partial outputs. However, this approach is computa- tionally costly and does not scale efficiently for long sequences. In parallel, we witness efforts to make Transformers more efficient, e.g. the Linear Transformer (LT) with a re- currence mechanism. In this work, we exam- ine the feasibility of LT for incremental NLU in English. Our results show that the recur- rent LT model has better incremental perfor- mance and faster inference speed compared to the standard Transformer and LT with restart- incrementality, at the cost of part of the non- incremental (full sequence) quality. We show that the performance drop can be mitigated by training the model to wait for right context be- fore committing to an output and that training with input prefixes is beneficial for delivering correct partial outputs. 1 Introduction One fundamental property of human language pro- cessing is incrementality (Keller, 2010). Humans process language on a word-by-word basis by main- taining a partial representation of the sentence meaning at a fast pace and with great accuracy (Marslen-Wilson, 1973). The garden path effect, for example, shows that language comprehension is approximated incrementally before committing to a careful syntactic analysis (Frazier and Rayner, 1982; Altmann and Steedman, 1988; Trueswell et al., 1994, inter alia). The notion of order along the time axis during computation is a key aspect of incremental pro- cessing and thus a desirable property both of cog- nitively plausible language encoders as well as in applications such as interactive systems (Skantze and Schlangen, 2009). RNNs, for example, are inherently able to process words sequentially while updating a recurrent state representation. However, the Transformer architecture (Vaswani et al., 2017), which has brought significant improvements on several NLP tasks, processes the input sequence as a whole, thus prioritising parallelisation to the detriment of the notion of linear order. One way to employ non-incremental models in incremental settings is resorting to an incremental interface, like in Beuck et al. (2011), where a com- plete recomputation of the available partial input happens at each time step to deliver partial output. Madureira and Schlangen (2020) examined the out- put stability of non-incremental encoders in this restart-incremental fashion. While qualitatively feasible, this procedure is computationally costly, especially for long sequences, since it requires as many forward passes as the number of input tokens. In parallel, there is …

Content truncated to 500 words