Efficient Encoders for Streaming Sequence Tagging

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 418–429 May 2-6, 2023 ©2023 Association for Computational Linguistics Efficient Encoders for Streaming Sequence Tagging Ayush Kaushal♣∗ Aditya Gupta♠ Shyam Upadhyay♠ Manaal Faruqui♠ ♠Google Assistant ♣The University of Texas at Austin ayushk4@utexas.edu, {gaditya, shyamupa, mfaruqui}@google.com Abstract A naive application of state-of-the-art bidirec- tional encoders for streaming sequence tag- ging would require re-encoding all tokens from scratch whenever a new token appears in an incremental streaming input (like transcribed speech). The lack of re-usability of previ- ous computation leads to a higher number of Floating Point Operations (or FLOPs) and higher number of unnecessary label flips. In- creased FLOPs consequently lead to higher wall-clock time and increased label flipping leads to poorer streaming performance. In this work, we present Hybrid Encoder with Adaptive Restart (HEAR) that addresses these issues while maintaining the performance of bidirectional encoders over offline (or com- plete) inputs and improving performance on streaming (or incomplete) inputs. HEAR uses a HYBRID unidirectional-bidirectional encoder architecture to perform sequence tagging, along with an Adaptive Restart Module (ARM) to se- lectively guide the restart of bidirectional por- tion of the encoder. Across four sequence tag- ging tasks, HEAR offers FLOPs savings in streaming settings upto 71.1% and also outper- forms bidirectional encoders for streaming pre- dictions by upto +10% streaming exact match. 1 Introduction State-of-the-art text encoding methods assume the offline setting, where the entire input text is avail- able when encoding it. This differs from the streaming setting where the input text grows over time (such as transcribed speech or a typed query) (Cho and Esipova, 2016; Gu et al., 2017; Chang et al., 2022). Processing streaming input incre- mentally can enable suggestions on partial inputs (Iranzo Sanchez et al., 2022), reduced final latency (Zhou et al., 2022) and lead to more interactive NLU agents (Cai and de Rijke, 2016). Existing state-of-the-art bidirectional encoders (such as Devlin et al. (2019)) are computationally ∗Work done as part of an internship at Google. Model GFLOPs Offline F1 Streaming EM BiDi encoder 74.7 93.1 75.1 HYBRID encoder 43.4 93.0 72.5 HEAR (HYBRID + ARM) 21.6 93.0 85.1 Table 1: Computation cost (Giga FLOPs), Offline F1 and Streaming Exact Match accuracy of standard bidirectional (BiDi) encoder, HYBRID encoder model and HEAR (HYBRID encoder with ARM for guiding Adaptive Restarts) on SNIPS test set. expensive for streaming processing. When a new token is received, these models require a restart, i.e., re-computation of representation of each token by re-running the bidirectional layer (Kahardipraja et al., 2021). This adds to the computational cost (i.e., Higher FLOPs) during streaming and leads to higher wall clock time. Another limitation of these encoders is poorer generalization to partial inputs (Madureira and Schlangen, 2020), stemming from these models being trained only on complete (and offline) inputs. Despite these disadvantages, such bidirectional models offer better offline per- formance than unidirectional models across sev- eral NLP tasks like sequence tagging (Kahardipraja et al., 2021). We address these …

Content truncated to 500 words