LittleBird: Efficient Faster & Longer Transformer for Question Answering

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5261 - 5277 December 7-11, 2022 ©2022 Association for Computational Linguistics LittleBird: Efficient Faster & Longer Transformer for Question Answering Minchul Lee and Kijong Han and Myeong Cheol Shin Kakao Enterprise Corp., South Korea {phil.i,mat.h,index.sh}@kakaoenterprise.com Abstract BERT has shown a lot of sucess in a wide variety of NLP tasks. But it has a limitation dealing with long inputs due to its attention mechanism. Longformer, ETC and BigBird ad- dressed this issue and effectively solved the quadratic dependency problem. However we find that these models are not sufficient, and propose LittleBird, a novel model based on BigBird with improved speed and memory foot- print while maintaining accuracy. In particular, we devise a more flexible and efficient posi- tion representation method based on Attention with Linear Biases (ALiBi). We also show that replacing the method of global information rep- resented in the BigBird with pack and unpack attention is more effective. The proposed model can work on long inputs even after being pre- trained on short inputs, and can be trained ef- ficiently reusing existing pre-trained language model for short inputs. This is a significant benefit for low-resource languages where large amounts of long text data are difficult to ob- tain. As a result, our experiments show that LittleBird works very well in a variety of lan- guages, achieving high performance in question answering tasks, particularly in KorQuAD2.0, Korean Question Answering Dataset for long paragraphs. 1 Introduction Transformer (Vaswani et al., 2017) and pre-trained language models (Devlin et al., 2019; Liu et al., 2019) based on it have shown a lot of success in a wide variety of NLP tasks. However, the quadratic dependency problem that comes from the attention mechanism makes it impractical to process long documents. Many techniques have been studied to overcome this problem and BigBird (Zaheer et al., 2020) showed robust and state-of-the-art perfor- mance on various NLP downstream tasks. In this study, we propose a new model LittleBird by analyzing and improving the shortcomings of BigBird. LittleBird shows improved speed and memory footprint compared to BigBird while main- taining the overall accuracy of the question answer- ing (QA) benchmarks and showing better accuracy in some of them. In this study, we propose three major improve- ments compared to BigBird. The first is the method for position representation. In BigBird, trainable positional embedding is used similar to BERT (De- vlin et al., 2019), and in ETC, relative positional encoding is used similar to T5 (Raffel et al., 2020). However, trainable positional embedding cannot handle longer inputs than those used for training and the relative position encoding is relatively slow and uses extra memory and parameters (Ma et al., 2021). Press et al. (2021) introduced the attention with linear biases (ALiBi) method that resolves these problems, but it was designed for causal lan- guage modeling, not autoencoding language mod- eling, which is typically useful for QA tasks. Thus, we devise a new method based on …

Content truncated to 500 words