End-to-End Bias Mitigation by Modelling Biases in Corpora

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 823–835 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 823 Fast and Accurate Deep Bidirectional Language Representations for Unsupervised Learning Joongbo Shin, Yoonhyung Lee, Seunghyun Yoon, Kyomin Jung Seoul National University Republic of Korea {jbshin, cpi1234, mysmilish, kjung}@snu.ac.kr Abstract Even though BERT has achieved success- ful performance improvements in various su- pervised learning tasks, BERT is still lim- ited by repetitive inferences on unsupervised tasks for the computation of contextual lan- guage representations. To resolve this limita- tion, we propose a novel deep bidirectional lan- guage model called a Transformer-based Text Autoencoder (T-TA). The T-TA computes con- textual language representations without repe- tition and displays the benefits of a deep bidi- rectional architecture, such as that of BERT. In computation time experiments in a CPU en- vironment, the proposed T-TA performs over six times faster than the BERT-like model on a reranking task and twelve times faster on a se- mantic similarity task. Furthermore, the T-TA shows competitive or even better accuracies than those of BERT on the above tasks. Code is available at https://github.com/joongbo/tta. 1 Introduction A language model is an essential component of many natural language processing (NLP) applica- tions ranging from automatic speech recognition (ASR) (Chan et al., 2016; Panayotov et al., 2015) to neural machine translation (NMT) (Sutskever et al., 2014; Sennrich et al., 2016; Vaswani et al., 2017). Recently, the Bidirectional Encoder Rep- resentations from Transformers (BERT) (Devlin et al., 2019) and its variations have led to signif- icant improvements in learning natural language representation and have achieved state-of-the-art performances on various downstream tasks such as the General Language Understanding Evalua- tion (GLUE) benchmark (Wang et al., 2019) and question answering (Rajpurkar et al., 2016). BERT continues to succeed in various unsupervised tasks, such as the N-best list reranking for ASR and NMT (Shin et al., 2019; Salazar et al., 2019), con- firming that deep bidirectional language models are useful in unsupervised applications as well. However, concerning its applications to unsuper- vised learning tasks, BERT is significantly ineffi- cient at computing language representations at the inference stage (Salazar et al., 2019). During train- ing, BERT adopts the masked language modeling (MLM) objective, which is to predict the original word of the explicitly masked word from the input sequence. Following the MLM objective, each con- textual word representation should be computed by a two-step process: masking a word in the input and feeding the result to BERT. During the inference stage, this process is repeated n times to obtain the representations of all the words within a text sequence (Wang and Cho, 2019; Shin et al., 2019; Salazar et al., 2019), resulting in a computational complexity of O(n3)1 in terms of the number of words n. Hence, it is necessary to reduce the com- putational complexity when applying the model to situations where the inference time is critical, e.g., mobile environments and real-time systems (Sanh et al., 2019; Lan et al., …

Content truncated to 500 words