Attentively Embracing Noise for Robust Latent Representation in BERT

Proceedings of the 28th International Conference on Computational Linguistics, pages 3479–3491 Barcelona, Spain (Online), December 8-13, 2020 3479 Attentively Embracing Noise for Robust Latent Representation in BERT Gwenaelle Cunha Sergio1, Dennis Singh Moirangthem1, Minho Lee2 1 School of Electronics and Electrical Engineering 2 Department of Artificial Intelligence Kyungpook National University, Daegu, 41566, South Korea {gwena.cs,mdennissingh,mholee}@gmail.com Abstract Modern digital personal assistants interact with users through voice. Therefore, they heavily rely on automatic speech recognition (ASR) in order to convert speech to text and perform further tasks. We introduce EBERT, which stands for EmbraceBERT, with the goal of extracting more robust latent representations for the task of noisy ASR text classification. Conventionally, BERT is fine-tuned for downstream classification tasks using only the [CLS] starter token, with the remaining tokens being discarded. We propose using all encoded transformer tokens and further encode them using a novel attentive embracement layer and multi-head attention layer. This approach uses the otherwise discarded tokens as a source of additional information and the multi- head attention in conjunction with the attentive embracement layer to select important features from clean data during training. This allows for the extraction of a robust latent vector resulting in improved classification performance during testing when presented with noisy inputs. We show the impact of our model on both the Chatbot and Snips corpora for intent classification with ASR error. Results, in terms of F1-score and mean between 10 runs, show that our model significantly outperforms the baseline model.1 1 Introduction In recent year, machine learning methods have significantly contributed to the development of automatic speech recognition (ASR) (Deng and Li, 2013; Padmanabhan and Johnson Premkumar, 2015; Benkerzaz et al., 2019; Nassif et al., 2019). The fast advances in the ASR field can be explained by the urgent need of efficiently bridging the gap between humans and the ever evolving technologies surrounding us. As technology becomes smaller and more portable, it becomes clear that the only feasible way of interaction is through a digital personal assistant with voice interface (Tulshan and Dhage, 2018). However, high error rates still prevail and pose a tremendous hurdle in the widespread adoption of speech technology by users worldwide (Zavareh et al., 2013; Errattahi et al., 2018). Researchers in the computational linguistics community analyze noisy text (Subramaniam, 2010) ob- tained through informal settings, such as social networks, or through processing techniques, such as optical character recognition (Granet et al., 2018; Tomokiyo et al., 2018), ASR (Vinciarelli, 2005; Agar- wal et al., 2007; Shrestha et al., 2019), or machine translation models (Sergio et al., 2018). We focus on noisy ASR text classification. Researchers in this area have simulated noise introduced through ASR sys- tems and they have investigated the performances of Naive Bayes and Support Vector Machine (SVM) classifiers in noisy text classification (Vinciarelli, 2005; Agarwal et al., 2007). Shrestha et al. (2019) have also investigated the impact in performance of using text embedded with speech data with neural networks and logistic regression. The aforementioned researches, however, focus mainly on …

Content truncated to 500 words