An Empirical Study of the Downstream Reliability of Pre-Trained Word Embeddings

Proceedings of the 28th International Conference on Computational Linguistics, pages 3371–3388 Barcelona, Spain (Online), December 8-13, 2020 3371 An Empirical Study of the Downstream Reliability of Pre-Trained Word Embeddings Anthony Rios and Brandon Lwowski Department of Information Systems and Cyber Security University of Texas at San Antonio San Antonio, TX 78249, USA anthony.rios@utsa.edu and oek437@my.utsa.edu Abstract While pre-trained word embeddings have been shown to improve the performance of down- stream tasks, many questions remain regarding their reliability: Do the same pre-trained word embeddings result in the best performance with slight changes to the training data? Do the same pre-trained embeddings perform well with multiple neural network architectures? Do imputation strategies for unknown words impact reliability? In this paper, we introduce two new metrics to understand the downstream reliability of word embeddings. We find that downstream reliability of word embeddings depends on multiple factors, including, the evaluation metric, the handling of out-of-vocabulary words, and whether the embeddings are fine-tuned. 1 Introduction Pre-trained word embeddings have been shown to improve neural networks’ performance across a wide variety of tasks. For instance, pretrained word embeddings improve the performance of models for text classification (Kim, 2014), relation extraction (Nguyen and Grishman, 2015), named entity recogni- tion (Lample et al., 2016), and machine translation (Qi et al., 2018). However, neural network perfor- mance is unstable when retrained multiple times on the same dataset (Kolen and Pollack, 1991). Small changes in the training data can result in substantial differences in overall performance. Similarly, after retraining word embeddings instead of the model (i.e., to incorporate out-of-vocabulary words or cap- ture changes in their semantic meanings), the instability of the word embeddings themselves, can cause differences in downstream performance (Leszczynski et al., 2020). Yet, retraining is still important. Otherwise, performance will deteriorate over time (Kim et al., 2017). In this paper, we evaluate the downstream reliability of pre-trained word embeddings. Leszczynski et al. (2020) discuss how model instability can increase a company’s cost of deploying and keeping natural language processing (NLP) pipelines in production. Let X and ¯X be two different sets of embeddings, and let gX and g ¯X be two models trained on X and ¯X. Leszczynski et al. (2020) define instability as DIT (X, ¯X) = 1 N N � i=1 L(gX(zi), g ¯X(zi)), (1) where {zi}N i=1 is a held-out test set for task some task T, and L is a fixed loss function. If L is the zero- one loss, then the measure captures the fraction of predictions that disagree between models trained on each set of embeddings. They analyze the model disagreement between two sets of embeddings trained on corpora collected from the same source at different time periods. They show that slight changes made to the beginning of an NLP pipeline (word embeddings) have a considerable impact on downstream performance. Intuitively, NLP engineers can potentially spend a lot of time searching for potential bugs caused by concept drift in word embeddings. Moreover, the time and resources required to train many state-of-the-art …

Content truncated to 500 words