Semantic Shift Stability: Efficient Way to Detect Performance Degradation of Word Embeddings and Pre-trained Language Models

Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 205–216 November 20–23, 2022. ©2022 Association for Computational Linguistics 205 Semantic Shift Stability: Efficient Way to Detect Performance Degradation of Word Embeddings and Pre-trained Language Models Shotaro Ishihara ∗ Nikkei, Inc. Hiromu Takahashi ∗ Independent researcher shotaro.ishihara@nex.nikkei.com hiromu.takahashi56@gmail.com hono.shirai@nex.nikkei.com Hono Shirai Nikkei, Inc. Abstract Word embeddings and pre-trained language models have become essential technical ele- ments in natural language processing. While the general practice is to use or fine-tune pub- licly available models, there are significant ad- vantages in creating or pre-training unique mod- els that match the domain. The performance of the models degrades as language changes or evolves continuously (semantic shift), but the high cost of model building inhibits regular re-training, especially for the language mod- els. This study designs a methodology for ob- serving time-series performance degradation of word embeddings and pre-trained language models using semantic shift in a corpus. We define an efficiently computable metric named Semantic Shift Stability based on the degree of semantic shift. In the experiments, we cre- ate models that vary by time series and reveal the performance degradation in two datasets, Japanese and English. Several case studies demonstrate that Semantic Shift Stability sup- ports decision-making as to whether a model should be re-trained. The source code is avail- able at https://github.com/Nikkei/ semantic-shift-stability. 1 Introduction The use of word embeddings and pre-trained lan- guage models has become common practice in natural language processing. Word embeddings like word2vec (Mikolov et al., 2013) are used in many applications, and pre-trained language mod- els starting with BERT (Devlin et al., 2019) are updating state-of-the-art performance on a daily basis. Researchers and developers use or fine-tune such kinds of models to their own tasks. While the general practice is to start from pub- licly available models, there are also significant advantages in creating or pre-training unique mod- els that match the domain. In regard to pre-trained ∗ These authors contributed equally. Figure 1: Methodology for observing time-series per- formance degradation by Semantic Shift Stability. It is difficult from a cost perspective to create a pre-train lan- guage model each time and compare the performance. Instead, by monitoring the degree of semantic shift of the corpora from period to period, we can estimate time- series performance degradation. language models, for example, SciBERT (Beltagy et al., 2019), BioBERT (Lee et al., 2020), and Fin- BERT (Araci, 2019) are proposed. These models have performed better than other BERT models on downstream domain-specific tasks. A similar ap- proach is traditionally used in word embeddings. There are numerous studies and applications of obtaining word embeddings in their own corpora. In creating domain-specific language models, we have to be careful of time-series changes in the characteristics of the corpus. Language changes continuously, especially when there are some so- cially important events. The semantic shift (Kutu- zov et al., 2018) of existing words …

Content truncated to 500 words