Graph-based Relation Mining for Context-free Out-of-vocabulary Word Embedding Learning

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 14133–14149 July 9-14, 2023 ©2023 Association for Computational Linguistics Graph-based Relation Mining for Context-free Out-of-vocabulary Word Embedding Learning Ziran Liang and Yuyin Lu and Hegang Chen and Yanghui Rao∗ School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China {liangzr5,luyy37,chenhg25}@mail2.sysu.edu.cn, raoyangh@mail.sysu.edu.cn Abstract The out-of-vocabulary (OOV) words are dif- ficult to represent while critical to the perfor- mance of embedding-based downstream mod- els. Prior OOV word embedding learning methods failed to model complex word forma- tion well. In this paper, we propose a novel graph-based relation mining method, namely GRM, for OOV word embedding learning. We first build a Word Relationship Graph (WRG) based on word formation and associate OOV words with their semantically relevant words, which can mine the relational information in- side word structures. Subsequently, our GRM can infer high-quality embeddings for OOV words through passing and aggregating se- mantic attributes and relational information in the WRG, regardless of contextual richness. Extensive experiments demonstrate that our model significantly outperforms state-of-the- art baselines on both intrinsic and downstream tasks when faced with OOV words. 1 Introduction Pre-trained word embedding models, such as Word2Vec (Mikolov et al., 2013) and BERT (De- vlin et al., 2019), can not only boost the perfor- mance of downstream tasks but also accelerate the convergence of downstream models (Kuratov and Arkhipov, 2019; Kao and Lee, 2021). However, in real-world scenarios, the pre-trained models trained with generic large-scaled corpora may en- counter a lot of words never seen before in down- stream tasks due to domain specificity. These out- of-vocabulary (OOV) words rarely appear, result- ing in a scarcity of their contexts, while traditional word embedding methods require a large number of contexts to learn high-quality word embeddings (Herbelot and Baroni, 2017). The OOV words may cause a dramatic performance degradation in downstream tasks because of their poor word em- beddings (Nayak et al., 2020; Schick and Schütze, ∗The corresponding author. 2020; Won et al., 2021), which leads to the OOV problem. Thus, it is vital to explore an effective way of learning high-quality OOV word embed- dings in natural language processing. Traditional methods for tackling the OOV prob- lem injected sub-units of words into the training process of pre-trained word embedding models to get the sub-unit embeddings and then calculated the OOV word embedding as a summation of them (Bojanowski et al., 2017; Cao et al., 2018; De- vlin et al., 2019). These methods require training from scratch, which are time consuming. To save computing resources, two categories of methods have been proposed. Methods in the first cate- gory attempted to fully utilize limited contextual information carried by OOV words directly with- out modifying the training process of background models (Garneau et al., 2018; Hu et al., 2019; Schick and Schütze, 2019). These methods are of- ten lightweight, but they cannot deal with some frequently occurring situations where the OOV words are extremely context-less. To break the limitation …

Content truncated to 500 words