Delta Embedding Learning

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3329–3334 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3329 Delta Embedding Learning Xiao Zhang∗ Ji Wu∗ Dejing Dou† ∗Department of Electronic Engineering, Tsinghua University †Department of Computer and Information Science, University of Oregon xiphzx@gmail.com wuji ee@mail.tsinghua.edu.cn dou@cs.uoregon.edu Abstract Unsupervised word embeddings have become a popular approach of word representation in NLP tasks. However there are limita- tions to the semantics represented by unsuper- vised embeddings, and inadequate fine-tuning of embeddings can lead to suboptimal perfor- mance. We propose a novel learning technique called Delta Embedding Learning, which can be applied to general NLP tasks to improve performance by optimized tuning of the word embeddings. A structured regularization is ap- plied to the embeddings to ensure they are tuned in an incremental way. As a result, the tuned word embeddings become better word representations by absorbing semantic infor- mation from supervision without “forgetting.” We apply the method to various NLP tasks and see a consistent improvement in performance. Evaluation also confirms the tuned word em- beddings have better semantic properties. 1 Introduction Unsupervised word embeddings have become the basis for word representation in NLP tasks. Mod- els such as skip-gram (Mikolov et al., 2013a) and Glove (Pennington et al., 2014) capture the statis- tics of a large corpus and have good properties that corresponds to the semantics of words (Mikolov et al., 2013b). However there are certain problems with unsupervised word embeddings, such as the difficulty in modeling some fine-grained word se- mantics. For example words in the same category but with different polarities are often confused be- cause those words share common statistics in the corpus (Faruqui et al., 2015; Mrkˇsi´c et al., 2016). In supervised NLP tasks, these unsupervised word embeddings are often used in one of two ways: keeping fixed or using as initialization (fine- tuning). The decision is made based on the amount of available training data in order to avoid overfit- ting. Nonetheless, underfitting with keeping fixed and certain degrees of overfitting with fine-tuning is inevitable. Because this all or none optimiza- tion of the word embeddings lacks control over the learning process, the embeddings are not trained to an optimal point, which can result in suboptimal task performance, as we will show later. In this paper, we propose delta embedding learning, a novel method that aims to address the above problems together: using regularization to find the optimal fine-tuning of word embed- dings. Better task performance can be reached with properly optimized embeddings. At the same time, the regularized fine-tuning effectively com- bines semantics from supervised learning and un- supervised learning, which addresses some limi- tations in unsupervised embeddings and improves the quality of embeddings. Unlike retrofitting (Yu and Dredze, 2014; Faruqui et al., 2015), which learns directly from lexical resources, our method provides a way to learn word semantics from supervised NLP tasks. Embeddings usually become task-specific and lose its generality when trained along with …

Content truncated to 500 words