RetroGAN: A Cyclic Post-Specialization System for Improving Out-of-Knowledge and Rare Word Representations

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2086–2095 August 1–6, 2021. ©2021 Association for Computational Linguistics 2086 RetroGAN: A Cyclic Post-Specialization System for Improving Out-of-Knowledge and Rare Word Representations Pedro Colon-Hernandez1, Yida Xin2, Henry Lieberman3, Catherine Havasi4, Cynthia Breazeal1, and Peter Chin2 1MIT Media Lab {pe25171,cynthiab}@media.mit.edu 2Boston University {yxin,spchin}@cs.bu.edu 3MIT CSAIL lieber@media.mit.edu 4Basis Technologies∗ havasi@basistech.com Abstract Retrofitting is a technique used to move word vectors closer together or further apart in their space to reflect their relationships in a Knowl- edge Base (KB). However, retrofitting only works on concepts that are present in that KB. RetroGAN uses a pair of Generative Ad- versarial Networks (GANs) to learn a one- to-one mapping between concepts and their retrofitted counterparts. It applies that map- ping (post-specializes) to handle concepts that do not appear in the original KB in a manner similar to how some natural language systems handle out-of-vocabulary entries. We test our system on three word-similarity benchmarks and a downstream sentence simplification task, and achieve the state of the art (CARD-660). Altogether, our results demonstrate our sys- tem’s effectiveness for out-of-knowledge and rare word generalization. 1 Introduction Retrofitting word embeddings with a KB (Faruqui et al., 2015; Speer and Chin, 2016; Mrkˇsi´c et al., 2017) means taking a vector space of word embed- dings and finding a mapping that moves some of these word vectors closer together and others fur- ther apart, such that these vectors’ new positions in the vector space are in better agreement with the relationships between the same words (a.k.a., concepts) in a KB (Speer and Chin, 2016; Mrkˇsi´c et al., 2017). However, the retrofitting process can only work on concepts that are actually present in the KB (a.k.a., constraints), which means that retrofitting can get us improved performance in semantic tasks only on the overlapping vocabu- lary between the KB and the word embeddings. ∗Work done while at the MIT Media Lab Post-specialization(Vuli´c et al., 2018; Kamath et al., 2019) is a solution to this problem; it is a series of techniques that try to (1) learn the mapping that retrofitting establishes and (2) generalize the map- ping to the rest of the embedding vocabulary. We develop and present a post-specialization system called RetroGAN that builds upon the ap- proach presented as AuxGAN (Ponti et al., 2018) by extending it to have a pair of Generative Adver- sarial Networks (GANs) (Goodfellow et al., 2014). A regular GAN minimizes the loss when learning the function for post-specialization. Our pair works in a cyclic manner to minimize the losses of both the post-specialization and the inverse to ensure that there is a one-to-one mapping between the two domains. This constrains the outputs for unseen data in both domains and leads to achieving higher performance for unseen concepts. 2 Related Work Within the field of retrofitting, work has been done in exploring the various ways of infusing constraints or KBs into word embeddings. The original work by (Faruqui et al., 2015) only used synonymy relationships but not …

Content truncated to 500 words