Topic Sensitive Attention on Generic Corpora Corrects Sense Bias in Pretrained Embeddings

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1717–1726 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1717 Topic Sensitive Attention on Generic Corpora Corrects Sense Bias in Pretrained Embeddings Vihari Piratla∗ IIT Bombay Sunita Sarawagi IIT Bombay Soumen Chakrabarti IIT Bombay Abstract Given a small corpus DT pertaining to a lim- ited set of focused topics, our goal is to train embeddings that accurately capture the sense of words in the topic in spite of the limited size of DT . These embeddings may be used in various tasks involving DT . A popular strat- egy in limited data settings is to adapt pre- trained embeddings E trained on a large cor- pus. To correct for sense drift, fine-tuning, reg- ularization, projection, and pivoting have been proposed recently. Among these, regulariza- tion informed by a word’s corpus frequency performed well, but we improve upon it us- ing a new regularizer based on the stability of its cooccurrence with other words. How- ever, a thorough comparison across ten top- ics, spanning three tasks, with standardized settings of hyper-parameters, reveals that even the best embedding adaptation strategies pro- vide small gains beyond well-tuned baselines, which many earlier comparisons ignored. In a bold departure from adapting pretrained em- beddings, we propose using DT to probe, at- tend to, and borrow fragments from any large, topic-rich source corpus (such as Wikipedia), which need not be the corpus used to pretrain embeddings. This step is made scalable and practical by suitable indexing. We reach the surprising conclusion that even limited corpus augmentation is more useful than adapting em- beddings, which suggests that non-dominant sense information may be irrevocably obliter- ated from pretrained embeddings and cannot be salvaged by adaptation. 1 Introduction Word embeddings (Mikolov et al., 2013; Penning- ton et al., 2014) benefit many natural language processing (NLP) tasks. Often, a group of tasks may involve a limited corpus DT pertaining to a few focused topics, e.g., discussion boards on ∗ vihari@cse.iitb.ac.in Physics, video games, or Unix, or a forum for discussing medical literature. Because DT may be too small to train word embeddings to suf- ficient quality, a prevalent practice is to harness general-purpose embeddings E pretrained on a broad-coverage corpus, not tailored to the topics of interest. The pretrained embeddings are some- times used as-is (‘pinned’). Even if E is trained on a ‘universal’ corpus, considerable sense shift may exist in the meaning of polysemous words and their cooccurrences and similarities with other words. In a corpus about Unix, ‘cat’ and ‘print’ are more similar than in Wikipedia. ‘Charge’ and ‘potential’ are more related in a Physics corpus than in Wikipedia. Thus, pinning can lead to poor target task performance in case of serious sense mismatch. Another popular practice is to initial- ize the target embeddings to the pretrained vectors, but then “fine-tune” using DT to improve perfor- mance in the target (Mou et al., 2015; Min et al., 2017; Howard and …

Content truncated to 500 words