Understanding Hard Negatives in Noise Contrastive Estimation

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1090–1101 June 6–11, 2021. ©2021 Association for Computational Linguistics 1090 Understanding Hard Negatives in Noise Contrastive Estimation Wenzheng Zhang and Karl Stratos Department of Computer Science Rutgers University {wenzheng.zhang, karl.stratos}@rutgers.edu Abstract The choice of negative examples is important in noise contrastive estimation. Recent works find that hard negatives—highest-scoring in- correct examples under the model—are effec- tive in practice, but they are used without a formal justification. We develop analytical tools to understand the role of hard negatives. Specifically, we view the contrastive loss as a biased estimator of the gradient of the cross- entropy loss, and show both theoretically and empirically that setting the negative distribu- tion to be the model distribution results in bias reduction. We also derive a general form of the score function that unifies various architec- tures used in text retrieval. By combining hard negatives with appropriate score functions, we obtain strong results on the challenging task of zero-shot entity linking. 1 Introduction Noise contrastive estimation (NCE) is a widely used approach to large-scale classification and re- trieval. It estimates a score function of input- label pairs by a sampled softmax objective: given a correct pair (x, y1), choose negative examples y2 . . . yK and maximize the probability of (x, y1) in a softmax over the scores of (x, y1) . . . (x, yK). NCE has been successful in many applications, in- cluding information retrieval (Huang et al., 2013), entity linking (Gillick et al., 2019), and open- domain question answering (Karpukhin et al., 2020). It is well known that making negatives “hard” can be empirically beneficial. For example, Gillick et al. (2019) propose a hard negative mining strat- egy in which highest-scoring incorrect labels under the current model are chosen as negatives. Some works even manually include difficult examples based on external information such as a ranking function (Karpukhin et al., 2020) or a knowledge base (Févry et al., 2020). While it is intuitive that such hard negatives help improve the final model by making the learning task more challenging, they are often used without a formal justification. Existing theoretical results in contrastive learning are not suitable for understand- ing hard negatives since they focus on uncondi- tional negative distributions (Gutmann and Hyväri- nen, 2012; Mnih and Teh, 2012; Ma and Collins, 2018; Tian et al., 2020) or consider a modified loss divergent from practice (Bengio and Senécal, 2008). In this work, we develop analytical tools to un- derstand the role of hard negatives. We formalize hard-negative NCE with a realistic loss (5) using a general conditional negative distribution, and view it as a biased estimator of the gradient of the cross- entropy loss. We give a simple analysis of the bias (Theorem 3.1). We then consider setting the negative distribution to be the model distribution, which recovers the hard negative mining strategy of Gillick et al. (2019), and show that it yields …

Content truncated to 500 words