Learning with Noise-Contrastive Estimation: Easing training by learning to scale

Proceedings of the 27th International Conference on Computational Linguistics, pages 3090–3101 Santa Fe, New Mexico, USA, August 20-26, 2018. 3090 Learning with Noise-Contrastive Estimation: Easing training by learning to scale. Matthieu Labeau and Alexandre Allauzen LIMSI, CNRS, Univ. Paris-Sud, Universit´e Paris-Saclay, 91405 Orsay, France {matthieu.labeau,alexandre.allauzen}@limsi.fr Abstract Noise-Contrastive Estimation (NCE) is a learning criterion that is regularly used to train neural language models in place of Maximum Likelihood Estimation, since it avoids the computational bottleneck caused by the output softmax. In this paper, we analyse and explain some of the weaknesses of this objective function, linked to the mechanism of self-normalization, by closely monitoring comparative experiments. We then explore several remedies and modifications to propose tractable and efficient NCE training strategies. In particular, we propose to make the scaling factor a trainable parameter of the model, and to use the noise distribution to initialize the output bias. These solutions, yet simple, yield stable and competitive performances in either small and large scale language modelling tasks. 1 Introduction In many tasks, such as machine translation and speech recognition, statistical language models1 play a key role. Neural models (Bengio et al., 2003; Mikolov et al., 2010; J´ozefowicz et al., 2016) have recently shown great improvement. However most of them share a common issue: a large output vocabulary implies a prohibitive computation time, due to the output normalization, along with a prediction challenge in a such high dimensional space. A workaround is to reduce the vocabulary size by considering sub-word units like morphemes or even characters. For several applications, this is an efficient solution, though the main issue is not directly addressed. Other trends consist in changing the output structure by using shortlists (Schwenk, 2007) or hierarchical softmax (Morin and Bengio, 2005; Mnih and Hinton, 2009; Le et al., 2011), while self-normalisation techniques (Devlin et al., 2014; Andreas et al., 2015; Chen et al., 2016) partially solve the issue at test time. Finally, sampling-based techniques like Importance sampling (Bengio and S´en´ecal, 2003; Jean et al., 2015), and Noise-Contrastive Estimation or NCE (Mnih and Teh, 2012) are also promising alternatives. In this work, we focus on NCE, which uses a discriminative objective that approximates negative log- likelihood. Its main advantage is to consider the model as un-normalized instead of trying to approximate its normalization. With this objective function the model is explicitly learnt to estimate un-normalized probability distributions, while the partition function (or the scaling factor) is parametrized separately. While this method is very appealling in theory, empirical issues arise in large vocabulary applications: training divergence and instability, along with poor performance when compared to similar approaches, notably Importance Sampling. The contributions of this paper are twofold: first an empirical exploration of the NCE allows us to better explain the estimation process and why it sometimes fails; then we propose and explore several remedies yielding to tractable and efficient NCE training strategies. While section 2 reviews two widely used training criteria for un-normalized model, Importance Sampling and NCE, section 3 provides an empirical analysis of …

Content truncated to 500 words