On Importance Sampling-Based Evaluation of Latent Language Models

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2171–2176 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2171 On Importance Sampling-Based Evaluation of Latent Language Models Robert L. Logan IV Univ. of California, Irvine rlogan@uci.edu Matt Gardner Allen Institute for AI mattg@allenai.org Sameer Singh Univ. of California, Irvine sameer@uci.edu Abstract Language models that use additional latent structures (e.g., syntax trees, coreference chains, and knowledge graph links) provide several advantages over traditional language models. However, likelihood-based evaluation of these models is often intractable as it re- quires marginalizing over the latent space. Ex- isting methods avoid this issue by using im- portance sampling. Although this approach has asymptotic guarantees, analysis is rarely conducted on the effect of decisions such as sample size, granularity of sample aggregation, and the proposal distribution on the reported estimates. In this paper, we measure the effect these factors have on perplexity estimates for three different latent language models. In addi- tion, we elucidate subtle differences in how im- portance sampling is applied, which can have substantial effects on the final estimates, as well as provide theoretical results that rein- force the validity of importance sampling for evaluating latent language models. 1 Introduction Latent language models are generative models of text that jointly represent the text and the latent structure underlying it, such as: the syntactic parse, coreference chains between entity mentions, or links of entities and relations mentioned in the text to an external knowledge graph. The benefits of modeling such structure include interpretabil- ity (Hayashi et al., 2020), better performance on tasks requiring structure (Dyer et al., 2016; Ji et al., 2017), and improved ability to generate consistent mentions of entities (Clark et al., 2018) and fac- tually accurate text (Logan et al., 2019). Unfor- tunately, demonstrating that these models provide better performance than traditional language mod- els by evaluating their likelihood on benchmark data can be difficult, as exact computation requires marginalizing over all possible latent structures. Existing approaches evaluate their models by es- timating likelihoods using importance sampling, i.e. a weighted average over latent states sampled from a proposal distribution. Although convergence of importance sampled estimates is asymptotically guaranteed, results are typically produced using a small number of samples for which this guaran- tee does not necessarily apply. Furthermore, these works employ a variety of heuristics—such as sam- pling from proposal distributions that are condi- tioned on future gold tokens the model is being evaluated on, and changing the temperature of the proposal distribution—without providing measure- ments of the effect these decisions have on esti- mated perplexity, and often omitting details crucial to replicating their results. In this paper, we seek to fill in this missing knowledge, and put this practice on more rigorous footing. First, we review the theory of importance sampling, providing proof that importance sampled perplexity estimates are stochastic upper bounds of the true perplexity—a previously unnoted justifica- tion for this evaluation technique. In addition, we compile a list of common practices used …

Content truncated to 500 words