Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4249–4259, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4249 Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds John P. Lalor1∗, Hao Wu2, Hong Yu1,3,4 1College of Information and Computer Sciences, UMass Amherst 2Department of Psychology and Human Development, Vanderbilt University 3Department of Computer Science, UMass Lowell 4Center for Healthcare Organization & Implementation Research, Bedford VA john.lalor@nd.edu, hao.wu.1@vanderbilt.edu, hong.yu@umassmed.edu Abstract Incorporating Item Response Theory (IRT) into NLP tasks can provide valuable informa- tion about model performance and behavior. Traditionally, IRT models are learned using hu- man response pattern (RP) data, presenting a significant bottleneck for large data sets like those required for training deep neural net- works (DNNs). In this work we propose learn- ing IRT models using RPs generated from arti- ficial crowds of DNN models. We demonstrate the effectiveness of learning IRT models using DNN-generated data through quantitative and qualitative analyses for two NLP tasks. Param- eters learned from human and machine RPs for natural language inference and sentiment analysis exhibit medium to large positive cor- relations. We demonstrate a use-case for la- tent difficulty item parameters, namely train- ing set filtering, and show that using difficulty to sample training data outperforms baseline methods. Finally, we highlight cases where hu- man expectation about item difficulty does not match difficulty as estimated from the machine RPs. 1 Introduction What is the most difficult example in the Stanford Natural Language Inference (SNLI) data set (Bow- man et al., 2015) or in the Stanford Sentiment Tree- bank (SSTB) (Socher et al., 2013)? A priori the answer is not clear. How does one quantify the difficulty of an example and does it pertain to a specific model, or more generally? There has been much recent work trying to as- sess the quality of data sets used for NLP tasks, (e.g. Lalor et al., 2016; Sakaguchi and Van Durme, ∗Current affiliation: Mendoza College of Business, Uni- versity of Notre Dame 2018; Kaushik and Lipton, 2018). In particular, a common finding is that different examples within the same class have very different qualities such as difficulty, and these differences affect models’ performance. For example, one study found that a subset of reading comprehension questions were so difficult as to be unanswerable (Kaushik and Lipton, 2018). In another work, the difficulty of specific items was found to be a significant pre- dictor of whether a model would classify the item correctly (Lalor et al., 2018). While a number of methods exist for estimating difficulty, in this work we focus on Item Response Theory (IRT) (Baker, 2001; Baker and Kim, 2004), a widely used method in psychometrics. IRT mod- els fit parameters of data points (called “items”) such as difficulty based on a large number of an- notations (“response patterns” or RPs), typically gathered from a human population (“subjects”). It has been shown to be …

Content truncated to 500 words