A Weak Supervision Approach for Predicting Difficulty of Technical Interview Questions

Proceedings of the 29th International Conference on Computational Linguistics, pages 4537–4543 October 12–17, 2022. 4537 A Weak Supervision Approach for Predicting Difficulty of Technical Interview Questions Arpita Kundu, Subhasish Ghosh, Pratik Saini, Tapas Nayak and Indrajit Bhattacharya TCS Research, India {arpita.kundu1, g.subhasish, pratik.saini, nayak.tapas, b.indrajit}@tcs.com Abstract Predicting difficulty of questions is crucial for technical interviews. However, such questions are long-form and more open-ended than fac- toid and multiple choice questions explored so far for question difficulty prediction. Existing models also require large volumes of candidate response data for training. We study weak- supervision and use unsupervised algorithms for both question generation and difficulty pre- diction. We create a dataset of interview ques- tions with difficulty scores for Deep Learning and use it to evaluate SOTA models for ques- tion difficulty prediction trained using weak supervision. Our analysis brings out the task’s difficulty as well as the promise of weak super- vision for it. 1 Introduction For effective technical interviewing, it is important to know the question difficulty — the probability of a student from a cohort, e.g. senior undergraduate CS students, correctly answering the question. We address the problem of predicting the difficulty of interview questions for candidate cohorts. Predicting difficulty from the question statement, answer choices and related documents has been studied for multiple choice or factoid questions for reading comprehension and exams (Wang et al., 2014; Huang et al., 2017; Pado´, 2017; Qiu et al., 2019; Benedetto et al., 2020; Yaneva et al., 2020; Benedetto et al., 2021; Cheng et al., 2021; Byrd and Srivastava, 2022). All publicly available datasets (Benedetto et al., 2021; Cheng et al., 2021; Yaneva et al., 2020; Qiu et al., 2019) also contain mul- tiple choice or factoid questions. The nature of technical assessment questions in interviews is dif- ferent. These look to assess knowledge and under- standing rather than memorization of facts and are more open-ended. Answers are long-form, typi- cally spanning 2-5 sentences. Existing approaches, particularly recent deep models (Xue et al., 2020; Qiu et al., 2019; Benedetto et al., 2021), require large volumes of candidate response data to train the models. This is a challenge when creating a question bank for a new domain or a subject, since field tests need to be performed with real students. In contrast, we ex- plore training question difficulty prediction models using weak supervision based on subject textbooks and Bloom’s Taxonomy. This removes dependence on candidate responses and answer assessment. We explore various strategies of creating weakly- supervised training data. Weak supervision has been explored extensively for many NLP tasks (Li- son et al., 2020; Ratner et al., 2020; Ren et al., 2020; Awasthi et al., 2020). For question difficulty, the training data requires not just difficulty scores but interview questions as well. We explore pre-trained large language models (GPT3) and template-based algorithms for generating training questions. We then assign difficulty to these questions using an unsupervised algorithm that uses subject textbooks and Bloom’s Taxonomy (Bloom, 1956; Anderson and Krathwohl, 2014). While Bloom’s Taxonomy …

Content truncated to 500 words