On the Importance of Diversity in Question Generation for QA

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5651–5656 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 5651 On the Importance of Diversity in Question Generation for QA Md Arafat Sultan† Shubham Chandel‡ Ramón F. Astudillo† Vittorio Castelli† †IBM Research AI, T.J. Watson Research Center, New York, USA ‡New York University, New York, USA {arafat.sultan,ramon.astudillo}@ibm.com, shubhamchandel@nyu.edu, vittorio@us.ibm.com Abstract Automatic question generation (QG) has shown promise as a source of synthetic train- ing data for question answering (QA). In this paper we ask: Is textual diversity in QG beneficial for downstream QA? Using top-p nucleus sampling to derive samples from a transformer-based question generator, we show that diversity-promoting QG indeed pro- vides better QA training than likelihood maxi- mization approaches such as beam search. We also show that standard QG evaluation metrics such as BLEU, ROUGE and METEOR are in- versely correlated with diversity, and propose a diversity-aware intrinsic measure of overall QG quality that correlates well with extrinsic evaluation on QA. 1 Question Generation and Diversity Besides areas such as dialog (Bordes et al., 2017) and tutoring systems (Lindberg et al., 2013), auto- matic question generation (QG) has recently been applied with great success to generating synthetic training examples for question answering (QA) (Al- berti et al., 2019; Dong et al., 2019). Yet an im- portant question has remained unexplored: Does increased textual diversity in automatically gener- ated questions lead to better QA? In Figure 1 we show four questions generated by one of our QG models (details in Section 2) from a SQuAD (Rajpurkar et al., 2016) passage and an answer span (the QG prompt). The questions are different not only lexically, but also in what infor- mation about the answer entity they draw upon and even their use of world knowledge, e.g., Tesla’s reputation as a “mad scientist”. Intuitively, such sample diversity, if sufficiently accurate, could pro- vide QA models with rich training signal. Existing QG work has predominantly relied on customary beam search decoding for generation and n-gram similarity metrics such as BLEU for evaluation (Du et al., 2017; Alberti et al., 2019; On Tesla’s 75th birthday in 1931, Time magazine put him on its cover. The cover caption “All the world’s his power house” noted his contribution to electrical power generation. He received congra- tulatory letters from more than 70 pioneers in science and engineering, including Albert Einstein. ✏ Who appeared on Time magazine’s cover on his 75th birthday? ✏ Which famous scientist was in the cover of Time Magazine in 1931? ✏ Which mad scientist received more than a 70 people congratulating him on his birthday? ✏ What famous scientist was also 75? Figure 1: A passage with an underlined answer span ("Tesla"), and corresponding questions generated by our model. The generated questions exhibit both lex- ical and factual diversity. Dong et al., 2019; Zhang and Bansal, 2019).1 Such methods/metrics solely optimize/reward similarity with human-generated reference questions treated as the ground truth (GT). However, in many open- ended …

Content truncated to 500 words