One size does not fit all: Investigating strategies for differentially-private learning across NLP tasks

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 46 - 61 December 7-11, 2022 ©2022 Association for Computational Linguistics Learning to Generate Question by Asking Question: A Primal-Dual Approach with Uncommon Word Generation Qifan Wang1, Li Yang2, Xiaojun Quan3, Fuli Feng4, Dongfang Liu5, Zenglin Xu6, Sinong Wang1 and Hao Ma1 1Meta AI 2Google Research 3Sun Yat-sen University 4University of Science and Technology of China 5Rochester Institute of Technology 6Harbin Institute of Technology wqfcr@fb.com lyliyang@google.com Abstract Automatic question generation (AQG) is the task of generating a question from a given pas- sage and an answer. Most existing AQG meth- ods aim at encoding the passage and the an- swer to generate the question. However, limited work has focused on modeling the correlation between the target answer and the generated question. Moreover, unseen or rare word gen- eration has not been studied in previous works. In this paper, we propose a novel approach which incorporates question generation with its dual problem, question answering, into a unified primal-dual framework. Specifically, the question generation component consists of an encoder that jointly encodes the answer with the passage, and a decoder that produces the question. The question answering component then re-asks the generated question on the pas- sage to ensure that the target answer is obtained. We further introduce a knowledge distillation module to improve the model generalization ability. We conduct an extensive set of experi- ments on SQuAD and HotpotQA benchmarks. Experimental results demonstrate the superior performance of the proposed approach over sev- eral state-of-the-art methods. 1 Introduction Question answering (Hsu et al., 2021) plays a cru- cial role in both the growth of human beings and the improvement of artificial intelligent systems. As a dual task of question answering, automatic ques- tion generation (AQG) (Cheng et al., 2021) based on a passage and a target answer has attracted much attention in recent years. One of its key applica- tions is to generate questions for educational ma- terials (Heilman and Smith, 2010). Another appli- cation is automatically producing question-answer pairs to enhance machine reading comprehension systems (Du et al., 2017; Lyu et al., 2021). Besides, AQG is also widely used in building web answering system (Shou et al., 2020; You et al., 2021; Wang Figure 1: An example of generated questions from hu- man, base models and our model. The purple text in the passage indicates the target answer. Our model generates a more desirable question compared to the questions generated from the base models. Our model is able to generate the uncommon word, attend, that does not appear in the passage. et al., 2022), conversational dialog systems (Liu et al., 2021; Shen et al., 2021; Huang et al., 2022) and chatbots (Gros et al., 2021) such as Siri, Cor- tana, Alexa and Google Assistant, helping them to start and continue a conversation with human users. Automatic question generation is a challenging task due to the unstructured nature of textual data. Early research (Rus et al., 2010; Labutov et …

Content truncated to 500 words