Enhancing Pre-trained Models with Text Structure Knowledge for Question Generation

Proceedings of the 29th International Conference on Computational Linguistics, pages 6564–6574 October 12–17, 2022. 6564 Enhancing Pre-trained Models with Text Structure Knowledge for Question Generation Zichen Wu, Xin Jia, Fanyi Qu, Yunfang Wu∗ Key Laboratory of Computational Linguistics, Ministry of Education, China School of Computer Science, Peking University, China {wuzichen,jemmryx,fanyiqu,wuyf}@pku.edu.cn Abstract Today the pre-trained language models achieve great success for question generation (QG) task and significantly outperform traditional sequence-to-sequence approaches. However, the pre-trained models treat the input passage as a flat sequence and are thus not aware of the text structure of input passage. For QG task, we model text structure as answer posi- tion and syntactic dependency, and propose an- swer localness modeling and syntactic mask attention to address these limitations. Specially, we present localness modeling with a Gaussian bias to enable the model to focus on answer- surrounded context, and propose a mask atten- tion mechanism to make the syntactic struc- ture of input passage accessible in question generation process. Experiments on SQuAD dataset show that our proposed two modules im- prove performance over the strong pre-trained model ProphetNet, and combing them together achieves very competitive results with the state- of-the-art pre-trained model. 1 Introduction Question generation (QG) aims to generate ques- tions for a given passage, which is a challenging task and shows great value in many practical appli- cations. For instance, QG helps in building reading comprehension tests for course assessments (Kurdi et al., 2020); QG can be an important skill for chatbots to start a conversation with human users (Mostafazadeh et al., 2016); QG can help reduce the human labor for collecting large-scale question- answer datasets to improve question answering sys- tems (Duan et al., 2017). Existing QG methods mostly employ neural network models by treating it as a sequence-to- sequence generative task, where researchers have tried to incorporate text structure to improve the performance. For answer-aware QG, the answer ∗Corresponding author. Feed Forward Localness Modeling Syntactic Mask Attention encoder layer 𝐿𝐿 × 𝐻𝐻𝑙𝑙−1 𝐻𝐻𝑙𝑙 𝐻𝐻𝑘𝑘 𝑙𝑙 𝐻𝐻𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙 𝑙𝑙 Gaussian bias Syntactic mask Figure 1: An overall structure of our proposed methods position is widely explored. Song et al. (2018) pro- pose three strategies to match the answer with pas- sage; Sun et al. (2018) model the relative distance between context words and answer to help generate answer-related questions; Ma et al. (2020) enhance model’s ability in capturing semantic and positional information via multi-task learning. Some other neural network models dig out deep text structure for generating better questions. For instance, Chen et al. (2020) construct a passage graph to utilize text structure; Pan et al. (2020) construct a semantic- level graph to learn the global structure of a docu- ment for deep QG. However, most of these methods adopt standard neural network, and few of them are based on the strong pre-trained models. Nowadays, pre-trained models largely boost the performance of QG systems (Dong et al., 2019; Bao et al., 2020; Qi et al., 2020; Xiao et al., 2020), which utilize huge amounts of …

Content truncated to 500 words