Self-Attention Architectures for Answer-Agnostic Neural Question Generation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6027–6032 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6027 Self-Attention Architectures for Answer-Agnostic Neural Question Generation Thomas Scialom LIP6 - Sorbonne Universit´es reciTAL thomas@recital.ai Benjamin Piwowarski CNRS LIP6 - Sorbonne Universit´es UPMC Univ Paris 06 UMR 7606 Benjamin.Piwowarski@lip6.fr Jacopo Staiano reciTAL jacopo@recital.ai Abstract Neural architectures based on self-attention, such as Transformers, recently attracted inter- est from the research community, and obtained significant improvements over the state of the art in several tasks. We explore how Trans- formers can be adapted to the task of Neu- ral Question Generation without constraining the model to focus on a specific answer pas- sage. We study the effect of several strategies to deal with out-of-vocabulary words such as copy mechanisms, placeholders, and contex- tual word embeddings. We report improvements obtained over the state-of-the-art on the SQuAD dataset accord- ing to automated metrics (BLEU, ROUGE), as well as qualitative human assessments of the system outputs. 1 Introduction The Machine Reading Comprehension (MRC) community focuses on the development of mod- els and algorithms allowing machines to correctly represent the meaning imbued in natural sen- tences, in order to perform useful and valuable high-level downstream tasks such as providing an- swers to questions, generate summaries, and gen- erate relevant questions given a piece of text. Per- formance on those downstream tasks is indicative of the extent to which the different proposed archi- tectures are able to capture meaning from natural language input. Recently, neural architectures based on self- attention have obtained significant improvements over the state of the art in several tasks such as language modelling and machine translation, for which abundant data is available. Yet, they have not been thoroughly evaluated on problems for which relatively scarcer datasets are available. We thus investigate the application of Transformers to the task of Neural Question Generation (NQG): given a text snippet, the model is called to gen- erate relevant and meaningful questions about it. Question Generation (QG) is an active field of research within the context of machine reading. it matches human behavior when assessing com- prehension on a given topic: an expert is able to ask the relevant questions to others to assess their competences. Its potential applications cover a broad range of scenarios, such as Information Re- trieval, chat-bots, AI-supported learning technolo- gies. Furthermore, it can be used as a strategy for data augmentation in the context of Question An- swering systems. The QG task has been originally tackled using rule-based systems (Rus et al., 2010), with the re- search community turning to neural approaches in recent years. In its most popular declination, the task is answer-aware, i.e. the target answer within the source text is known and given as in- put to the QG model (Zhou et al., 2017). Under this scenario, Song et al. (2017) proposed a gener- ative model, jointly trained for question generation and answering. More recently, Zhao et al. (2018) obtained state-of-the-art …

Content truncated to 500 words