Improving Temporal Generalization of Pre-trained Language Models with Lexical Semantic Change

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 670 - 688 December 7-11, 2022 ©2022 Association for Computational Linguistics Generative Language Models for Paragraph-Level Question Generation Asahi Ushio and Fernando Alva-Manchego and Jose Camacho-Collados Cardiff NLP, School of Computer Science and Informatics, Cardiff University, UK {UshioA,AlvaManchegoF,CamachoColladosJ}@cardiff.ac.uk Abstract Powerful generative models have led to recent progress in question generation (QG). How- ever, it is difficult to measure advances in QG research since there are no standardized resources that allow a uniform comparison among approaches. In this paper, we intro- duce QG-Bench, a multilingual and multido- main benchmark for QG that unifies existing question answering datasets by converting them to a standard QG setting. It includes general- purpose datasets such as SQuAD (Rajpurkar et al., 2016) for English, datasets from ten do- mains and two styles, as well as datasets in eight different languages. Using QG-Bench as a reference, we perform an extensive analy- sis of the capabilities of language models for the task. First, we propose robust QG base- lines based on fine-tuning generative language models. Then, we complement automatic eval- uation based on standard metrics with an ex- tensive manual evaluation, which in turn sheds light on the difficulty of evaluating QG models. Finally, we analyse both the domain adaptabil- ity of these models as well as the effective- ness of multilingual models in languages other than English. QG-Bench is released along with the fine-tuned models presented in the paper,1 which are also available as a demo.2 1 Introduction Question generation (QG, Mitkov and Ha, 2003) is the task of generating a question given an in- put context consisting of a document, a paragraph or a sentence, and an answer where the question is anchored (see Figure 1). QG has been widely studied in natural language processing communi- ties (Du et al., 2017; Zhou et al., 2017; Du and Cardie, 2018), and it has recently been exploited to train question answering (QA) models without human supervision (Lewis et al., 2019; Zhang and 1https://github.com/asahi417/ lm-question-generation 2https://autoqg.net/ Figure 1: Overview of paragraph-level QG. Bansal, 2019; Puri et al., 2020), or as a means of data augmentation (Shakeri et al., 2020; Bartolo et al., 2021). It has also been applied to develop educational systems (Heilman and Smith, 2010; Lindberg et al., 2013), information retrieval mod- els (Pyatkin et al., 2021; Lewis et al., 2021), and for model interpretation (Perez et al., 2020; Lee et al., 2020). Despite its success in downstream applications, the development of neural QG models has received less attention. For example, the choice of the base pre-trained model is arbitrary (without proper justi- fication in most cases) as it is not straightforward to compare different models. As a consequence, while ERNIE-GEN (Xiao et al., 2021) and UniLMv2 (Bao et al., 2020) are current SotA in the SQuAD QG benchmark (Du et al., 2017), T5 (Raffel et al., 2020) and BART (Lewis et al., 2020a) are used in many applications in practice (Paranjape et al., 2021; …

Content truncated to 500 words