Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly

This paper was retracted. For more information, see https://aclanthology.org/2020.acl-main.69. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 752–765 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 752 Syn-QG: Syntactic and Shallow Semantic Rules for Question Generation Kaustubh D. Dhole Amelia Science RnD, IPsoft New York, NY 10004 kdhole@ipsoft.com Christopher D. Manning Department of Computer Science Stanford University Stanford, CA 94305 manning@stanford.edu Abstract Question Generation (QG) is fundamentally a simple syntactic transformation; however, many aspects of semantics influence what questions are good to form. We implement this observation by developing Syn-QG, a set of transparent syntactic rules leveraging uni- versal dependencies, shallow semantic pars- ing, lexical resources, and custom rules which transform declarative sentences into question- answer pairs. We utilize PropBank argument descriptions and VerbNet state predicates to incorporate shallow semantic content, which helps generate questions of a descriptive na- ture and produce inferential and semantically richer questions than existing systems. In or- der to improve syntactic fluency and eliminate grammatically incorrect questions, we employ back-translation over the output of these syn- tactic rules. A set of crowd-sourced eval- uations shows that our system can generate a larger number of highly grammatical and relevant questions than previous QG systems and that back-translation drastically improves grammaticality at a slight cost of generating ir- relevant questions. 1 Introduction Automatic Question Generation (QG) is the task of generating question-answer pairs from a declar- ative sentence. It has direct use in education and generating engagement, where a system automati- cally generates questions about passages that some- one has read. A more recent secondary use is for automatic generation of questions as a data augmen- tation approach for training Question Answering (QA) systems. QG was initially approached by syn- tactic rules for question-generation, followed by some form of statistical ranking of goodness, e.g., (Heilman and Smith, 2009, 2010). In recent years, as in most areas of NLP, the dominant approach has been neural network generation (Du et al., 2017), Figure 1: The SRL structure is leveraged to invoke a template, and a simple rearrangement of the modifying arguments is performed. in particular using a sequence-to-sequence architec- ture, which exploits the data in the rapidly growing number of large QA data sets. Previous rule-based approaches suffer from a significant lack of variety in the questions they gen- erate, sticking to a few simple and reliable syntactic transformation patterns. Neural architectures pro- vide a pathway to solving this limitation since they can exploit QA datasets to learn the broad array of human question types, providing the usual neural network advantages of a data-exploiting, end-to- end trainable architecture. Nevertheless, we ob- serve that the quality of current neural QG systems is still lacking: The generated questions lack syn- tactic fluency, and the models lack transparency and an easy way to improve them. We argue that in essence QG can be governed by simple syntactic “question transformations” – while the implementation details vary, this is in accord with all major linguistic viewpoints, such …

Content truncated to 500 words