Designing Templates for Eliciting Commonsense Knowledge from Pretrained Sequence-to-Sequence Models

Proceedings of the 28th International Conference on Computational Linguistics, pages 3449–3453 Barcelona, Spain (Online), December 8-13, 2020 3449 Designing Templates for Eliciting Commonsense Knowledge from Pretrained Sequence-to-Sequence Models Sheng-Chieh Lin∗1, Jheng-Hong Yang∗1, Rodrigo Nogueira2, Ming-Feng Tsai1,3, Chuan-Ju Wang1 and Jimmy Lin2 1Research Center for Information Technology Innovation, Academia Sinica 2David R. Cheriton School of Computer Science, University of Waterloo 3Department of Computer Science, National Chenchi University Abstract While internalized “implicit knowledge” in pretrained transformers has led to fruitful progress in many natural language understanding tasks, how to most effectively elicit such knowledge remains an open question. Based on the text-to-text transfer transformer (T5) model, this work explores a template-based approach to extract implicit knowledge for commonsense reasoning on multiple-choice (MC) question answering tasks. Experiments on three representative MC datasets show the surprisingly good performance of our simple template, coupled with a logit normalization technique for disambiguation. Furthermore, we verify that our proposed template can be easily extended to other MC tasks with contexts such as supporting facts in open-book question answering settings. Starting from the MC task, this work initiates further research to find generic natural language templates that can effectively leverage stored knowledge in pretrained models. 1 Introduction Scaling and unifying deep neural models via building task-agnostic architectures and pretraining objectives have brought about significant progress in performing various downstream natural language understanding tasks (Devlin et al., 2019; Yang et al., 2019; Liu et al., 2019; Lewis et al., 2019; Raffel et al., 2020). According to recent findings, impressive performance can be attributed to a sort of internalized implicit “knowledge base” contained in pretrained models (Petroni et al., 2019; Jiang et al., 2019). Since such knowledge is drawn from unstructured and unlabeled text via the design of neural architectures and pretraining objectives, it is non-trivial to identify an effective way to explicitly elicit such stored knowledge for each NLP task. Challenging NLP tasks that require commonsense reasoning abilities (or background knowledge) could benefit from implicit knowledge captured in pretrained models. Various research lines have explored techniques that effectively leverage reasoning abilities based on BERT (Devlin et al., 2019). For example, Sakaguchi et al. (2019) use a classification head capped on pretrained BERT variants to tackle commonsense reasoning. Zhu et al. (2020) propose to perturb token embeddings to improve model performance and robustness. Alternatively, many researchers recast the multiple-choice (MC) problem setting, which is frequently used in verbal reasoning benchmarks, as a ranking problem regarding statement possibilities (Li et al., 2019; Pirtoaca et al., 2019). Tamborrino et al. (2020) further present a hybrid method that combines full-text inputs, pretrained token representations, and learning-to-rank objectives. Finally, Khashabi et al. (2020) propose to unify and fine-tune pretrained models on many QA tasks jointly. As an alternative to designing task-specific architectures on top of pretrained transformers, we propose a new approach to downstream NLP tasks based on task-specific templates exploiting existing pretrained sequence-to-sequence models. We aim to identify a text-to-text approach that effectively elicits implicit knowledge embedded in the pretrained text-to-text transfer transformer …

Content truncated to 500 words