Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1192–1197, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1192 Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks Zi-Yi Dou, Keyi Yu, Antonios Anastasopoulos Language Technologies Institute, Carnegie Mellon University {zdou, keyiy, aanastas}@cs.cmu.edu Abstract Learning general representations of text is a fundamental problem for many natural lan- guage understanding (NLU) tasks. Previ- ously, researchers have proposed to use lan- guage model pre-training and multi-task learn- ing to learn robust representations. However, these methods can achieve sub-optimal perfor- mance in low-resource scenarios. Inspired by the recent success of optimization-based meta- learning algorithms, in this paper, we explore the model-agnostic meta-learning algorithm (MAML) and its variants for low-resource NLU tasks. We validate our methods on the GLUE benchmark and show that our proposed models can outperform several strong base- lines. We further empirically demonstrate that the learned representations can be adapted to new tasks efficiently and effectively. 1 Introduction With the ability to learn rich distributed represen- tations of data in an end-to-end fashion, deep neu- ral networks have achieved the state of the arts in a variety of fields (He et al., 2017; Vaswani et al., 2017; Povey et al., 2018; Yu et al., 2018). For natural language understanding (NLU) tasks, ro- bust and flexible language representations can be adapted to new tasks or domains efficiently. Aim- ing at learning representations that are not exclu- sively tailored to any specific tasks or domains, re- searchers have proposed several ways to learn gen- eral language representations. Recently, there is a trend of learning universal language representations via language model pre- training (Dai and Le, 2015; Peters et al., 2018; Radford et al., 2018). In particular, Devlin et al. (2019) present the BERT model which is based on a bidirectional Transformer (Vaswani et al., 2017). BERT is pre-trained with both masked language model and next sentence prediction ob- Figure 1: Differences between multi-task learning and meta learning. Multi-task learning may favor high- resource tasks over low-resource ones while meta- learning aims at learning a good initialization that can be adapted to any task with minimal training samples. The figure is adapted from Gu et al. (2018). jectives and exhibits strong performance on sev- eral benchmarks, attracting huge attention from re- searchers. Another line of research tries to apply multi-task learning to representation learning (Liu et al., 2015; Luong et al., 2015). Multi-task learn- ing allows the model to leverage supervision sig- nals from related tasks and prevents the model from overfitting to a single task. By combining the strengths of both language model pre-training and multi-task learning, Liu et al. (2019) improve the BERT model with multi-task learning and their proposed MT-DNN model successfully achieves state-of-the-art results on several NLU tasks. Although multi-task learning can achieve promising performance, there still exist some po- tential problems. As shown in Figure 1, multi- task learning may favor tasks …

Content truncated to 500 words