Diverse Few-Shot Text Classification with Multiple Metrics

Proceedings of NAACL-HLT 2018, pages 1206–1215 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Diverse Few-Shot Text Classification with Multiple Metrics Mo Yu⇤ Xiaoxiao Guo⇤ Jinfeng Yi⇤ Shiyu Chang Saloni Potdar Yu Cheng Gerald Tesauro Haoyu Wang Bowen Zhou AI Foundations – Learning, IBM Research IBM T. J. Watson Research Center, Yorktown Heights, NY 10598 Abstract We study few-shot learning in natural lan- guage domains. Compared to many ex- isting works that apply either metric-based or optimization-based meta-learning to image domain with low inter-task variance, we con- sider a more realistic setting, where tasks are diverse. However, it imposes tremendous diffi- culties to existing state-of-the-art metric-based algorithms since a single metric is insufficient to capture complex task variations in natu- ral language domain. To alleviate the prob- lem, we propose an adaptive metric learn- ing approach that automatically determines the best weighted combination from a set of met- rics obtained from meta-training tasks for a newly seen few-shot task. Extensive quantita- tive evaluations on real-world sentiment anal- ysis and dialog intent classification datasets demonstrate that the proposed method per- forms favorably against state-of-the-art few shot learning algorithms in terms of predictive accuracy. We make our code and data avail- able for further study.1 1 Introduction Few-shot learning (FSL) (Miller et al., 2000; Li et al., 2006; Lake et al., 2015) aims to learn classifiers from few examples per class. Re- cently, deep learning has been successfully ex- ploited for FSL via learning meta-models from a large number of meta-training tasks. These meta-models can be then used for rapid-adaptation for the target/meta-testing tasks that only have few training examples. Examples of such meta- models include: (1) metric-/similarity-based mod- els, which learn contextual, and task-specific sim- ilarity measures (Koch, 2015; Vinyals et al., 2016; ⇤Equal contributions from the corresponding authors: yum@us.ibm.com, xiaoxiao.guo@ibm.com, jinfengy@us.ibm.com. 1https://github.com/Gorov/DiverseFewShot_Amazon Snell et al., 2017); and (2) optimization-based models, which receive the input of gradients from a FSL task and predict either model parameters or parameter updates (Ravi and Larochelle, 2017; Munkhdalai and Yu, 2017; Finn et al., 2017; Wang et al., 2017). In the past, FSL has mainly considered im- age domains, where all tasks are often sampled from one huge collection of data, such as Om- niglot (Lake et al., 2011) and ImageNet (Vinyals et al., 2016), making tasks come from a single do- main thus related. Due to such a simplified set- ting, almost all previous works employ a com- mon meta-model (metric-/optimization-based) for all few-shot tasks. However, this setting is far from the realistic scenarios in many real-world ap- plications of few-shot text classification. For ex- ample, on an enterprise AI cloud service, many clients submit various tasks to train text classifica- tion models for business-specific purposes. The tasks could be classifying customers’ comments or opinions on different products/services, moni- toring public reactions to different policy changes, or determining users’ intents in different types of personal assistant services. As most of the clients cannot collect enough data, their submitted tasks …

Content truncated to 500 words