Feeding What You Need by Understanding What You Learned

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5858 - 5874 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Feeding What You Need by Understanding What You Learned Xiaoqiang Wang1∗, Bang Liu2∗†, Fangli Xu3, Bo Long4, Siliang Tang1‡ and Lingfei Wu5‡ 1Zhejiang University, 2Université de Montréal & Mila, 3Squirrel AI Learning 4JD.COM, 5JD.COM Silicon Valley Research Center {xq.wang, siliang}@zju.edu.cn bang.liu@umontreal.ca, lili@yixue.us {bo.long, lingfei.wu}@jd.com Abstract Machine Reading Comprehension (MRC) re- veals the ability to understand a given text pas- sage and answer questions based on it. Ex- isting research works in MRC rely heavily on large-size models and corpus to improve the performance evaluated by metrics such as Exact Match (EM) and F1. However, such a paradigm lacks sufficient interpretation to model capability and can not efficiently train a model with a large corpus. In this paper, we argue that a deep understanding of model ca- pabilities and data properties can help us feed a model with appropriate training data based on its learning status. Specifically, we design an MRC capability assessment framework that assesses model capabilities in an explainable and multi-dimensional manner. Based on it, we further uncover and disentangle the con- nections between various data properties and model performance. Finally, to verify the effectiveness of the proposed MRC capabil- ity assessment framework, we incorporate it into a curriculum learning pipeline and de- vise a Capability Boundary Breakthrough Cur- riculum (CBBC) strategy, which performs a model capability-based training to maximize the data value and improve training efficiency. Extensive experiments demonstrate that our approach significantly improves performance, achieving up to an 11.22% / 8.71% improve- ment of EM / F1 on MRC tasks. 1 Introduction A competency assessment is used to measure some- one’s capabilities against the requirements of their job (Cheryl Lasse, 2020). In other words, it mea- sures how (behaviors) someone does the what (task or skill). By showing what it looks like to be good ∗Equal contribution. †Canada CIFAR AI Chair. ‡Corresponding authors. in a job, a competency assessment can effectively empower and engage people who want to under- stand and improve their unique skill profile and tell them what action to take to close any gaps so they can own their development. A natural ques- tion that arises here is: can we develop competency assessments for machine learning models to help better understand their capabilities and improve their performance on a given task? In this paper, we focus on competency assess- ments for machine reading comprehension (MRC). MRC is a core task in natural language processing (NLP) that aims to teach machines to understand human languages and answer questions (Zeng et al., 2020; Chen et al., 2019). Recently, pre- trained language models (LMs) (Mikolov et al., 2013; Peters et al., 2018; Pennington et al., 2014; Devlin et al., 2018) have demonstrated superior per- formance on MRC tasks by pre-training on large amounts of unlabeled corpus and fine-tuning on MRC datasets. The performance is usually eval- uated by metrics …

Content truncated to 500 words