MATINF: A Jointly Labeled Large-Scale Dataset for Classification, Question Answering and Summarization

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3586–3596 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3586 MATINF: A Jointly Labeled Large-Scale Dataset for Classification, Question Answering and Summarization Canwen Xu1∗, Jiaxin Pei2∗, Hongtao Wu3, Yiyu Liu3, Chenliang Li3† 1 School of Computer Science, Wuhan University, China 2 School of Information, University of Michigan, United States 3 School of Cyber Science and Engineering, Wuhan University, China 1,3 {xucanwen,wuhongtao,liuyiyu,cllee}@whu.edu.cn 2 pedropei@umich.edu Abstract Recently, large-scale datasets have vastly facil- itated the development in nearly all domains of Natural Language Processing. However, there is currently no cross-task dataset in NLP, which hinders the development of multi-task learning. We propose MATINF, the first jointly labeled large-scale dataset for classification, question answering and summarization. MAT- INF contains 1.07 million question-answer pairs with human-labeled categories and user- generated question descriptions. Based on such rich information, MATINF is applicable for three major NLP tasks, including classifica- tion, question answering, and summarization. We benchmark existing methods and a novel multi-task baseline over MATINF to inspire further research. Our comprehensive compar- ison and experiments over MATINF and other datasets demonstrate the merits held by MAT- INF. 1 1 Introduction In recent years, large-scale datasets (e.g., Ima- geNet (Deng et al., 2009) and SQuAD (Rajpurkar et al., 2016)) have inspired remarkable progress in many areas like Computer Vision (CV) and Natu- ral Language Processing (NLP). On the one hand, well-annotated data provide essential information for training supervised machine learning models. On the other hand, benchmarked datasets make it possible to evaluate and compare the capability of different methods on the same stage. Due to the high cost of data annotation, existing NLP datasets are usually labeled for only one par- ticular task (e.g., SQuAD (Rajpurkar et al., 2016) for question answering, CNN/DM (Hermann et al., ∗ The first two authors contribute equally to this paper. † Chenliang Li is the corresponding author. 1The implementation of MTF-S2S and information about obtaining access to the dataset can be found at https:// github.com/WHUIR/MATINF. 2015) for summarization and AGNews (Zhang et al., 2015) for text classification). These single- task datasets hinder the development of learning common and task-invariant knowledge (Liu et al., 2017). Although multi-task learning and transfer learning have delivered encouraging results, we still cannot determine whether the improvement is from the extension of input or supervision. Further- more, task-specific data make the models tend to learn task-specific leakage features (Zhang et al., 2019) rather than meaningful knowledge that could generalize to other tasks. However, as a key step to Artificial General Intelligence (AGI), knowledge acquisition requires the model to learn more gen- eral knowledge instead of overfitting on a specific task. Therefore, a large-scale and cross-task dataset is in huge demand for future NLP research. Never- theless, to the best of our knowledge, none of the existing datasets could meet such demand. In this paper, we propose Maternal and Infant Dataset (MATINF), the first large-scale dataset cov- ering three major NLP tasks: text classification, …

Content truncated to 500 words