AUTOSUMM: Automatic Model Creation for Text Summarization

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10162–10172 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 10162 AUTOSUMM: Automatic Model Creation for Text Summarization Sharmila Reddy Nangi1∗, Atharv Tyagi2∗, Jay Mundra2∗, Sagnik Mukherjee2∗ Snehal Raj2∗, Aparna Garimella3, Niyati Chhaya3 1Stanford University, USA 3Adobe Research, India 2Indian Institute of Technology Kanpur, India srnangi@stanford.edu1 {garimell,nchhaya}@adobe.com3 {atharv,jaym,sagnikm,snehal}@iitk.ac.in2 Abstract Recent efforts to develop deep learning mod- els for text generation tasks such as extrac- tive and abstractive summarization have re- sulted in state-of-the-art performances on var- ious datasets. However, obtaining the best model configuration for a given dataset re- quires an extensive knowledge of deep learn- ing specifics like model architecture, tuning parameters etc., and is often extremely chal- lenging for a non-expert. In this paper, we propose methods to automatically create deep learning models for the tasks of extractive and abstractive text summarization. Based on the recent advances in Automated Machine Learn- ing and the success of large language models such as BERT and GPT-2 in encoding knowl- edge, we use a combination of Neural Ar- chitecture Search (NAS) and Knowledge Dis- tillation (KD) techniques to perform model search and compression using the vast knowl- edge provided by these language models to develop smaller, customized models for any given dataset. We present extensive empir- ical results to illustrate the effectiveness of our model creation methods in terms of infer- ence time and model size, while achieving near state-of-the-art performances in terms of accu- racy across a range of datasets. 1 Introduction Machine learning algorithms, particularly, deep learning techniques have led to the simplification of several computationally expensive tasks. How- ever, training and optimizing these models for dif- ferent tasks demand the experienced engineering resources and require expertise, making it difficult for non-experts. Automated Machine Learning is a strategy to automate this pipeline for model cre- ation including automated generation of the model itself. ∗Work done while authors were at Adobe Research. In the case of Natural Language Processing and Text analysis, the advent of large language models such as BERT(Devlin et al., 2019), GPT2(Radford et al., 2019), and more recently GPT3(Brown et al., 2020) have created resources that can be exploited for the creation of robust models for several down- stream NLP tasks. However, the need for ML ex- pertise creates a bottleneck. Further, these deep learning models have thousands of parameters and need fairly large datasets and computational re- sources for training. We focus on providing algorithms for auto- generation of ML models for complex NLP tasks such as extraction and generation, making them ac- cessible to non experts. Our proposed approaches feed off the knowledge available in large pretrained models to auto-generate new, smaller, customized models for a custom dataset. Specifically, the ma- jor contributions are as follows. (1) We propose a method to create machine learn- ing models that are efficient and customized to a given dataset for the tasks of extractive and abstrac- tive summarization, using a combination of neural architecture search and …

Content truncated to 500 words