On Knowledge distillation from complex networks for response prediction

Proceedings of NAACL-HLT 2019, pages 3813–3822 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3813 On Knowledge Distillation from Complex Networks for Response Prediction Siddhartha Arora1,2 and Mitesh M. Khapra1,2 and Harish G. Ramaswamy1,2 1Department of Computer Science and Engineering, Indian Institute of Technology Madras 2Robert Bosch Centre for Data Science and AI (RBC-DSAI), Indian Institute of Technology Madras {sidarora,miteshk,hariguru}@cse.iitm.ac.in Abstract Recent advances in Question Answering have lead to the development of very complex mod- els which compute rich representations for query and documents by capturing all pair- wise interactions between query and docu- ment words. This makes these models ex- pensive in space and time, and in practice one has to restrict the length of the docu- ments that can be fed to these models. Such models have also been recently employed for the task of predicting dialog responses from available background documents (e.g., Holl- E dataset). However, here the documents are longer, thereby rendering these complex mod- els infeasible except in select restricted set- tings. In order to overcome this, we use stan- dard simple models which do not capture all pairwise interactions, but learn to emulate cer- tain characteristics of a complex teacher net- work. Specifically, we first investigate the conicity of representations learned by a com- plex model and observe that it is significantly lower than that of simpler models. Based on this insight, we modify the simple archi- tecture to mimic this characteristic. We go further by using knowledge distillation ap- proaches, where the simple model acts as a student and learns to match the output from the complex teacher network. We experiment with the Holl-E dialog data set and show that by mimicking characteristics and matching out- puts from a teacher, even a simple network can give improved performance. 1 Introduction The advent of large scale datasets for QA has lead to the development of increasing complex neural models with specialized components for (i) en- coding the query (ii) encoding the document(s) (iii) capturing interactions between document and query words and (iv) generating/extracting the correct answer span from the given document (Seo et al., 2016; Hu et al., 2017; Yu et al., 2018). While these models give state-of-the-art performance on a variety of datasets, they have very high space and time complexity. This is a concern, and in prac- tice, it is often the case that one has to resort to restricting the maximum length of the input docu- ment such that the model can run with reasonable resources (say, a single 12GB Tesla K80 GPU). Such complex span prediction models are also being adapted for other NLP tasks such as dialog response prediction (Moghe et al., 2018), which is the focus of this work. In particular, we refer to the Holl-E dataset where the task is to extract the next response from a document which is relevant to the conversation (see Figure 1). This setup is very similar to QA wherein the input is {context, document} as opposed …

Content truncated to 500 words