Interactive Classification by Asking Informative Questions

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2664–2680 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2664 Interactive Classification by Asking Informative Questions Lili Yu1, Howard Chen1, Sida I. Wang1,2,Tao Lei1 and Yoav Artzi1,3 1ASAPP Inc., New York, USA 2Princeton University, New Jersey, USA 3Cornell University, New York, USA {liliyu, hchen, tao}@asapp.com sidaw@cs.princeton.edu yoav@cs.cornell.edu Abstract We study the potential for interaction in nat- ural language classification. We add a lim- ited form of interaction for intent classifica- tion, where users provide an initial query us- ing natural language, and the system asks for additional information using binary or multi- choice questions. At each turn, our system decides between asking the most informative question or making the final classification pre- diction.The simplicity of the model allows for bootstrapping of the system without interac- tion data, instead relying on simple crowd- sourcing tasks. We evaluate our approach on two domains, showing the benefit of interac- tion and the advantage of learning to balance between asking additional questions and mak- ing the final prediction. 1 Introduction Responding to natural language queries through simple, single-step classification has been studied extensively in many applications, including user in- tent prediction (Chen et al., 2019; Qu et al., 2019), and information retrieval (Kang and Kim, 2003; Rose and Levinson, 2004). Typical methods rely on a single user input to produce an output, and do not interact with the user to reduce ambiguity and improve the final prediction. For example, users may under-specify a request due to incomplete un- derstanding of the domain; or the system may fail to correctly interpret the nuances of the input query. In both cases, a low quality decision could be miti- gated by further interaction with the user. In this paper we take a low-overhead approach to add limited interaction to intent classification. Our goal is two-fold: (a) study the effect of interaction on the system performance, and (b) avoid the cost and complexities of interactive data collection. We build an interactive system that poses a sequence of binary and multiple choice questions follow- What is the bill length of the bird: shorter, similar, or longer than head? Shorter than head. Is the bird underpart orange? Yes. The identified bird is: American Redstart FAQ Suggestion What data limits apply when roaming internationally? American Crow Bobolink … American Redstart How do I sign up for Sprint Global Roaming? . . . How do I purchase a High Speed Data Roaming Pass? Bird Identification Travel out of country. Do you need to activate global roaming service? Yes. Do you want high speed data roaming? No. Got it! The article below might be helpful: How do I sign up for Sprint Global Roaming? Saw a little black bird with black eyes. Figure 1: Two examples of interactive classification systems: providing a trouble-shooting FAQ suggestion (left) and helping identifying bird species from a de- scriptive text query (right). The top parts show ex- ample classification labels: FAQ documents …

Content truncated to 500 words