“I’d rather just go to bed”: Understanding Indirect Answers

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7411–7425, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7411 “I’d rather just go to bed”: Understanding Indirect Answers Annie Louis Google Research, UK annielouis@google.com Dan Roth∗ University of Pennsylvania danroth@seas.upenn.edu Filip Radlinski Google Research, UK filiprad@google.com Abstract We revisit a pragmatic inference problem in dialog: Understanding indirect responses to questions. Humans can interpret ‘I’m starving.’ in response to ‘Hungry?’, even without direct cue words such as ‘yes’ and ‘no’. In dialog sys- tems, allowing natural responses rather than closed vocabularies would be similarly bene- ficial. However, today’s systems are only as sensitive to these pragmatic moves as their lan- guage model allows. We create and release1 the first large-scale English language corpus ‘Circa’ with 34,268 (polar question, indirect answer) pairs to enable progress on this task. The data was collected via elaborate crowd- sourcing, and contains utterances with yes/no meaning, as well as uncertain, middle-ground, and conditional responses. We also present BERT-based neural models to predict such cat- egories for a question-answer pair. We find that while transfer learning from entailment works reasonably, performance is not yet suf- ficient for robust dialog. Our models reach 82- 88% accuracy for a 4-class distinction, and 74- 85% for 6 classes. 1 Introduction Humans produce and interpret complex utterances even in simple scenarios. For example, for the polar (yes/no) question ‘Want to get dinner?’, there are many perfectly natural responses in addition to ‘yes’ and ‘no’, as in Table 1. How should a dialog system interpret these INDIRECT answers? Many can be understood based on the answer text alone, e.g. ‘I would like that’ or ‘I’d rather just go to bed’. For others, the question is crucial, e.g. ‘Dinner would be lovely.’ is a positive reply here, but a negative answer to ‘Want to get lunch?’. In this ∗* Work done at Google 1The corpus can be downloaded from http://goo. gle/circa “Want to get some dinner together?” “I know a restaurant we could get a reservation at.” “I have already eaten recently.” “I hope to make it home by supper but I’m not sure I can.” “Dinner would be lovely.” “I’d rather just go to bed.” “There’s a few new restaurants we could go to.” “I would like that.” “We could do dinner this weekend.” “I would like to go somewhere casual.” “I’d like to try the new Italian place.” Table 1: A polar question with 10 indirect responses, taken from our corpus. paper, we present the first large scale corpus and models for interpreting such indirect answers. Previous attempts to interpret indirect yes/no answers have been small scale and without data- driven techniques (Green and Carberry, 1999; de Marneffe et al., 2010). However, recent suc- cess on many language understanding problems (Wang et al., 2019), the impressive generation ca- pabilities of modern dialog systems (Zhang et al., 2019; Adiwardana et al., 2020), as well as the huge interest in yes/no question-answering (Choi et al., 2018; Clark et al., …

Content truncated to 500 words