Disentangling Indirect Answers to Yes-No Questions in Real Conversations

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4677 - 4695 July 10-15, 2022 ©2022 Association for Computational Linguistics Disentangling Indirect Answers to Yes-No Questions in Real Conversations Krishna C. Sanagavarapu1, Jathin P. Singaraju2, Anusha Kakileti2, Anirudh Kaza2, Aaron A. Mathews2, Helen Li2, Nathan R. Brito2, Eduardo Blanco1 1Arizona State University, 2University of North Texas, {ksanagav, eduardo.blanco}@asu.edu, {pran.singaraju, arkaza820}@gmail.com, {akakileti, AaronMathews, helenli, Nathanbrito}@unt.edu Abstract In this paper, we explore the task of deter- mining indirect answers to yes-no questions in real conversations. We work with tran- scripts of phone conversations in the Switch- board Dialog Act (SwDA) corpus and create SwDA-IndirectAnswers (SwDA-IA), a subset of SwDA consisting of all conversations con- taining a yes-no question with an indirect an- swer. We annotate the underlying direct an- swers to the yes-no questions (yes, probably yes, middle, probably no, or no). We show that doing so requires taking into account con- versation context: the indirect answer alone is insufficient to determine the ground truth. Experimental results also show that taking into account context is beneficial. More importantly, our results demonstrate that existing corpora with synthetic indirect answers to yes-no ques- tions are not beneficial when working with real conversations. Our best models outperform the majority baseline by a substantial margin, but the task remains a challenge (F1: 0.46). 1 Introduction Dialogue systems have become a reality enabled by large datasets and deep neural networks. Task- oriented (Wen et al., 2017) and open-domain (Tang et al., 2019) dialogue systems are commonplace. Neural approaches are popular (Zhang et al., 2018) although systems based on logical inference and rules outperform networks in open-domain dia- logue (Finch et al., 2021). State-of-the-art systems face challenges keeping track of a conversation and avoiding inconsistencies (Welleck et al., 2019). Evaluation is also an open issue as automated met- rics have several drawbacks (Liu et al., 2016). An alternative is to evaluate dialogue systems based on whether they can collaborate with humans to solve a problem (Lewis et al., 2017) or elicit some action such as donating to a cause (Wang et al., 2019). People do not explicitly say what they mean when they speak to each other yet they seamlessly A: Do you work outside of the home? B: No, I am not working currently. Underlying direct answer: No A: Do you work outside of the home? B: Uh, last month I left my company. Underlying direct answer: No A: Do you work outside of the home? B: Uh, last month I left my company. A: What happened? Stress? B: But now I work for a marketing firm where I travel a lot. Underlying direct answer: Yes Figure 1: Conversation snippets with a yes-no ques- tion (first turn by Sparker A). In the first example, the underlying direct answer is obvious given the answer by Speaker B. In the second example, determining the underlying direct answer requires commonsense knowl- edge. In the third example, it requires not …

Content truncated to 500 words