Disentangling Categorization in Multi-agent Emergent Communication

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 458 - 482 July 10-15, 2022 ©2022 Association for Computational Linguistics Mapping the Design Space of Human-AI Interaction in Text Summarization Ruijia Cheng ⇤ University of Washington rcheng6@uw.edu Alison Smith-Renner Dataminr Inc. arenner@dataminr.com Ke Zhang Dataminr Inc. kzhang@dataminr.com Joel R. Tetreault Dataminr Inc. jtetreault@dataminr.com Alejandro Jaimes Dataminr Inc. ajaimes@dataminr.com Abstract Automatic text summarization systems com- monly involve humans for preparing data or evaluating model performance, yet, there lacks a systematic understanding of humans’ roles, experience, and needs when interacting with or being assisted by AI. From a human-centered perspective, we map the design opportunities and considerations for human-AI interaction in text summarization and broader text generation tasks. We first conducted a systematic literature review of 70 papers, developing a taxonomy of five interactions in AI-assisted text generation and relevant design dimensions. We designed text summarization prototypes for each interac- tion. We then interviewed 16 users, aided by the prototypes, to understand their expectations, experience, and needs regarding efficiency, con- trol, and trust with AI in text summarization and propose design considerations accordingly. 1 Introduction In this era of rapid information consumption, ac- cess to high-quality summaries, such as online news highlights and research paper abstracts, is increasingly important. However, summarization is difficult for humans, demanding high cognitive load and expertise (Hidi and Anderson, 1986). Al- gorithmic approaches can automate summarization but typically underperform humans and require many high-quality human-written summaries for training (Durrett et al., 2016; Huang et al., 2021). AI systems that involve humans typically constrain their input to data preparation (Lloret et al., 2013) or final evaluation (Khashabi et al., 2021) as the first or last step in the summarization workflow. How can humans work together with AI to produce better summaries? Other text generation tasks, such as machine translation and creative writing, offer inspiration. ⇤ Research work was done while the author was interning at Dataminr Inc. Figure 1: Five human-AI interactions in text generation from Study 1, illustrated as summarization tasks. Expla- nation of the actions and visual elements are in §2.2. Beyond being data providers and evaluators, hu- mans can generate text with the assistance of AI. For example, humans can revise AI translation (i.e., “post-editing”) (Green et al., 2013) or write with AI suggestions (Clark et al., 2018). These interaction techniques may apply to summarization, yet our un- derstanding of the possible human-AI interactions and how to design for them is incomplete. To that end, we conducted two studies to explore: RQ1 What are the different ways that humans and AI can interact in text generation? RQ2 What are humans’ experience and needs with these different types of AI assistance in the context of summariza- tion? For these studies, we took a human-centered approach—focusing on what humans want from AI and how to improve their experience, rather than 1 458 improving AI models. In Study 1, we conducted a systematic literature review of 70 …

Content truncated to 500 words