Analysis of Dialogue in Human-Human Collaboration in Minecraft

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4051–4059 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4051 Analysis of Dialogue in Human-Human Collaboration in Minecraft Takuma Ichikawa, Ryuichiro Higashinaka Graduate School of Informatics, Nagoya University Furo-cho, Chikusa-ku, Nagoya, 464-8601, Japan {ichikawa.takuma.w0@s.mail,higashinaka@i}.nagoya-u.ac.jp Abstract Recently, many studies have focused on developing dialogue systems that enable collaborative work; however, they rarely focus on creative tasks. Collaboration for creative work, in which humans and systems collaborate to create new value, will be essential for future dialogue systems. In this study, we collected 500 dialogues of human-human collaboration in Minecraft as a basis for developing a dialogue system that enables creative collaborative work. We conceived the Collaborative Garden Task, where two workers interact and collaborate in Minecraft to create a garden, and we collected dialogue, action logs, and subjective evaluations. We also collected third-person evaluations of the gardens and analyzed the relationship between dialogue and collaborative work that received high scores on the subjective and third-person evaluations in order to identify dialogic factors for high-quality collaborative work. We found that two essential aspects in creative collaborative work are performing more processes to ask for and agree on suggestions between workers and agreeing on a particular image of the final product in the early phase of work and then discussing changes and details. Keywords: collaboration, creative task, dialogue system, minecraft, data collection 1. Introduction Recently, as dialogue systems become part of our daily lives, towards more advanced dialogue systems, re- searchers have been working on dialogue systems that can collaborate with humans to complete tasks (Meena et al., 2014; He et al., 2017; Kim et al., 2019; Zhang et al., 2021). Many studies related to collaborative work with di- alogue systems use Minecraft1 (Szlam et al., 2019). Minecraft is a sandbox-type 3D game in which the player controls an avatar and can move freely in the world. It also supports multiplayer, making it easy for multiple players to collaborate in the same environ- ment. Most studies using Minecraft have focused on problem- solving collaborative work (Narayan-Chen et al., 2019; Ogawa et al., 2020; Bara et al., 2021), and few studies have focused on creative collaborative work. To realize a society in which humans and systems can create new value, it is essential that systems and users can work together on creative collaborative work. This study collected data on creative collaborative work between humans in Minecraft as a basis for devel- oping a dialogue system that enables creative collab- orative work. We devised the Collaborative Garden Task and collected text chats, action logs, and ques- tionnaire results. We also collected third-person evalu- ations of the created gardens scored in terms of origi- nality and beauty in order to objectively assess the qual- ity of the collaboration. Furthermore, we analyzed the relationship between dialogue and collaborative work that received high scores on the subjective and third- person evaluations in order to identify dialogic factors for high-quality collaborative …

Content truncated to 500 words