PhotoChat: A Human-Human Dialogue Dataset With Photo Sharing Behavior For Joint Image-Text Modeling

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6142–6152 August 1–6, 2021. ©2021 Association for Computational Linguistics 6142 PhotoChat: A Human-Human Dialogue Dataset with Photo Sharing Behavior for Joint Image-Text Modeling Xiaoxue Zang1, Lijuan Liu1, Maria Wang1, Yang Song2∗, Hao Zhang1, Jindong Chen1 1 Google Research, 2 Kuaishou Technology 1{xiaoxuez, lijuanliu, mariawang, haozhangthu, jdchen}@google.com, 2 yangsong@kuaishou.com Abstract We present a new human-human dialogue dataset - PhotoChat, the first dataset that casts light on the photo sharing behavior in online messaging. PhotoChat contains 12k dialogues, each of which is paired with a user photo that is shared during the conversation. Based on this dataset, we propose two tasks to fa- cilitate research on image-text modeling: a photo-sharing intent prediction task that pre- dicts whether one intends to share a photo in the next conversation turn, and a photo retrieval task that retrieves the most relevant photo according to the dialogue context. In addition, for both tasks, we provide baseline models using the state-of-the-art models and report their benchmark performances. The best image retrieval model achieves 10.4% re- call@1 (out of 1000 candidates) and the best photo intent prediction model achieves 58.1% F1 score, indicating that the dataset presents in- teresting yet challenging real-world problems. We are releasing PhotoChat to facilitate future research work among the community. 1 Introduction As instant messaging tools gain enormous pop- ularity in the recent decades, sharing photos as an approach to enhance the engagement of an on- line messaging conversation has become a perva- sive routine communicative act (Lobinger, 2016). A survey conducted in 2010 reveals that 74% of teenagers in the US reported messaging a photo or video using their cell phone (Lenhart et al., 2010). In Britain, almost 70% of the internet users shared photos in 2013 (Dutton and Blank, 2013). Consid- ering the proliferation of photo sharing, it’s desir- able to have an intelligent system that can assist users efficiently engaging in this process, i.e. sug- gesting the most relevant photos in correct timings. In order to achieve this goal, the intelligent system is expected to not only understand how humans ∗Research conducted while working at Google. Figure 1: An example of how people share photos in a daily conversation. communicate with each other, e.g. the natural lan- guage human speak, but also perceive images as human do. How to facilitate building such multi- modal system is the goal of this paper. Though recently many image-text tasks have been proposed and are being actively studied to bridge language and vision, the majority of them are formulated as choosing or composing the text based on the understanding of given images, e.g. image captioning (Anderson et al., 2018), visual question answering (Antol et al., 2015), visual commonsense reasoning (Zellers et al., 2019), and image-grounded dialogue generation (Shuster et al., 2020). Contrary to these tasks, the photo sharing task focuses on the reverse process, i.e. selecting the image based on the …

Content truncated to 500 words