Navigating Connected Memories with a Task-oriented Dialog System

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2495 - 2507 December 7-11, 2022 ©2022 Association for Computational Linguistics Navigating Connected Memories with a Task-oriented Dialog System Seungwhan Moon∗, Satwik Kottur∗, Alborz Geramifard, Babak Damavandi Meta Reality Labs & Meta AI � {shanemoon,skottur,alborzg,babakd}@fb.com Abstract Recent years have seen an increasing trend in the volume of personal media captured by users, thanks to the advent of smartphones and smart glasses, resulting in large media collections. Despite conversation being an intuitive human- computer interface, current efforts focus mostly on single-shot natural language based media re- trieval to aid users query their media and re-live their memories. This severely limits the search functionality as users can neither ask follow- up queries nor obtain information without first formulating a single-turn query. In this work, we propose dialogs for connected memories as a powerful tool to empower users to search their media collection through a multi- turn, interactive conversation. Towards this, we collect a new task-oriented dialog dataset COMET, which contains 11.5k user↔assistant dialogs (totalling 103k utterances), grounded in simulated personal memory graphs. We em- ploy a resource-efficient, two-phase data collec- tion pipeline that uses: (1) a novel multimodal dialog simulator that generates synthetic dia- log flows grounded in memory graphs, and, (2) manual paraphrasing to obtain natural language utterances. We analyze COMET, formulate four main tasks to benchmark meaningful progress, and adopt state-of-the-art language models as strong baselines, in order to highlight the mul- timodal challenges captured by our dataset1. 1 Introduction The rise of smartphones and smart glasses has con- tributed to a surge in the amount of personal media (photos, videos, montages, etc.) captured by users on a day-to-day basis in the past decade. For in- stance, it is estimated that about 1.5 trillion photos would be clicked in the year 2022 (Pantic, 2021). As a result, personal media collections typically grow at an alarming rate, making it cumbersome ∗Joint first authors 1Our code & data is made available at github.com/ facebookresearch/comet_memory_dialog Figure 1: Illustration of COMET: COnnected MEmories with a Task-oriented Dialog. (a) Each dialog turn is fully annotated with dialog acts and multimodal coreference labels, accompanied with photos associated with the re- quest. (b) These media are from the underlying memory graph, a structured collection of personal media. for users to manually search, retrieve, and re-live their captured memories2. To alleviate this situation, solutions that perform natural language query-based media retrieval (Tan et al., 2019; Vo et al., 2019; Tellex and Roy, 2009; Barbu et al., 2013; Li et al., 2017; Guo et al., 2018a; Saha et al., 2018) have been proposed. However, such approaches exhibit two drawbacks. First, they are single-shot interactions without any context carry-over, e.g., Show me some photos from the beach last week.. This limits the functionality and 2Memories and media files are used interchangeably. 2495 does not let users ask any follow-up queries like ‘Display photos from the first time I was here?’, since understanding here requires the query history. …

Content truncated to 500 words