PaCE: Unified Multi-modal Dialogue Pre-training with Progressive and Compositional Experts

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 13402–13416 July 9-14, 2023 ©2023 Association for Computational Linguistics PaCE: Unified Multi-modal Dialogue Pre-training with Progressive and Compositional Experts Yunshui Li1,2∗† Binyuan Hui3∗ Zhichao Yin1,4 Min Yang1‡ Fei Huang3 Yongbin Li3‡ 1Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences 2University of Chinese Academy of Sciences 3DAMO Academy, Alibaba Group 4University of Science and Technology of China {ys.li, min.yang}@siat.ac.cn, {binyuan.hby, shuide.lyb}@alibaba-inc.com http://github.com/AlibabaResearch/DAMO-ConvAI/pace Abstract Perceiving multi-modal information and fulfill- ing dialogues with humans is a long-term goal of artificial intelligence. Pre-training is com- monly regarded as an effective approach for multi-modal dialogue. However, due to the lim- ited availability of multi-modal dialogue data, there is still scarce research on multi-modal dialogue pre-training. Yet another intriguing challenge emerges from the encompassing na- ture of multi-modal dialogue, which involves various modalities and tasks. Moreover, new forms of tasks may arise at unpredictable points in the future. Hence, it is essential for designed multi-modal dialogue models to possess suf- ficient flexibility to adapt to such scenarios. This paper proposes PaCE, a unified, struc- tured, compositional multi-modal dialogue pre- training framework. It utilizes a combination of several fundamental experts to accommo- date multiple dialogue-related tasks and can be pre-trained using limited dialogue and ex- tensive non-dialogue multi-modal data. Fur- thermore, we propose a progressive training method where old experts from the past can assist new experts, facilitating the expansion of their capabilities. Experimental results demon- strate that PaCE achieves state-of-the-art results on eight multi-modal dialog benchmarks. 1 Introduction Enabling seamless communication between hu- mans and machines is a long-standing goal of artifi- cial intelligence research. The recent emergence of chatGPT 1 has increased confidence in the potential for achieving this goal. Beyond the use of textual language as a unique interface between humans and machines, perceiving and utilizing multi-modal information, especially visual information, has be- *Equal contribution. †Work done during an intern at Alibaba DAMO Academy. ‡Corresponding author. 1https://chat.openai.com/ Hello, I am from Los Angeles and interested in visiting a history museum to learn more about the peranakan culture. Do you have any recommendations? Hi, if you're traveling to Asia, the Palace Museum is worth seeing, you can get to see many ancient artifacts there. Wow, that sounds interesting to me. Could you tell me more about some of the stories related to the Palace Museum? The Palace Museum is located in the Forbidden City, which was the imperial palace of the Ming and Qing dynasties in China. It is called the "Forbidden City" because ordinary people were not allowed to enter without permission. Is the place in this picture worth going to? It looks mysterious, Could u Give me some introduction about it ? Sure ! The Temple of Heaven is also a culturally significant site, known for its historical use as a location for imperial sacrifices in ancient China. Excellent! Would you be able to assist me with booking the flight to Beijing on Tuesday morning …

Content truncated to 500 words