Neural Stylistic Response Generation with Disentangled Latent Variables

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 388–403 August 1–6, 2021. ©2021 Association for Computational Linguistics 388 Self-Supervised Multimodal Opinion Summarization Jinbae Im, Moonki Kim, Hoyeop Lee, Hyunsouk Cho, Sehee Chung Knowledge AI Lab., NCSOFT Co., South Korea {jinbae,kmkkrk,hoyeoplee,dakgalbi,seheechung}@ncsoft.com Abstract Recently, opinion summarization, which is the generation of a summary from multiple re- views, has been conducted in a self-supervised manner by considering a sampled review as a pseudo summary. However, non-text data such as image and metadata related to reviews have been considered less often. To use the abundant information contained in non-text data, we propose a self-supervised multimodal opinion summarization framework called Mul- timodalSum. Our framework obtains a repre- sentation of each modality using a separate en- coder for each modality, and the text decoder generates a summary. To resolve the inher- ent heterogeneity of multimodal data, we pro- pose a multimodal training pipeline. We first pretrain the text encoder–decoder based solely on text modality data. Subsequently, we pre- train the non-text modality encoders by con- sidering the pretrained text decoder as a pivot for the homogeneous representation of multi- modal data. Finally, to fuse multimodal rep- resentations, we train the entire framework in an end-to-end manner. We demonstrate the su- periority of MultimodalSum by conducting ex- periments on Yelp and Amazon datasets. 1 Introduction Opinion summarization is the task of automatically generating summaries from multiple documents containing users’ thoughts on businesses or prod- ucts. This summarization of users’ opinions can provide information that helps other users with their decision-making on consumption. Unlike con- ventional single-document or multiple-document summarization, where we can obtain the prevalent annotated summaries (Nallapati et al., 2016; See et al., 2017; Paulus et al., 2018; Liu et al., 2018; Liu and Lapata, 2019; Perez-Beltrachini et al., 2019), opinion summarization is challenging; it is difficult to find summarized opinions of users. Accordingly, (a) Unimodal framework (b) Multimodal framework Figure 1: Self-supervised opinion summarization frameworks studies used an unsupervised approach for opinion summarization (Ku et al., 2006; Paul et al., 2010; Carenini et al., 2013; Ganesan et al., 2010; Gerani et al., 2014). Recent studies (Braˇzinskas and Titov, 2020; Amplayo and Lapata, 2020; Elsahar et al., 2021) used a self-supervised learning framework that creates a synthetic pair of source reviews and a pseudo summary by sampling a review text from a training corpus and considering it as a pseudo summary, as in Figure 1a. Users’ opinions are based on their perception of a specific entity and perceptions originate from var- ious characteristics of the entity; therefore, opinion summarization can use such characteristics. For instance, Yelp provides users food or menu images and various metadata about restaurants, as in Fig- ure 1b. This non-text information influences the review text generation process of users (Truong and Lauw, 2019). Therefore, using this additional information can help in opinion summarization, especially under unsupervised settings (Su et al., 2019; Huang et al., 2020). Furthermore, the …

Content truncated to 500 words