Cross-Modal Commentator: Automatic Machine Commenting Based on Cross-Modal Information

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2680–2686 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 2680 Cross-Modal Commentator: Automatic Machine Commenting Based on Cross-Modal Information Pengcheng Yang1,2∗, Zhihan Zhang2∗, Fuli Luo2, Lei Li2, Chengyang Huang3, Xu Sun1,2 1Deep Learning Lab, Beijing Institute of Big Data Research, Peking University 2MOE Key Lab of Computational Linguistics, School of EECS, Peking University 3Beijing University of Posts and Telecommunications {yang pc, zhangzhihan, luofuli, xusun}@pku.edu.cn tobiaslee@foxmail.com, cyhuang@bupt.edu.cn Abstract Automatic commenting of online articles can provide additional opinions and facts to the reader, which improves user experience and engagement on social media platforms. Pre- vious work focuses on automatic commenting based solely on textual content. However, in real-scenarios, online articles usually contain multiple modal contents. For instance, graphic news contains plenty of images in addition to text. Contents other than text are also vital be- cause they are not only more attractive to the reader but also may provide critical informa- tion. To remedy this, we propose a new task: cross-model automatic commenting (CMAC), which aims to make comments by integrating multiple modal contents. We construct a large- scale dataset for this task and explore several representative methods. Going a step further, an effective co-attention model is presented to capture the dependency between textual and visual information. Evaluation results show that our proposed model can achieve better performance than competitive baselines. 1 1 Introduction Comments of online articles can provide rich sup- plementary information, which reduces the diffi- culty of understanding the article and enhances in- teractions between users. Therefore, achieving au- tomatic commenting is necessary since it can con- tribute to improving user experience and increas- ing the activeness of social media platforms. Due to the importance described above, some work (Qin et al., 2018; Lin et al., 2018; Ma et al., 2018) has explored this task. However, these ef- forts are all focus on automatic commenting based solely on textual content. In real-scenarios, online ∗Equal Contribution. 1The dataset and code are available at https:// github.com/lancopku/CMAC News Images News Title ���� ��������� (Spring is coming! Thousands of acres are filled with intoxicating peach blossoms in Shanxi.) News Body ���������������������������� ����� (Recently, thousands of acres of peach blossoms are in full bloom at Pinglu, Shanxi Province. Visitors are immersed in the beautiful flowers, enjoying the breath of spring.) Comments 1. ��������� (Beautiful flowers! I can’t move my eyes from them.) 2. �����������������(Peach blossoms seem to be a little less pretty without any green grass as background.) 3. �������� (It would be better if there is more greenness.) Figure 1: An example in the constructed dataset. Red words indicate the content that is not included in the text but depicted in the images. articles on social media usually contain multiple modal contents. Take graphic news as an exam- ple, it contains plenty of images in addition to text. Other contents except text are also vital to improv- ing automatic commenting. These contents may contain some …

Content truncated to 500 words