Guiding the Flowing of Semantics: Interpretable Video Captioning via POS Tag

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2068–2077, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2068 Guiding the Flowing of Semantics: Interpretable Video Captioning via POS Tag Xinyu Xiao1,2, Lingfeng Wang1, Bin Fan1, Shiming Xiang1,2, Chunhong Pan1 1NLPR, Institute of Automation, Chinese Academy of Sciences 2School of Artificial Intelligence, University of Chinese Academy of Sciences {xinyu.xiao, lfwang, bfan, smxiang, chpan}@nlpr.ia.ac.cn Abstract In the current video captioning models, the video frames are collected in one network and the semantics are mixed into one feature, which not only increase the difficulty of the caption decoding, but also decrease the inter- pretability of the captioning models. To ad- dress these problems, we propose an Adaptive Semantic Guidance Network (ASGN), which instantiates the whole video semantics to dif- ferent POS-aware semantics with the supervi- sion of part of speech (POS) tag. In the encod- ing process, the POS tag activates the related neurons and parses the whole semantic infor- mation into corresponding encoded video rep- resentations. Furthermore, the potential of the model is stimulated by the POS-aware video features. In the decoding process, the related video features of noun and verb are used as the supervision to construct a new adaptive at- tention model which can decide whether to at- tend to the video feature or not. With the ex- plicit improving of the interpretability of the network, the learning process is more transpar- ent and the results are more predictable. Ex- tensive experiments demonstrate the effective- ness of our model when compared with state- of-the-art models. 1 Introduction Video captioning, which transforms the semantic information in a video to a natural statement, has received wide attention recently. The series of scenes (both related and unrelated) in video frames bring a huge challenge for the task of video cap- tioning. Therefore, mastering the ability to pro- cess the correlated and irrelevant semantic infor- mation can improve the performance and inter- pretability of the model of video captioning. The classical deep learning based video cap- tioning methods (Venugopalan et al., 2015a,b) in- corporate both CNN and LSTM together as an a man is lifting weights DET NOUN AUX VERB NOUN a girl is talking on the phone to another girl DET NOUN AUX VERB ADP DET NOUN ADP DET NOUN (a) (b) Figure 1: Examples of video description and its POS tags. The definition of POS tag can be seen in Section 3.1. From these examples, it can be seen that the words that are belonging to the same POS tags undergo inflec- tion for similar semantic properties and different POS tags reflect different semantic properties distinctly. encoder-decoder architecture which extracts all the semantic information in one stream and fuses them in a single feature. In the architecture, the transformation of the semantics to words predic- tion is uncertainty. Some other recent studies (Dong et al., 2017; Wu et al., 2018) attempt to improve the …

Content truncated to 500 words