Zero-Shot Learning of Language Models for Describing Human Actions Based on Semantic Compositionality of Actions

PACLIC 28 !85 Zero-Shot Learning of Language Models for Describing Human Actions Based on Semantic Compositionality of Actions Hideki ASOH National Institute of Advanced Industrial Science and Technology Tsukuba, Ibaraki 305-8568 Japan h.asoh@aist.go.jp Ichiro KOBAYASHI Graduate School of Humanities and Sciences, Ochanomizu University Bunkyo-ku, Tokyo 112-8610 Japan koba@is.ocha.ac.jp Abstract We propose a novel framework for zero-shot learning of topic-dependent language models, which enables the learning of language mod- els corresponding to specific topics for which no language data is available. To realize zero- shot learning, we exploit the semantic compo- sitionality of the target topics. Complex topics are normally composed of several elementary semantic components. We found that the lan- guage model that corresponds to a particular topic can be approximated with a linear com- bination of language models corresponding to elementary components of the target topics. On the basis of the findings, we propose sim- ple methods of zero-shot learning. To confirm the effectiveness of the proposed framework, we apply the methods to the problem of gen- erating natural language descriptions of short Kinect videos of simple human actions. 1 Introduction Constructing topic-dependent language models is useful for many applications such as text mining, speech recognition, statistical machine translation, natural language interfaces, and textual descrip- tion of images or video contents. In most meth- ods of topic-dependent language model construc- tion, one general model is first constructed from a large amount of language data, and then the general model is modified with a small amount of language data regarding the target topic. The technique of taking the weighted sum of language models is of- ten used for the modification (Bacchiani and Roark, 2003; Jelinek and Mercer, 1980). However, correct- ing language data for all target topics is demanding and difficult. In particular, when each target topic becomes narrower and the number of target topics increases, it becomes impractical to correct language data for all topics. In this paper, we propose a novel framework for zero-shot learning of topic-dependent language models, which enables the learning of language models corresponding to specific topics without ob- serving language data regarding the topics on the basis of the semantic compositionality of the target topics. In the following, we consider rather fine-grained topics such as human activities. Such detailed top- ics are normally composed of several elementary se- mantic components. For example, a human action “raising left leg in the forward direction” is consid- ered as a topic. The action includes components such as “up (raise)”, “left”, “leg”, and “in the for- ward direction”. Another action “raising left hand in the side direction” shares the common elements “up” and “left” with the previous action. In this way, actions are related to each other through com- mon components. Hence, the language models gen- erated from natural language sentences describing those actions are also expected to be related to each other. We will show that using this kind of compo- sitionality, we can generate language models corre- sponding to actions for which we do not have …

Content truncated to 500 words