Learning Word Meanings and Grammar for Describing Everyday Activities in Smart Environments

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2249–2254, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Learning Word Meanings and Grammar for Describing Everyday Activities in Smart Environments Muhammad Attamimi1 Yuji Ando1 Tomoaki Nakamura1 Takayuki Nagai1 Daichi Mochihashi2 Ichiro Kobayashi3 Hideki Asoh4 1 The University of Electro-Communications, 1-5-1 Chofugaoka, Chofu-shi, Tokyo, Japan 2 Institute of Statistical Mathematics, 10-3 Midori-cho, Tachikawa, Tokyo, Japan 3 Ochanomizu University, 2-1-1 Otsuka Bunkyo-ku Tokyo, Japan 4 National Institute of Advanced Industrial Science and Technology, 1-1-1 Umezono, Tsukuba, Ibaraki, Japan {m att, ando, naka t}@apple.ee.uec.ac.jp, tnagai@ee.uec.ac.jp, daichi@ism.ac.jp, koba@is.ocha.ac.jp, h.asoh@aist.go.jp Abstract If intelligent systems are to interact with humans in a natural manner, the ability to describe daily life activities is impor- tant. To achieve this, sensing human ac- tivities by capturing multimodal informa- tion is necessary. In this study, we con- sider a smart environment for sensing ac- tivities with respect to realistic scenarios. We next propose a sentence generation system from observed multimodal infor- mation in a bottom up manner using mul- tilayered multimodal latent Dirichlet allo- cation and Bayesian hidden Markov mod- els. We evaluate the grammar learning and sentence generation as a complete process within a realistic setting. The experimen- tal result reveals the effectiveness of the proposed method. 1 Introduction Describing daily life activities is an important abil- ity of intelligent systems. In fact, we can use this ability to achieve a monitoring system that is able to report on an observed situation, or cre- ate an automatic diary of a user. Recently, sev- eral studies have been performed to generate sen- tences that describe images using Deep Learning (Vinyals et al., 2014; Fang et al., 2014; Donahue et al., 2014; Kiros et al., 2015). Although these results were good, we are interested in unsuper- vised frameworks. This is necessary to achieve a system that can adapt to the user, that is, one that can learn a user-unique language and gener- ate it automatically. Moreover, the use of crowd- sourcing should be avoided to respect the privacy of the user. Regarding this, studies on sentence generation from RGB videos have been discussed in (Yu and Siskind, 2013; Regneri et al., 2013). A promising result for language learning has been shown in (Yu and Siskind, 2013) and a quite chal- lenging effort to describe cooking activities was made in (Regneri et al., 2013). However, these studies rely only on visual information, while we aim to build a system that is able to describe every- day activities using multimodal information. To realize such systems, we need to consider two problems. The first problem is the sensing of daily life activities. In this paper, we utilize a smart house (Motooka et al., 2010) for sensing human activities. Thanks to the smart house, multimodal information such as visual, motion, and audio data can be captured. The second problem to be tack- led is verbalization of the observed scenes. To solve this problem, a multilayered multimodal la- tent Dirichlet …

Content truncated to 500 words