Learning Representations from Imperfect Time Series Data via Tensor Rank Regularization

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1569–1576 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1569 Learning Representations from Imperfect Time Series Data via Tensor Rank Regularization Paul Pu Liang♣∗, Zhun Liu♢∗, Yao-Hung Hubert Tsai♣ Qibin Zhao♡, Ruslan Salakhutdinov♣, Louis-Philippe Morency♢ ♣Machine Learning Department, Carnegie Mellon University, USA ♢Language Technologies Institute, Carnegie Mellon University, USA ♡Tensor Learning Unit, RIKEN Center for Artificial Intelligence Project, Japan {pliang,zhunl,yaohungt,rsalakhu,morency}@cs.cmu.edu qibin.zhao@riken.jp Abstract There has been an increased interest in mul- timodal language processing including multi- modal dialog, question answering, sentiment analysis, and speech recognition. However, naturally occurring multimodal data is often imperfect as a result of imperfect modalities, missing entries or noise corruption. To ad- dress these concerns, we present a regulariza- tion method based on tensor rank minimiza- tion. Our method is based on the observation that high-dimensional multimodal time series data often exhibit correlations across time and modalities which leads to low-rank tensor rep- resentations. However, the presence of noise or incomplete values breaks these correlations and results in tensor representations of higher rank. We design a model to learn such ten- sor representations and effectively regularize their rank. Experiments on multimodal lan- guage data show that our model achieves good results across various levels of imperfection. 1 Introduction Analyzing multimodal language sequences spans various fields including multimodal dialog (Das et al., 2017; Rudnicky, 2005), question answer- ing (Antol et al., 2015; Tapaswi et al., 2015; Das et al., 2018), sentiment analysis (Morency et al., 2011), and speech recognition (Palaskar et al., 2018). Generally, these multimodal se- quences contain heterogeneous sources of infor- mation across the language, visual and acous- tic modalities. For example, when instructing robots, these machines have to comprehend our verbal instructions and interpret our nonverbal be- haviors while grounding these inputs in their vi- sual sensors (Schmerling et al., 2017; Iba et al., 2005). Likewise, comprehending human inten- tions requires integrating human language, speech, ∗first two authors contributed equally clean multimodal data <latexit sha1_base64="h1pzHCMR0RKSBQVEr4zlmGh8c=">ACBXicbVA9SwNBEN3z2/gVtdRiMQhW4U4FLYM2lgomCskR5vY2yeLu3rE7J4YjY1/xcZCE Vv/g53/xk1yhSY+GHi8N8PMvCiVwqLvf3szs3PzC4tLy6WV1bX1jfLmVsMmWG8zhKZmNsILJdC8zoKlPw2NRxUJPlNdHc+9G/ubEi0dfYT3moKtFRzBAJ7XLuy3kD5gzyUFTlUkUKolB0hgQBu1yxa/6I9BpEhSkQgpctstfrThmeIamQRrm4GfYpiDQeE2DEqtzPIU2B10edNRDYrbMB9MaD7TolpJzGuNKR+nsiB2VtX0WuU wH27KQ3FP/zmhl2TsNc6DRDrtl4USeTFBM6jITGwnCGsu8IMCPcrZT1wABDF1zJhRBMvjxNGofV4KjqXx1XamdFHEtkh+yRAxKQE1IjF+S1Akj+SZvJI378l78d69j3HrjFfMbJM/8D5/AB0omPM=</latexit> imperfect multimodal data <latexit sha1_base64="I3wmMCkoG+gn08puQZqeM8eRbQ=">ACXicbVC7SgNBFJ31GeMramkzGASrsKuClkEbywhGhSEu7N34+DM7jJzVwxLWht/xcZCEVv/wM 6/cfIo1Hhg4HDOvdw5J8yUtOT7X97M7Nz8wmJpqby8srq2XtnYvLRpbgQ2RapScx2CRSUTbJIkhdeZQdChwqvw9nToX92hsTJNLqifYUdDL5GxFEBO6lZ4m/CeCqkzNDEK4jpXJHUageIREAy6lapf80fg0ySYkCqboNGtfLajVOQaExIKrG0FfkadAgxJoXBQbucWMxC30MOWowlotJ1ilGTAd50S8Tg17iXER+rPjQK0tX0dukNdGP/ekPxP6+VU 3zcKWS5YSJGB+Kc8Up5cNaeCSNS6/6joAw0v2VixswIMiV3YlBH8jT5PL/VpwUPD6v1k0kdJbNdtgeC9gRq7Mz1mBNJtgDe2Iv7NV79J69N+9PDrjTXa2C94H9+GKZrX</latexit> tensor rank regularization <latexit sha1_base64="qWvflx4gMCU5rXKIOGbeflghyQ=">ACDHicbVC7TgJBFJ3F+ILtbSZSEysyK6aEm0scREHgkQMjvchQmzs5uZu0bc8AE2 /oqNhcbY+gF2/o3DQqHgrU7OTf3nuPHUh03W8nt7S8srqWXy9sbG5t7xR39+omSjSHGo9kpJs+MyCFghoKlNCMNbDQl9Dwh1cTvXEH2ohI3eIohk7I+koEgjO0VLdYaiPcox+kCMpEmqmhlRDP5FMi4fMNLYut+xmQxeBNwMlMptqt/jV7kU8CUEhl8yYlufG2EmZRsEljAvtxEDM+JD1oWhYiGYTpqFGdMjy/RoYH8 JIoU0Y39vpCw0ZhT61hkyHJh5bUL+p7USDC46qVBxYrPy6aEgkRQjOmG9oQGjnJkAeNa2F8pHzDNONr+CrYEbz7yIqiflL3TsntzVqpczurIkwNySI6JR85JhVyTKqkRTh7JM3klb86T8+K8Ox9Ta86Z7eyTP+N8/gBHLZxj</latexit> clean entries <latexit sha1_base64="wuMzvz1f+N4dxLXkrZwJSF97FMk=">AB/XicbVDLSgMxFM3UV62v+ti5CRbBVZlRQZdFNy4r2Ae0Q8mkt21oJjMkd8Q6FH/FjQtF3Pof7vwb03YW2nogcDjnvnKCWA qDrvt5JaWV1bX8uFjc2t7Z3i7l7dRInmUORjHQzYAakUFBDgRKasQYWBhIawfB64jfuQRsRqTscxeCHrK9ET3CGVuoUD9oID5hyCUxRUKgFmHGnWHL7hR0kXgZKZEM1U7xq92NeBLaAVwyY1qeG6OfMo3CTh4X2omBmPEh60PLUsVCMH46vX5Mj63Spb1I26eQTtXfHSkLjRmFga0MGQ7MvDcR/NaCfYu/VSoOEFQfLaol0iKEZ1EQbtCA0c5soRxLeytlA+YZhxtYAUbgjf/5UVSPy17Z2X39rxUu criyJNDckROiEcuSIXckCqpEU4eyTN5JW/Ok/PivDsfs9Kck/Xskz9wPn8AKAWVqw=</latexit> imperfect entries <latexit sha1_base64="uEy9xlvD1dXmU2/rxG3w4dCph+Q=">ACAXicbVDLSgMxFM3UV62vUTeCm2ARXJUZFXRZdOygn1AO5RMeqcNzTxI7ohlqBt/xY0LRdz6F+78G9N2Ftp6IHA4594k5/iJFBod59 sqLC2vrK4V10sbm1vbO/buXkPHqeJQ57GMVctnGqSIoI4CJbQSBSz0JT94fXEb96D0iKO7nCUgBeyfiQCwRkaqWsfdBAeMBNhAioAjhQiVAL0uGuXnYozBV0kbk7KJEeta391ejFPQ3MBl0zrtusk6GVMoeASxqVOqiFhfMj60DY0YiFoL5smGNjo/RoECtzIqRT9fdGxkKtR6FvJkOGAz3vTcT/vHaKwaWXiShJESI+eyhIJcWYTuqgPaFMajkyhHElzF8pHzDFOJrSqYEdz7yImcVtyzinN7Xq5e5XUySE5IifEJRekSm 5IjdQJ4/kmbySN+vJerHerY/ZaMHKd/bJH1ifP4Lml48=</latexit> Figure 1: Clean multimodal time series data (in shades of green) exhibits correlations across time and across modalities, leading to redundancy in low rank tensor representations. On the other hand, the presence of imperfect entries (in gray, blue, and red) breaks these correlations and leads to higher rank tensors. In these scenarios, we use tensor rank regularization to learn tensors that more accurately represent the true correla- tions and latent structures in multimodal data. facial behaviors, and body postures (Mihalcea, 2012; Rossiter, 2011). However, as much as more modalities are required for improved performance, we now face a challenge of imperfect data where data might be 1) incomplete due to mismatched modalities or sensor failure, or 2) corrupted with random or structured noise. As a result, an im- portant research question involves learning robust …

Content truncated to 500 words