Learning to Segment Actions from Observation and Narration

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2569–2588 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2569 Learning to Segment Actions from Observation and Narration Daniel Fried‡ Jean-Baptiste Alayrac† Phil Blunsom† Chris Dyer† Stephen Clark† Aida Nematzadeh† †DeepMind, London, UK ‡Computer Science Division, UC Berkeley dfried@berkeley.edu {jalayrac,pblunsom,cdyer,clarkstephen,nematzadeh}@google.com Abstract We apply a generative segmental model of task structure, guided by narration, to action seg- mentation in video. We focus on unsupervised and weakly-supervised settings where no ac- tion labels are known during training. De- spite its simplicity, our model performs com- petitively with previous work on a dataset of naturalistic instructional videos. Our model al- lows us to vary the sources of supervision used in training, and we find that both task structure and narrative language provide large benefits in segmentation quality. 1 Learning to Segment Actions Finding boundaries in a continuous stream is a crucial process for human cognition (Martin and Tversky, 2003; Zacks and Swallow, 2007; Levine et al., 2019; ¨Unal et al., 2019). To understand and remember what happens in the world around us, we need to recognize the action boundaries as they unfold and also distinguish the important actions from the insignificant ones. This process, referred to as temporal action segmentation, is also an im- portant first step in systems that ground natural lan- guage in videos (Hendricks et al., 2017). These sys- tems must identify which frames in a video depict actions – which amounts to distinguishing these frames from background ones – and identify which actions (e.g., boiling potatoes) each frame depicts. Despite recent advances (Miech et al., 2019; Sun et al., 2019), unsupervised action segmentation in videos remains a challenge. The recent availability of large datasets of natu- ralistic instructional videos provides an opportunity for modeling of action segmentation in a rich task context (Yu et al., 2014; Zhou et al., 2018; Zhukov et al., 2019; Miech et al., 2019; Tang et al., 2019); Work begun while DF was interning at DeepMind. Code is available at https://github.com/dpfried/action-segmentation. in these videos, a person teaches a specific high- level task (e.g., making croquettes) while describ- ing the lower-level steps involved in that task (e.g., boiling potatoes). However, the real-world nature of these datasets introduces many challenges. For example, more than 70% of the frames in one of the YouTube instructional video datasets, CrossTask (Zhukov et al., 2019), consist of background re- gions (e.g., the video presenter is thanking their viewers), which do not correspond to any of the steps for the video’s task. These datasets are interesting because they pro- vide (1) narrative language that roughly corre- sponds to the activities demonstrated in the videos and (2) structured task scripts that define a strong signal of the order in which steps in a task are typi- cally performed. As a result, these datasets provide an opportunity to study the extent to which task structure and language can guide action segmen- tation. Interestingly, young children can segment actions without …

Content truncated to 500 words