Automatic In-the-wild Dataset Annotation with Deep Generalized Multiple Instance Learning

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 3542–3550 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 3542 Automatic In-the-wild Dataset Annotation with Deep Generalized Multiple Instance Learning Joana Correia 1,2, Bhiksha Raj 1, Isabel Trancoso 2 1Language Technologies Institute - Carnegie Mellon University, 2INESC-ID - University of Lisbon 5000 Forbes Avenue, Pittsburgh, PA 15213, USA, Rua Alves Redol, 9, 1000-029 Lisboa, Portugal {joanac, bhiksha}@cs.cmu.edu, isabel.trancoso@inesc-id.pt Abstract The automation of the diagnosis and monitoring of speech affecting diseases in real life situations, such as Depression or Parkinson’s disease, depends on the existence of rich and large datasets that resemble real life conditions, such as those collected from in-the-wild multimedia repositories like YouTube. However, the cost of manually labeling these large datasets can be prohibitive. In this work, we propose to overcome this problem by automating the annotation process, without any requirements for human intervention. We formulate the annotation problem as a Multiple Instance Learning (MIL) problem, and propose a novel solution that is based on end-to-end differentiable neural networks. Our solution has the additional advantage of generalizing the MIL framework to more scenarios where the data is still organized in bags, but does not meet the MIL bag label conditions. We demonstrate the performance of the proposed method in labeling an in-the-Wild Speech Medical (WSM) Corpus, using simple textual cues extracted from videos and their metadata. Furthermore we show what is the contribution of each type of textual cues for the final model performance, as well as study the influence of the size of the bags of instances in determining the difficulty of the learning problem. Keywords: multiple instance learning, deep learning, automatic dataset annotation, in-the-wild data 1. Introduction Speech is a complex bio-signal that is intrinsically related to human physiology and cognition. It has the potential to pro- vide a rich bio-marker for health, and to allow a non-invasive route to early diagnosis and monitoring of a range of con- ditions and diseases that affect speech (Orozco-Arroyave et al., 2015)(Schuller et al., 2017), from depression, to Parkin- son’s disease, Alzheimer’s disease, or a simple case of flu. With the rise of speech related machine learning applica- tions over the last decade, there has been a growing in- terest in developing tools that automatically perform non- invasive disease diagnosis and monitoring based on speech (Dibazar et al., 2002)(López-de Ipiña et al., 2013)(Lopez- de Ipiña et al., 2015)(Cummins et al., 2015)(Correia et al., 2016b)(Correia et al., 2018a). The advantages of such tools are clear: their non-invasive nature allows for an increase in patient comfort; additionally, their automatic nature makes these solutions easy to scale. However, one of the major factors that is slowing down the development of reliable tools for automatic diagnosis based on speech is the limited amount of speech medical data that is currently available, cataloged, and labeled. The existing datasets are relatively small, and have been col- lected in very controlled conditions, showing limited appli- cability in real …

Content truncated to 500 words