Multi-modal Intent Classification for Assistive Robots with Large-scale Naturalistic Datasets

Multi-modal Intent Classification for Assistive Robots with Large-scale Naturalistic Datasets Karun Mathew ♣0 Venkata S Aditya Tarigoppula♦♥ Lea Frermann♠ ♣Newline Structures Pvt Ltd. ♦Department of Biomedical Engineering, The University of Melbourne ♥ARC Training Centre in Cognitive Computing for Medical Technologies ♠School of Computing and Information Systems, The University of Melbourne karunmatthew@live.in aditya.tarigoppula@gmail.com lfrermann@unimelb.edu.au Abstract Recent years have brought a tremendous growth in assistive robots/prosthetics for peo- ple with partial or complete loss of upper limb control. These technologies aim to help the users with various reaching and grasping tasks in their daily lives such as picking up an ob- ject and transporting it to a desired location; and their utility critically depends on the ease and effectiveness of communication between the user and robot. One of the natural ways of communicating with assistive technologies is through verbal instructions. The meaning of natural language commands depends on the current configuration of the surrounding envi- ronment and needs to be interpreted in this multi-modal context, as accurate interpretation of the command is essential for a successful execution of the user’s intent by an assistive device. The research presented in this paper demonstrates how large-scale situated natural language datasets can support the development of robust assistive technologies. We lever- aged a navigational dataset comprising > 25k human-provided natural language commands covering diverse situations. We demonstrated a way to extend the dataset in a task-informed way and use it to develop multi-modal intent classifiers for pick and place tasks. Our best classifier reached > 98% accuracy in a 16-way multi-modal intent classification task, suggest- ing high robustness and flexibility. 1 Introduction Paralysis is a loss of motor function to varying de- grees of severity often resulting in severely reduced or complete loss of upper and/or lower limb control. Such impairments reduce the quality of life for mil- lions of people affected by paralysis (Armour et al., 2016) and increase their dependence upon others to perform day-to-day activities including self- or 0Work done while at Melbourne University. “Move a purple pillow from the couch to a black chair.” Multi-modal model Input Language command Agent Information Object Location Feature Extractor Intent Classifier Model Output: Intent GoToLocation + PickupObject + GoToLocation + PutObject Raw Data Object detection Object location Cross-modal entity linking Not tackled in this work Figure 1: High-level overview of our intent classifier. The system receives visual information extracted from the environment together with a natural language task command as input; and uses this to predicts the intent as a suitable sequence of actions necessary to execute the command. Visual scene parsing and cross-modal entity linking are not tackled in this work. object locomotion and object manipulation tasks like reaching, picking up an object and moving it to a desired location (pick and place). Assistive de- vices can compensate for some of the impairments provided that they can accurately infer and exe- cute user intents. Most assistive devices currently in use rely on manual control (e.g., wheelchairs controlled with joysticks), and cannot understand natural language …

Content truncated to 500 words