Human-in-the-loop Robotic Grasping Using BERT Scene Representation

Proceedings of the 29th International Conference on Computational Linguistics, pages 2992–3006 October 12–17, 2022. 2992 Human-in-the-loop Robotic Grasping using BERT Scene Representation Yaoxian Song1,2, Penglei Sun1, Pengfei Fang3, Linyi Yang2,4, Yanghua Xiao1, Yue Zhang2,4∗ 1School of Computer Science, Fudan University 2School of Engineering, Westlake University 3School of Engineering, Australian National University 4 Institute of Advanced Technology, Westlake Institute for Advanced Study {songyaoxian, yanglinyi, zhangyue}@westlake.edu.cn, {plsun20, shawyh}@fudan.edu.cn, u5765437@anu.edu.au Abstract Current NLP techniques have been greatly ap- plied in different domains. In this paper, we propose a human-in-the-loop framework for robotic grasping in cluttered scenes, investigat- ing a language interface to the grasping pro- cess, which allows the user to intervene by nat- ural language commands. This framework is constructed on a state-of-the-art grasping base- line, where we substitute a scene-graph rep- resentation with a text representation of the scene using BERT. Experiments on both sim- ulation and physical robot show that the pro- posed method outperforms conventional object- agnostic and scene-graph based methods in the literature. In addition, we find that with human intervention, performance can be significantly improved. Our dataset and code are available on our project website 1. 1 Introduction Grasping (Mahler et al., 2019) is a fundamental task for robot systems. It is useful for warehousing, manufacturing, medicine, retail, and service robots. One setting in robotic grasping is to grasp object orderly without disturbing the remaining in clut- tered scenes (Chen et al., 2021b; Mees and Burgard, 2020; Zhang et al., 2021a) (called collision-free grasp). To solve this problem, a typical method (Zhu et al., 2021) parses the input into a scene graph first (Figure. 1(b)), in order to infer the spa- tial relation between objects and select a collision- free object for grasping. In particular, as shown in Figure. 2(a), nodes in a scene graph represent ob- jects and edges represent their spatial relationship. Such structural knowledge can effectively improve the grasping performance as compared to an end- to-end model without scene structure information (Chu et al., 2018) (Figure. 1(a)). ∗ Corresponding Author 1https://sites.google.com/view/ hitl-grasping-bert Figure 1: Various model structures for robotic grasp- ing. (a) End2End (Chu et al., 2018) outputs an object- agnostic grasping by directly modeling on the input images. (b) Scene-Graph Representation fuses a gen- erated scene graph with visual feature to predict grasp- ing. (c) Scene-Text Representation (ours) makes use of language scene description and visual feature for achieving collision-free grasping. As a structured representation of the scene, the scene graph has a few limitations. For example, it can be costly to manually label scene graphs, and the amount of existing labeled data is quite small. In practice, due to variance in the working environ- ment, it can be necessary to calibrate a scene under- standing model when deployed on physical robots (Zhu et al., 2021). This problem can be regarded as a domain adaptation task, which requires a cer- tain amount of labeled scene graph data (Xu et al., 2017). In addition, graphs are relatively abstract and thus inconvenient for human-robot interaction. …

Content truncated to 500 words