OCID-Ref: A 3D Robotic Dataset With Embodied Language For Clutter Scene Grounding

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5333–5338 June 6–11, 2021. ©2021 Association for Computational Linguistics 5333 OCID-Ref: A 3D Robotic Dataset with Embodied Language for Clutter Scene Grounding Ke-Jyun Wang∗ Yun-Hsuan Liu∗ Hung-Ting Su Jen-Wei Wang Yu-Siang Wang Winston H. Hsu Wen-Chin Chen National Taiwan University Abstract To effectively apply robots in working environ- ments and assist humans, it is essential to de- velop and evaluate how visual grounding (VG) can affect machine performance on occluded objects. However, current VG works are lim- ited in working environments, such as offices and warehouses, where objects are usually oc- cluded due to space utilization issues. In our work, we propose a novel OCID-Ref dataset featuring a referring expression segmentation task with referring expressions of occluded ob- jects. OCID-Ref consists of 305,694 referring expressions from 2,300 scenes with providing RGB image and point cloud inputs. To re- solve challenging occlusion issues, we argue that it’s crucial to take advantage of both 2D and 3D signals to resolve challenging occlu- sion issues. Our experimental results demon- strate the effectiveness of aggregating 2D and 3D signals but referring to occluded objects still remains challenging for the modern vi- sual grounding systems. OCID-Ref is pub- licly available at https://github.com/ lluma/OCID-Ref 1 Introduction Visual grounding (VG), which aims to locate the object according to a structured language query, is a crucial task in natural language processing (NLP), computer vision (CV), and robotics. Recent VG studies most focus on web-crawled images such as (Kazemzadeh et al., 2014; Krishna et al., 2017; Mao et al., 2016; Yu et al., 2016). However, VG for human-robot interaction (HRI) is less explored. Most of the images in existing VG datasets are peo- ple and daily necessities, e.g., RefCOCO contains mainly persons, cars, and cats, which are separated and therefore easier to detect. Nevertheless, work- ing spaces such as offices or warehouses, where robots are usually applied to assist works, are usu- ally crowded, and objects are overlapped with each ∗ Equal contribution. Figure 1: A hard case where visual grounding (VG) network fails to predict the occluded object in clut- ter scene. Our dataset provides more such cases than other datasets, which are commonly seen in the work- ing spaces, like offices and warehouses. other to utilize space better. Therefore, objects in working environments are often occluded and hard to detect. Previous work (Ralph and Moussa, 2005) sug- gested that a system that uses language for human- computer interaction can help non-professionals instruct robots to complete technical work and col- laborate. Recent research pointed out that VG plays an important role in HRI. (Shridhar and Hsu, 2018) utilized VG to resolve ambiguity in grasping tasks. (Matuszek) studied how the robot learns about ob- jects and tasks in an environment via nature lan- guage queries. Therefore, explicit language instruc- tions and good referring (grounding) expressions are pivotal in human-robot interaction and improve communication between non-expert humans and 5334 robots. …

Content truncated to 500 words