Talk2Car: Taking Control of Your Self-Driving Car

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2088–2098, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2088 Talk2Car: Taking Control of Your Self-Driving Car Thierry Deruyttere1 Simon Vandenhende2 Dusan Grujicic2 Luc Van Gool2 Marie-Francine Moens1 1Department of Computer Science (CS) 2Department of Electrical Engineering (ESAT) KU Leuven {thierry.deruyttere, sien.moens}@cs.kuleuven.be {simon.vandenhende, dusan.grujicic, luc.vangool}@esat.kuleuven.be Abstract A long-term goal of artificial intelligence is to have an agent execute commands commu- nicated through natural language. In many cases the commands are grounded in a vi- sual environment shared by the human who gives the command and the agent. Exe- cution of the command then requires map- ping the command into the physical visual space, after which the appropriate action can be taken. In this paper we consider the former. Or more specifically, we consider the problem in an autonomous driving set- ting, where a passenger requests an action that can be associated with an object found in a street scene. Our work presents the Talk2Car dataset, which is the first object referral dataset that contains commands writ- ten in natural language for self-driving cars. We provide a detailed comparison with re- lated datasets such as ReferIt, RefCOCO, RefCOCO+, RefCOCOg, Cityscape-Ref and CLEVR-Ref. Additionally, we include a performance analysis using strong state-of- the-art models. The results show that the pro- posed object referral task is a challenging one for which the models show promising results but still require additional research in natu- ral language processing, computer vision and the intersection of these fields. The dataset can be found on our website: http:// macchina-ai.eu/ 1 Introduction Researchers have studied the problem of under- standing actions communicated through natural language in both simulated (Das et al., 2017; Gor- don et al., 2017; Hermann et al., 2017) and real environments (Loghmani et al., 2018; de Vries et al., 2018; Anderson et al., 2017). This paper focuses on the latter. More concretely, we con- sider the problem in an autonomous driving set- ting, where a passenger can control the actions of an Autonomous Vehicle (AV) by giving natu- ral language commands. We hereunder argue why this problem setting is particularly interesting. First, a recent study by Richardson and Davies (2018) has shown that the majority of the public is reluctant to step inside an AV. A possible expla- nation for this might be the lack of control which can be unsettling to some. Providing a way to communicate with the vehicle could help alleviate this uneasiness. Second, an AV can become hesitant in some situations (Robitzski, 2019). By giving a task or command, the passenger could guide the agent in its decision process. Third, some situations request feedback. For example, a passenger might indicate that they want to park in the shade during a sunny day. Finally, the problem of urban scene understanding is one of practical relevance that has been well studied (Cordts et al., 2016; Geiger et al., …

Content truncated to 500 words