Logic-Guided Message Generation from Raw Real-Time Sensor Data

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6899–6908 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6899 Logic-Guided Message Generation from Raw Real-Time Sensor Data Ernie Chang, Alisa Kovtunova, Stefan Borgwardt, Vera Demberg, Kathryn Chapman, Hui-Syuan Yeh Department of Computer Science, Saarland Informatics Campus, Germany Chair for Automata Theory, Technische Universit¨at Dresden, Germany {cychang@coli,s8kachap@teams,demberg@lst,yehhui@coli}.uni-saarland.de firstname.lastname@tu-dresden.de Abstract Natural language generation in real-time settings with raw sensor data is a challenging task. We find that formulating the task as an end-to-end problem leads to two major challenges in content selection – the sensor data is both redundant and diverse across environments, thereby making it hard for the encoders to select and reason on the data. We here present a new corpus for a specific domain that instantiates these properties. It includes handover utterances that an assistant for a semi-autonomous drone uses to communicate with humans during the drone flight. The corpus consists of sensor data records and utterances in 8 different environments. As a structured intermediary representation between data records and text, we explore the use of description logic (DL). We also propose a neural generation model that can alert the human pilot of the system state and environment in preparation of the handover of control. Keywords: message generation, content selection, domain variability, low resources, description logic, experiment 1. Introduction Sensor technology has evolved in the last decade driven by the need to delegate routine tasks to machines. An ex- ample of such delegation is the Internet of Things (IoT) represented in wearables, smart homes, autonomous driving, etc. In the high-stakes applications, the system must continuously monitor the sensor data stream to de- tect any situation deterioration and react on it promptly. Moreover, in the case of a constantly changing envi- ronment, faithful data records can be very diverse and contain a lot of superfluous information. In this set- ting, it becomes challenging to detect an abnormality automatically. In this work, we consider an example that embodies both of these aspects of sensor data: redundancy and di- versity. Particularly, as a recent technological advance, drones with impressive features, advanced sensors and capabilities have become commonplace (Fuhrman et al., 2019) (e.g. for aerial surveys, mapping, aerial movies and even selfie-drones). The amount of sensor informa- tion routinely processed during a flight such as altitude, wind speed, air pressure, temperature, etc. is enormous. This is related to the fact that drones are extremely use- ful in the most remote and hard-to-reach places where very little can be controlled by human operators. As these drones are used for an increasingly wide range of tasks, interacting with drones becomes more important. To enable these interactions, it is essential to devise a natural language generation (NLG) setup that can flexi- bly connect to a variety of data records collected by the drone and convey information reliably. In this paper, we propose a neural generation model (or drone assis- tant) that verbalizes messages from sensor data records …

Content truncated to 500 words