Draw Me a Flower: Processing and Grounding Abstraction in Natural Language

Draw Me a Flower: Processing and Grounding Abstraction in Natural Language Royi Lachmy1,2 Valentina Pyatkin1,2 Avshalom Manevich1 Reut Tsarfaty1,2 1Bar-Ilan University, Ramat Gan, Israel 2Allen Institute for Artificial Intelligence, Tel Aviv, Israel {royi.lachmy,reut.tsarfaty}@biu.ac.il {valpyatkin,avshalomman}@gmail.com Abstract Abstraction is a core tenet of human cognition and communication. When composing natural language instructions, humans naturally evoke abstraction to convey complex procedures in an efficient and concise way. Yet, interpret- ing and grounding abstraction expressed in NL has not yet been systematically studied in NLP, with no accepted benchmarks specifi- cally eliciting abstraction in NL. In this work, we set the foundation for a systematic study of processing and grounding abstraction in NLP. First, we deliver a novel abstraction elic- itation method and present HEXAGONS, a 2D instruction-following game. Using HEXAGONS we collected over 4k naturally occurring visually-grounded instructions rich with di- verse types of abstractions. From these data, we derive an instruction-to-execution task and assess different types of neural models. Our results show that contemporary models and modeling practices are substantially in- ferior to human performance, and that model performance is inversely correlated with the level of abstraction, showing less satisfying performance on higher levels of abstraction. These findings are consistent across mod- els and setups, confirming that abstraction is a challenging phenomenon deserving further attention and study in NLP/AI research. 1 Introduction As human–computer interaction in natural lan- guage (NL) becomes more and more pervasive (e.g., via smart devices and chatbots), a cogni- tive phenomenon known as abstraction, which is prevalent in human communication and cognition, is taking a central role in the way users communi- cate their intentions and needs to artificial agents. When communicating in NL with a human or an artificial agent, a human may issue a request for a single action such as ‘‘send an email’’ or ‘‘set my alarm’’. However, when engaging with more com- plex tasks that require multiple actions, humans often evoke abstraction in order to communicate their intentions in an economic-yet-precise way. Examples for evoking abstraction when issuing a complex request that consists of multiple actions may be: ‘‘Schedule a group meeting every other Wednesday until the end of the year, unless there are holidays.’’ For an autonomous car, an envi- sioned request might be ‘‘Circle the block looking for a shady parking spot, park at the first spot you see. Try this four times, and one more time allowing for non-shady parking. In no luck, try the adjacent block.’’ In fact, even the individual request ‘‘send an email’’ is in itself an abstraction over a sequence of multiple individual actions such as: ‘‘open your inbox, click New Mail, select a recipient,’’ and so forth. Abstraction is defined by Wing (2011) as ‘‘[let- ting] one object stand for many. It is used to capture essential properties common to a set of objects while hiding irrelevant distinctions among them’’. In the calendar example, a single utterance references multiple meetings in multiple days. Likewise, in the autonomous car example, the speaker evokes some sort of control …

Content truncated to 500 words