ChartDialogs: Plotting from Natural Language Instructions

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3559–3574 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3559 CHARTDIALOGS: Plotting from Natural Language Instructions Yutong Shao and Ndapa Nakashole Computer Science and Engineering University of California, San Diego La Jolla, CA 92093 {yshao, nnakashole}@eng.ucsd.edu Abstract This paper presents the problem of conversa- tional plotting agents that carry out plotting ac- tions from natural language instructions. To facilitate the development of such agents, we introduce CHARTDIALOGS, a new multi-turn dialog dataset, covering a popular plotting li- brary, matplotlib. The dataset contains over 15, 000 dialog turns from 3, 200 dialogs covering the majority of matplotlib plot types. Extensive experiments show the best- performing method achieving 61% plotting ac- curacy, demonstrating that the dataset presents a non-trivial challenge for future research on this task. 1 Introduction Advances in machine language understanding (Hirschberg and Manning, 2015) have sparked in- terest in using artificial intelligence to address diffi- cult problems involving language. In this work, we are interested in the problem of plotting via natural language instructions. Plotting is a method for visu- alizing data and mathematical functions. Plotting libraries such as matplotlib support functional- ity on a range of levels, from general, “change the X-axis from linear to log scale”, to specific, “color this screen pixel red”. Yet, using such libraries can be difficult for novice users and time consuming even for experts. This obstacle, coupled with the increasing popularity of the scientific method of gleaning information from data (Hey et al., 2009; Dhar, 2013), motivates our objective of designing natural language interfaces (NLIs) for plotting. NLIs for plotting can be organized into three categories based on what the user is expected to describe: the data, the function, or the plot. Describing the Data or the Function. In the first category of plotting NLIs, users are expected to de- scribe the data they would like to visualize, by pos- ing queries such as: “Show me medals for hockey and skating by country.” Queries may involve sim- ple data analysis: “Is there a seasonal trend for bike usage?” The system retrieves the relevant data, performs simple data analysis, and produces a visualization. This category of NLIs has been stud- ied in Human Computer Interaction and related areas (Gao et al., 2015; Setlur et al., 2016; Srini- vasan and Stasko, 2017; Yu and Silva, 2019; Sun et al., 2010). In the second category of plotting NLIs, users specify the function they would like to visualize. In this category, commercial products such as wolfra- malpha.com yield results for queries such as “plot the tangent to x2 at x = 0.5”. The system pro- cesses such queries by leveraging knowledge of functions and mathematical principles. Describing the Plot. In the two categories we have discussed, users only describe what data or func- tion they would like to visualize without describing how to visualize it. The system is in charge of all plotting details, which are not accessible to users. We …

Content truncated to 500 words