SemEval-2017 Task 11: End-User Development using Natural Language

Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 556–564, Vancouver, Canada, August 3 - 4, 2017. c⃝2017 Association for Computational Linguistics SemEval-2017 Task 11: End-User Development using Natural Language Juliano Efson Sales, Siegfried Handschuh, Andr´e Freitas Department of Computer Science and Mathematics University of Passau, Germany {juliano-sales, siegfried.handschuh, andre.freitas}@uni-passau.de Abstract This task proposes a challenge to support the interaction between users and appli- cations, micro-services and software APIs using natural language. It aims to support the evaluation and evolution of the discus- sions surrounding the application natural language processing techniques within the context of end-user natural language pro- gramming, under scenarios of high lexical and semantic heterogeneity. 1 Introduction The specific syntax of traditional programming languages and the user effort associated with find- ing, understanding and integrating multiple inter- faces within a software development task, defines the intrinsic complexity of programming. De- spite the widespread demand for automating ac- tions within a digital environment, even the ba- sic software development tasks require previous (usually extensive) software development exper- tise. Domain experts processing data, analysts au- tomating recurrent tasks, or a businessman testing an idea on the web depend on the mediation of programmers to materialise their demands, inde- pendently of the simplicity of the task to be ad- dressed and on the availability of existing services and libraries. Recent advances in natural language process- ing bring the opportunity of improving the interac- tion between users and software artefacts, support- ing users to program tasks using natural language- based communication. This ability to match users’ actions intents and information needs to formal actions within an application programming inter- face (API), using the semantics of natural lan- guage as the mediation layer between both, can drastically impact the accessibility of software de- velopment. Despite the fact that some software development tasks with stricter requirements will always depend on the precise semantic definition of programming languages, there is a vast spec- trum of applications with softer formalisation re- quirements. This subset of applications can be de- fined and built with the help of natural language descriptions. This SemEval task aims to develop the state-of- the-art discussions and techniques concerning the semantic interpretation of natural language com- mands and user action intents, bridging the seman- tic gap between users and software artefacts. The practical relevance of the challenge lies in the fact that addressing this task supports improving the accessibility of programming (meaning a system- atic specification of computational operations) to a large spectrum of users which have the demand for increasing automation within some specific tasks. Moreover, with the growing availability of soft- ware artefacts, such as APIs and services, there is a higher demand to support the discoverabil- ity of these resources, i.e. devising principled se- mantic interpretation approaches to semantically match interface descriptions with the intent from users. The proposed task also intersects with demands from the field of robotics, as part of the human- robot interaction area, which depends on a sys- tematic ability to address user …

Content truncated to 500 words