EUDAMU at SemEval-2017 Task 11: Action Ranking and Type Matching for End-User Development

Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 1000–1004, Vancouver, Canada, August 3 - 4, 2017. c⃝2017 Association for Computational Linguistics EUDAMU at SemEval-2017 Task 11: Action Ranking and Type Matching for End-User Development Marek Kubis and Paweł Skórzewski and Tomasz Zi˛etkiewicz Faculty of Mathematics and Computer Science Adam Mickiewicz University in Pozna´n {mkubis,pawel.skorzewski,tomasz.zietkiewicz}@amu.edu.pl Abstract The paper describes a system for end-user development using natural language. Our approach uses a ranking model to identify the actions to be executed followed by ref- erence and parameter matching models to select parameter values that should be set for the given commands. We discuss the results of evaluation and possible improve- ments for future work. 1 Introduction The goal of the end-user development (EUD) is to provide users of software systems with the tools to create, extend or modify software (Lieberman et al., 2006). End-User Development using Natu- ral Language is one of the tasks of SemEval-2017 International Workshop on Semantic Evaluation. The idea of using tools for software devel- opment more accessible than programming lan- guages has been discussed almost since the begin- ning of computers. The concept of programming with natural language commands and its conse- quences have been considered already in the 70s (Dijkstra, 1979). However, only in recent years these ideas could be put into practice with the rapid development of advanced natural language processing methods. A comprehensive overview of recent trends and achievements in EUD has been presented by Pa- ternò (2013). Although most common solution for EUD problem are various variations on graph- ical user interfaces, solutions using NLP are also present. One of the examples is the Koala (later: CoScripter) system (web browser extension), that uses “sloppy programming”, i.e. pseudo-natural language instructions, to automate business pro- cesses on the web (Little et al., 2007). Other sys- tems that use simple natural language commands to achieve programming-like goals, described in recent years, include SPOK, an EUD environment for smart homes (Coutaz et al., 2014), and Natu- ralMash, an EUD tool for creating web mashups (Aghaee and Pautasso, 2014). This paper describes our system for the end-user development using natural language and reports its performance according to the SemEval 2017 Task 11 evaluation criteria (Sales et al., 2017). 2 Data preparation We pre-process all the input data: both the action knowledge base and natural language commands. The pre-processing is done in several stages. It in- cludes basic text processing as well as adding lexi- cal features. We use these features to better match the commands to actions. The pre-processor op- erates on JSON files, in each step adding new fields to the JSON structure so the original data are not lost in the process and can be used in fur- ther steps. The input fields we annotate are: desc, name, value, nl_command_statment, provider, sample, tags and api-name. The first pre-processing stage is sentence splitting. The sentence splitter is applied to nl_command_statment and desc fields only. The second step is tokenization. The input data are amended …

Content truncated to 500 words