A Shared Task for Spoken CALL?

A Shared Task for Spoken CALL? Claudia Baur1, Johanna Gerlach1, Manny Rayner1, Martin Russell2, Helmer Strik3 (1)FTI/TIM, University of Geneva, Switzerland (2) Department of Electronic, Electrical and Systems Engineering, University of Birmingham (3) Centre for Language Studies (CLS), Radboud University Nijmegen Claudia.Baur@unige.ch, Johanna.Gerlach@unige.ch, Emmanuel.Rayner@unige.ch m.j.russell@bham.ac.uk, w.strik@let.ru.nl Abstract We argue that the field of spoken CALL needs a shared task in order to facilitate comparisons between different groups and methodolo- gies, and describe a concrete example of such a task, based on data collected from a speech-enabled online tool which has been used to help young Swiss German teens practise skills in English conversation. Items are prompt-response pairs, where the prompt is a piece of German text and the response is a recorded English audio file. The task is to label pairs as “accept” or “reject”, accepting responses which are grammatically and linguistically correct to match a set of hidden gold standard answers as closely as possible. Initial resources are provided so that a scratch system can be constructed with a minimal investment of effort, and in particular without necessarily using a speech recogniser. Training data for the task will be released in June 2016, and test data in January 2017. Keywords: CALL, shared tasks, speech recognition, metrics 1. Introduction The history of human language technology shows that the introduction of a shared task1 often has a positive effect. Friendly competition motivates people, and the ability to make direct comparisons between different approaches to solving the same problem makes it easier to identify the ideas that work, so that effort can be focused more pro- ductively. A prominent series of examples are the various tasks based on the Wall Street Journal corpus, including speech recognition (Bahl et al., 1995), parsing (Riezler et al., 2002) and several types of semantic analysis (Pradhan et al., 2007). Perhaps even more importantly, work on ma- chine learning during the 21st century has to a consider- able extent been driven by the handwritten digit recogni- tion task (Goodfellow et al., 2016). Other well-known ex- amples of shared tasks include ATIS in the early 90s (Zue et al., 1994), which had a strong effect on interactive spo- ken language systems; the Named Entity Recognition task (Tjong Kim Sang and De Meulder, 2003), which similarly influenced work on information extraction; and the Recog- nizing Textual Entailment task (Dagan et al., 2006), which has influenced work on question answering. In all these cases, introduction of the shared task created a new community with frequent productive interactions be- tween many groups, and substantially advanced a whole subfield inside the space of a few years. The sociology of the process has become familiar to many researchers. A shared task forces each group to look closely at what other groups are doing, and in particular to study meth- ods which are achieving high scores in the competitions. It encourages development of a common vocabulary of con- cepts. Above all, it introduces widely accepted evaluation procedures and metrics that permit objective comparisons, both …

Content truncated to 500 words