Rationally Reappraising ATIS-based Dialogue Systems

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5503–5507 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5503 Rationally Reappraising ATIS-based Dialogue Systems Jingcheng Niu and Gerald Penn Department of Computer Science University of Toronto Toronto, Canada {niu,gpenn}@cs.toronto.edu Abstract The Air Travel Information Service (ATIS) corpus has been the most common benchmark for evaluating Spoken Language Understand- ing (SLU) tasks for more than three decades since it was released. Recent state-of-the-art neural models have obtained F1-scores near 98% on the task of slot filling. We developed a rule-based grammar for the ATIS domain that achieves a 95.82% F1-score on our evaluation set. In the process, we furthermore discov- ered numerous shortcomings in the ATIS cor- pus annotation, which we have fixed. This paper presents a detailed account of these shortcomings, our proposed repairs, our rule- based grammar and the neural slot-filling ar- chitectures associated with ATIS. We also ra- tionally reappraise the motivations for choos- ing a neural architecture in view of this ac- count. Fixing the annotation errors results in a relative error reduction of between 19.4 and 52% across all architectures. We nevertheless argue that neural models must play a different role in ATIS dialogues because of the latter’s lack of variety. 1 Introduction Slot filling has received a great deal of recent at- tention from the SLU community. Typically, it is characterized as a sequence labeling problem in which certain tokens are identified as fillers that contribute argument values to a meaning repre- sentation through “slot” positions in the utterance. Wang et al. (2011) first used conditional random fields (CRF) for slot filling. A few years later, in- spired by the success of recurrent neural networks (RNN) in language modeling (Mikolov et al., 2011), Mesnil et al. (2013) developed the first RNN slot filler that achieved a relative error reduc- tion of 14%. Subsequently, different variations of RNN such as LSTM (Yao et al., 2014) were devel- oped for slot filling, followed by encoder-decoder models that could utilize information from the en- tire sentence (Kurata et al., 2016), both of which avail themselves of an attention mechanism (Zhu and Yu, 2017; Li et al., 2018). As recently as Wang et al. (2018), Deep Reinforcement Learn- ing (DRL) has been proposed as a way to refine encoder-decoder models on sparsely distributed tags; this has achieved the highest reported per- formance so far. This development has taken place in parallel, however, with work that has used qualitative er- ror analyses to cast doubt on the continued use of ATIS as a benchmark for progress in slot filling. Most recently, B´echet and Raymond (2018) con- clude that ATIS is simply too “shallow” to offer anything of additional substance for DNN-based architectures to achieve, formulating a three-way taxonomy of errors in the reference annotation for the ATIS corpus that account for roughly half of the remaining errors still faced by state-of-the-art slot filling models. Even prior to the recent pop- ularity …

Content truncated to 500 words