MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2950–2962 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2950 MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark Haoran Li Abhinav Arora Shuohui Chen Anchit Gupta Sonal Gupta Yashar Mehdad Facebook {aimeeli,abhinavarora,shuohui,anchit}@fb.com {mehdad,sonalgupta}@fb.com Abstract Scaling semantic parsing models for task- oriented dialog systems to new languages is of- ten expensive and time-consuming due to the lack of available datasets. Available datasets suffer from several shortcomings: a) they contain few languages b) they contain small amounts of labeled examples per language c) they are based on the simple intent and slot detection paradigm for non-compositional queries. In this paper, we present a new mul- tilingual dataset, called MTOP, comprising of 100k annotated utterances in 6 languages across 11 domains. We use this dataset and other publicly available datasets to conduct a comprehensive benchmarking study on us- ing various state-of-the-art multilingual pre- trained models for task-oriented semantic pars- ing. We achieve an average improvement of +6.3 points on Slot F1 for the two existing mul- tilingual datasets, over best results reported in their experiments. Furthermore, we demon- strate strong zero-shot performance using pre- trained models combined with automatic trans- lation and alignment, and a proposed distant supervision method to reduce the noise in slot label projection. 1 Introduction With the rising adoption of virtual assistant prod- ucts, task-oriented dialog systems have been attract- ing more attention in both academic and industrial communities. One of the first steps in these systems is to extract meaning from the natural language used in conversation to build a semantic representa- tion of the user utterance. Typical systems achieve this by classifying the intent of the utterance and tagging the corresponding slots. With the goal of handling more complex queries, recent approaches propose hierarchical representations (Gupta et al., 2018) that are expressive enough to capture the task-specific semantics of complex nested queries. Although, there have been sizable efforts around developing successful semantic parsing models for task-oriented dialog systems in English (Mesnil et al., 2013; Liu and Lane, 2016; Gupta et al., 2018; Rongali et al., 2020), we have only seen limited works for other languages. This is mainly due to the painstaking process of manually annotating and creating large datasets for this task in new lan- guages. In addition to the shortage of such datasets, existing datasets (Upadhyay et al., 2018; Schus- ter et al., 2019a) are not sufficiently diversified in terms of languages and domains, and do not capture complex nested queries. This makes it difficult to perform more systematic and rigorous experimen- tation and evaluation for this task across multiple languages. Building on these considerations and recent ad- vancements on cross-lingual pre-trained models (Devlin et al., 2019; Lample and Conneau, 2019; Conneau et al., 2020), this paper is making an ef- fort to bridge the above mentioned gaps. The main contributions of this paper can be summarized as follows: • MTOP Dataset: We release an almost-parallel multilingual …

Content truncated to 500 words