Data Augmentation and Learned Layer Aggregation for Improved Multilingual Language Understanding in Dialogue

Findings of the Association for Computational Linguistics: ACL 2022, pages 2017 - 2033 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Data Augmentation and Learned Layer Aggregation for Improved Multilingual Language Understanding in Dialogue Evgeniia Razumovskaia Ivan Vuli´c Anna Korhonen Language Technology Lab, University of Cambridge {er563, iv250, alk23}@cam.ac.uk Abstract Scaling dialogue systems to a multitude of do- mains, tasks and languages relies on costly and time-consuming data annotation for differ- ent domain-task-language configurations. The annotation efforts might be substantially re- duced by the methods that generalise well in zero- and few-shot scenarios, and also ef- fectively leverage external unannotated data sources (e.g., Web-scale corpora). We pro- pose two methods to this aim, offering im- proved dialogue natural language understand- ing (NLU) across multiple languages: 1) Multi-SentAugment, and 2) LayerAgg. Multi- SentAugment is a self-training method which augments available (typically few-shot) train- ing data with similar (automatically labelled) in-domain sentences from large monolingual Web-scale corpora. LayerAgg learns to select and combine useful semantic information scat- tered across different layers of a Transformer model (e.g., mBERT); it is especially suited for zero-shot scenarios as semantically richer representations should strengthen the model’s cross-lingual capabilities. Applying the two methods with state-of-the-art NLU models ob- tains consistent improvements across two stan- dard multilingual NLU datasets covering 16 diverse languages. The gains are observed in zero-shot, few-shot, and even in full-data sce- narios. The results also suggest that the two methods achieve a synergistic effect: the best overall performance in few-shot setups is at- tained when the methods are used together. 1 Introduction The aim of Natural Language Understanding (NLU) in task-oriented dialogue systems is to iden- tify the user’s need from their utterance (Xu et al., 2020). This comprises the following crucial in- formation: 1) intents, what the user intends to do, and 2) (typically predefined) slots, associated ar- guments of the intent (Tur et al., 2010; Tur and De Mori, 2011) which need to be filled with spe- cific values. Intent detection is often framed as a Figure 1: Illustration of two user utterances in the ATIS flight domain with associated intents and slot tags. standard sentence classification task, where every sentence maps to one or more intent classes; slot labelling is typically cast as a sequence labelling task, where each word is labelled with a BIO-style slot tag (Bunk et al., 2020), see Figure 1. The supervised models for NLU in English are plentiful and achieve extremely high accuracy (Louvan and Magnini, 2020a; Qin et al., 2021). At the same time, porting an NLU system to any new domain and language requires collecting a large in- domain dataset, and training a model for the target language (Xu et al., 2020). Such in-domain annota- tions in multiple languages are extremely expensive and time-consuming (Rastogi et al., 2020), also re- flected in the fact that large enough dialogue NLU datasets for other languages are still few and far be- tween (Razumovskaia et al., 2021). This in turn cre- ates the demand for strong multilingual …

Content truncated to 500 words