MetaTS: Meta Teacher-Student Network for Multilingual Sequence Labeling with Minimal Supervision

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3183–3196 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3183 MetaTS: Meta Teacher-Student Network for Multilingual Sequence Labeling with Minimal Supervision Zheng Li1, Danqing Zhang1, Tianyu Cao1, Ying Wei2, Yiwei Song1, Bing Yin1 1Amazon.com Inc, CA, USA 2City University of Hong Kong, Hong Kong, China 1{amzzhe,danqinz,caoty,ywsong,alexbyin}@amazon.com 2yingwei@cityu.edu.hk Abstract Sequence labeling aims to predict a fine- grained sequence of labels for the text. How- ever, such formulation hinders the effective- ness of supervised methods due to the lack of token-level annotated data. This is exacer- bated when we meet a diverse range of lan- guages. In this work, we explore multilin- gual sequence labeling with minimal super- vision using a single unified model for mul- tiple languages. Specifically, we propose a Meta Teacher-Student (MetaTS) Network, a novel meta learning method to alleviate data scarcity by leveraging large multilingual unla- beled data. Prior teacher-student frameworks of self-training rely on rigid teaching strate- gies, which may hardly produce high-quality pseudo-labels for consecutive and interdepen- dent tokens. On the contrary, MetaTS allows the teacher to dynamically adapt its pseudo- annotation strategies by the student’s feedback on the generated pseudo-labeled data of each language and thus mitigate error propagation from noisy pseudo-labels. Extensive exper- iments on both public and real-world multi- lingual sequence labeling datasets empirically demonstrate the effectiveness of MetaTS1. 1 Introduction Sequence labeling or tagging is the task of detect- ing the boundary of all occurring entity mentions from unstructured text and classifying them into predefined types, such as Named Entity Recog- nition (NER) (Chiu and Nichols, 2016), Aspect- Based Sentiment Analysis (ABSA) (Mitchell et al., 2013), etc. An entity mention should be a single word or a sequence of words that contain key in- formation, such as a person, location, or institution. In the E-commerce search domain, we need to rec- ognize product attributes from short queries, such as product type, brand, size, to better understand users’ preferences and intents. 1Our code is open-source and available at https:// github.com/amzn/x-metats 1. Ground-truth Labels [Mackie] [profx6v3] [6-channel] [mixer] [with] [usb] 2. Pseudo-Labels (Choice #1) [Mackie] [profx6v3 6-channel] [mixer with usb] 3. Pseudo-Labels (Choice #2) [Mackie] [profx6v3] [6-channel] [mixer] [with usb] Table 1: Ground-truth labels and noisy pseudo-labels for an English query NER example. We use colors to denote the entity type and use brackets to indicate the entity boundary. Entity labels: Brand, ProductLine, Size, ProductType, NonContent, Misc. Despite recent advances in deep learning models for sequence labeling (Huang et al., 2015; Raganato et al., 2017), they still rely on massive labeled data. Nonetheless, the sequence labeling tasks usually lie in the low-data regime due to costly and labor- intensive human annotation for token-level labels, especially for a variety of languages (Xie et al., 2018), as search engines or social networks usually cover a diverse range of countries and locales us- ing different languages. In this paper, we attempt to explore a unified multilingual sequence label- ing model with minimal supervision, where …

Content truncated to 500 words