Risk Minimization for Zero-shot Sequence Labeling

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4909–4920 August 1–6, 2021. ©2021 Association for Computational Linguistics 4909 Risk Minimization for Zero-shot Sequence Labeling Zechuan Hu⋄‡, Yong Jiang†∗, Nguyen Bach†, Tao Wang†, Zhongqiang Huang†, Fei Huang†, Kewei Tu⋄∗ ⋄School of Information Science and Technology, ShanghaiTech University ⋄Shanghai Engineering Research Center of Intelligent Vision and Imaging ⋄Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences ⋄University of Chinese Academy of Sciences †DAMO Academy, Alibaba Group {huzch,tukw}@shanghaitech.edu.cn, yongjiang.jy@alibaba-inc.com {nguyen.bach,leeo.wangt,z.huang,f.huang}@alibaba-inc.com Abstract Zero-shot sequence labeling aims to build a sequence labeler without human-annotated datasets. One straightforward approach is uti- lizing existing systems (source models) to gen- erate pseudo-labeled datasets and train a target sequence labeler accordingly. However, due to the gap between the source and the target languages/domains, this approach may fail to recover the true labels. In this paper, we pro- pose a novel unified framework for zero-shot sequence labeling with minimum risk train- ing and design a new decomposable risk func- tion that models the relations between the pre- dicted labels from the source models and the true labels. By making the risk function train- able, we draw a connection between minimum risk training and latent variable model learning. We propose a unified learning algorithm based on the expectation maximization (EM) algo- rithm. We extensively evaluate our proposed approaches on cross-lingual/domain sequence labeling tasks over twenty-one datasets. The results show that our approaches outperform state-of-the-art baseline systems. 1 Introduction Sequence labeling is an important task in natural language processing. It has many applications such as Part-of-Speech Tagging (POS) (DeRose, 1988; Toutanova et al., 2003) and Named Entity Recogni- tion (NER) (Ratinov and Roth, 2009; Ritter et al., 2011; Lample et al., 2016; Ma and Hovy, 2016; Hu et al., 2020). Approaches to sequence labeling are mostly based on supervised learning, which relies heavily on labeled data. However, the labeled data is generally expensive and hard to obtain (for low- resource languages/domains), which means that these supervised learning approaches fail in many cases. ∗Corresponding authors. ‡Work was done when Zechuan Hu was interning at Alibaba DAMO Academy. Learning knowledge from imperfect predictions from other rich-resource sources (such as cross- lingual, cross-domain transfer) (Yarowsky and Ngai, 2001; Guo et al., 2018; Huang et al., 2019; Hu et al., 2021) is a feasible and efficient way to tackle the low-resource problem. It transfers knowledge from rich-resource languages/domains to low-resource ones. One typical approach to this problem is utilizing existing systems to provide pre- dicted results for the zero-shot datasets. However, due to the gap between the source and the target languages/domains, this approach may fail to re- cover the true labels. Several previous approaches try to alleviate this problem by relying heavily on cross-lingual information (e.g., parallel text (Wang and Manning, 2014; Ni et al., 2017)), labeled data in source languages (Chen et al., 2019), and prior domain knowledge (Yang and Eisenstein, 2015) for different kinds of zero-shot …

Content truncated to 500 words