Improving Low-Resource Cross-lingual Parsing with Expected Statistic Regularization

Improving Low-Resource Cross-lingual Parsing with Expected Statistic Regularization Thomas Effland Columbia University, USA teffland@cs.columbia.edu Michael Collins Google Research, USA mjcollins@google.com Abstract We present Expected Statistic Regulariza tion (ESR), a novel regularization technique that utilizes low-order multi-task structural sta- tistics to shape model distributions for semi- supervised learning on low-resource datasets. We study ESR in the context of cross-lingual transfer for syntactic analysis (POS tagging and labeled dependency parsing) and present several classes of low-order statistic functions that bear on model behavior. Experimentally, we evaluate the proposed statistics with ESR for unsupervised transfer on 5 diverse target languages and show that all statistics, when estimated accurately, yield improvements to both POS and LAS, with the best statistic improving POS by +7.0 and LAS by +8.5 on average. We also present semi-supervised transfer and learning curve experiments that show ESR provides significant gains over strong cross-lingual-transfer-plus-fine-tuning baselines for modest amounts of label data. These results indicate that ESR is a promis- ing and complementary approach to model- transfer approaches for cross-lingual parsing.1 1 Introduction In recent years, great strides have been made on linguistic analysis for low-resource languages. These gains are largely attributable to transfer approaches from (1) massive pretrained multilin- gual language model (PLM) encoders (Devlin et al., 2019; Liu et al., 2019b); (2) multi-task training across related syntactic analysis tasks (Kondratyuk and Straka, 2019); and (3) multilin- gual training on diverse high-resource languages (Wu and Dredze, 2019; Ahmad et al., 2019; Kondratyuk and Straka, 2019). Combined, these approaches have been shown to be particularly effective for cross-lingual syntactic analysis, as shown by UDify (Kondratyuk and Straka, 2019). 1We have published for our implementation and exper- iments at https://github.com/teffland/expected -statistic-regularization. However, even with the improvements brought about by these techniques, transferred models still make syntactically implausible predictions on low-resource languages, and these error rates in- crease dramatically as the target languages be- come more distant from the source languages (He et al., 2019; Meng et al., 2019). In par- ticular, transferred models often fail to match many low-order statistics concerning the typol- ogy of the task structures. We hypothesize that enforcing regularity with respect to estimates of these structural statistics—effectively using them as weak supervision—is complementary to cur- rent transfer approaches for low-resource cross- lingual parsing. To this end, we introduce Expected Statistic Regularization (ESR), a novel differentiable loss that regularizes models on unlabeled target data- sets by minimizing deviation of descriptive statis- tics of model behavior from target values. The class of descriptive statistics usable by ESR are expressive and powerful. For example, they may describe cross-task interactions, encouraging the model to obey structural patterns that are not explicitly tractable in the model factorization. Additionally, the statistics may be derived from constraints dictated by the task formalism itself (such as ruling out invalid substructures) or by numerical parameters that are specific to the target dataset distribution (such as relative sub- structure frequencies). In the latter case, we also contribute a method for selecting those parame- ters using small amounts of …

Content truncated to 500 words