Hierarchical Low-Rank Tensors for Multilingual Transfer Parsing

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1857–1867, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Hierarchical Low-Rank Tensors for Multilingual Transfer Parsing Yuan Zhang CSAIL, MIT yuanzh@csail.mit.edu Regina Barzilay CSAIL, MIT regina@csail.mit.edu Abstract Accurate multilingual transfer parsing typ- ically relies on careful feature engineer- ing. In this paper, we propose a hierar- chical tensor-based approach for this task. This approach induces a compact feature representation by combining atomic fea- tures. However, unlike traditional tensor models, it enables us to incorporate prior knowledge about desired feature interac- tions, eliminating invalid feature combi- nations. To this end, we use a hierar- chical structure that uses intermediate em- beddings to capture desired feature com- binations. Algebraically, this hierarchi- cal tensor is equivalent to the sum of tra- ditional tensors with shared components, and thus can be effectively trained with standard online algorithms. In both unsu- pervised and semi-supervised transfer sce- narios, our hierarchical tensor consistently improves UAS and LAS over state-of-the- art multilingual transfer parsers and the traditional tensor model across 10 differ- ent languages.1 1 Introduction The goal of multilingual syntactic transfer is to parse a resource lean target language utilizing an- notations available in other languages. Recent ap- proaches have demonstrated that such transfer is possible, even in the absence of parallel data. As a main source of guidance, these methods rely on the commonalities in dependency structures across languages. These commonalities manifest them- selves through a broad and diverse set of indi- cators, ranging from standard arc features used in monolingual parsers to typological properties 1The source code is available at https://github. com/yuanzh/TensorTransfer. Verb-subject: {head POS=VERB} ∧ {modifier POS=NOUN} ∧{label=subj} ∧ {direction=LEFT}∧ {82A=SV} Noun-adjective: {head POS=NOUN} ∧ {modifier POS=ADJ}∧ {direction=LEFT} ∧ {87A=Adj-Noun} Table 1: Example verb-subject and noun-adjective typological features. 82A and 87A denote the WALS (Dryer et al., 2005) feature codes for verb- subject and noun-adjective ordering preferences. needed to guide cross-lingual sharing (e.g., verb- subject ordering preference). In fact, careful fea- ture engineering has been shown to play a cru- cial role in state-of-the-art multilingual transfer parsers (T¨ackstr¨om et al., 2013). Tensor-based models are an appealing alterna- tive to manual feature design. These models auto- matically induce a compact feature representation by factorizing a tensor constructed from atomic features (e.g., the head POS). No prior knowledge about feature interactions is assumed. As a result, the model considers all possible combinations of atomic features, and addresses the parameter ex- plosion problem via a low-rank assumption. In the multilingual transfer setting, however, we have some prior knowledge about legitimate fea- ture combinations. Consider for instance a ty- pological feature that encodes verb-subject pref- erences. As Table 1 shows, it is expressed as a conjunction of five atomic features. Ideally, we would like to treat this composition as a single non-decomposable feature. However, the tradi- tional tensor model decomposes this feature into multiple dimensions, and considers various com- binations of these features as well as their indi- vidual interactions with other features. …

Content truncated to 500 words