Word Reordering for Zero-shot Cross-lingual Structured Prediction

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4109–4120 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4109 Word Reordering for Zero-shot Cross-lingual Structured Prediction Tao Ji∗, Yong Jiang, Tao Wang, Zhongqiang Huang, Fei Huang, Yuanbin Wu, Xiaoling Wang School of Computer Science and Technology, East China Normal University DAMO Academy, Alibaba Group {taoji@stu,ybwu@cs,xlwang@cs}.ecnu.edu.cn {yongjiang.jy,leeo.wangt,z.huang,f.huang}@alibaba-inc.com Abstract Adapting word order from one language to an- other is a key problem in cross-lingual struc- tured prediction. Current sentence encoders (e.g., RNN, Transformer with position em- beddings) are usually word order sensitive. Even with uniform word form representations (MUSE, mBERT), word order discrepancies may hurt the adaptation of models. This pa- per builds structured prediction models with bag-of-words inputs. It introduces a new re- ordering module to organize words following the source language order, which learns task- specific reordering strategies from a general- purpose order predictor model. Experiments on zero-shot cross-lingual dependency parsing, POS tagging, and morphological tagging show that our model can significantly improve tar- get language performances, especially for lan- guages that are distant from the source lan- guage. 1 1 Introduction Extracting linguistic structures from natural lan- guage usually relies on high quality human anno- tations. To handle low resource scenarios, efforts have been devoted to sharing resources among lan- guages and adapting from high resource models. One crucial step of these methods is how to unify in- put and output spaces over languages. For example, the universal dependency project (McDonald et al., 2013) constructs a universal output space for cross- lingual dependency parsers, and cross-lingual word representation learning algorithms helps aligning word forms of different languages (Conneau et al., 2017; Devlin et al., 2019). Beyond word form, word order is another im- portant factor in cross-lingual structured prediction ∗This work was conducted when Tao Ji was interning at Alibaba DAMO Academy. 1https://github.com/AntNLP/zero-shot-structured- prediction. (Wang and Eisner, 2018b): it is possible that sen- tences in two different language have similar parse trees, but their words are organized in different orders (e.g., SVO or SOV). To share annotations among them, we need to handle word order dis- crepancies carefully: if a model learned on the source language is tightly coupled with the source language word order, performances on target lan- guages could be hurt as their word order could be incompatible (Wang et al., 2019). On the other side, if one completely drops word order (e.g., bag-of- words), the source language (and target languages) performances might be poor as order-sensitive fea- tures could be essential. Trade-offs have been made by using weak word order information (e.g., rel- ative positions instead of absolute positions (Ah- mad et al., 2019a)), but we still want to seek better adaptation of word order without sacrificing source language performances. In this work, we integrate new reordering mod- ules to help cross-lingual structured prediction. Given a bag-of-words from the target language, the module tries to reorder them to best resemble a source language sentence. The structured predic- tion part then receives inputs …

Content truncated to 500 words