On the Relation between Syntactic Divergence and Zero-Shot Performance

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4803–4817 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 4803 On the Relation between Syntactic Divergence and Zero-Shot Performance Ofir Arviv†∗ Dmitry Nikolaev‡∗ Taelin Karidi† Omri Abend† †Hebrew University of Jerusalem ‡Institute for Natural Language Processing, University of Stuttgart {ofir.arviv|taelin.karidi|omri.abend}@mail.huji.ac.il dnikolaev@fastmail.com Abstract We explore the link between the extent to which syntactic relations are preserved in translation and the ease of correctly construct- ing a parse tree in a zero-shot setting. While previous work suggests such a relation, it tends to focus on the macro level and not on the level of individual edges—a gap we aim to address. As a test case, we take the transfer of Universal Dependencies (UD) parsing from English to a diverse set of languages and conduct two sets of experiments. In one, we analyze zero-shot performance based on the extent to which En- glish source edges are preserved in translation. In another, we apply three linguistically mo- tivated transformations to UD, creating more cross-lingually stable versions of it, and assess their zero-shot parsability. In order to compare parsing performance across different schemes, we perform extrinsic evaluation on the down- stream task of cross-lingual relation extraction (RE) using a subset of a popular English RE benchmark translated to Russian and Korean.1 In both sets of experiments, our results suggest a strong relation between cross-lingual stabil- ity and zero-shot parsing performance. 1 Introduction Recent progress in cross-lingual transfer methods, such as multi-lingual embeddings (Devlin et al., 2018; Mulcaire et al., 2019), enabled significant advances in a wide range of cross-lingual natural language processing tasks. The transferred models, however, are not uniformly effective in addressing languages with different grammatical structures, and little is known about the settings under which cross-lingual transfer is more or less effective. A prominent way of facilitating transfer of gram- matical knowledge from one language to another ∗Equal contribution. Dmitry Nikolaev’s work was under- taken during his post-doc at Stockholm University. 1All resources are available at https://github. com/OfirArviv/translated_tacred and https: //github.com/OfirArviv/improving-ud Japanese company amod root sharikat yabania company Japanese amod root Nihon no kaisha Japan of company nmod case root Figure 1: A UD parse of the English phrase Japanese company (left) and of its translations to Arabic (mid- dle) and Japanese (right). Word order aside, the UD tree of the Arabic translation is identical to the English one while that of the Japanese translation is different. The amod edge in this phrase is stable in translation to Arabic but unstable in translation to Japanese. is through the use of cross-lingual symbolic rep- resentation schemes (Chen et al., 2017, 2018; Bugliarello and Okazaki, 2020). Many advances have been made in this area in recent years, most notably the development and quick adoption of Universal Dependencies (UD; Nivre et al., 2016), a cross-lingually applicable scheme that has become the de facto standard for syntactic annotation. While these schemes abstract away from many syntactic differences, there is still considerable vari- ability in …

Content truncated to 500 words