Annotations Matter: Leveraging Multi-task Learning to Parse UD and SUD

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3467–3481 August 1–6, 2021. ©2021 Association for Computational Linguistics 3467 Annotations Matter: Leveraging Multi-task Learning to Parse UD and SUD Zeeshan Ali Sayyed Indiana University Department of Computer Science zasayyed@iu.edu Daniel Dakota Uppsala University Department of Linguistics ddakota@lingfil.uu.se Abstract Using multiple treebanks to improve parsing performance has shown positive results. How- ever, to what extent similar, yet competing an- notation decisions play in parser behavior is unclear. We investigate this within a multi-task learning (MTL) dependency parser setup on two parallel treebanks, UD and SUD, which, while possessing similar annotation schemes, differ in specific linguistic annotation prefer- ences. We perform a set of experiments with different MTL architectural choices, compar- ing performance across various input embed- dings. We find languages tend to pattern in loose typological associations, but generally the performance within an MTL setting is lower than single model baseline parsers for each annotation scheme. The main contribut- ing factor seems to be the competing syntactic annotation information shared between tree- banks in an MTL setting, which is shown in experiments against differently annotated tree- banks. This suggests that the impact of how the signal is encoded for annotations and its in- fluence on possible negative transfer is more important than that of the input embeddings in an MTL setting. 1 Introduction Multi-task learning (MTL; Caruana, 1997) has shown promise in various NLP tasks such as se- mantic dependency parsing (Peng et al., 2017; Her- shcovich et al., 2018; Kurita and Søgaard, 2019), machine translation (Dong et al., 2015) and muli- tiword expression detection (Taslimipoor et al., 2019). MTL inherently is designed to share informa- tion between tasks, which has helped various NLP components (Collobert and Weston, 2008). One active research question however is what informa- tion in specific tasks should be shared, as well was what indicators can be used to predetermine the cost-benefit trade-offs of MTL for a given appli- cation. Findings have shown that label distribu- tions (Mart´ınez Alonso and Plank, 2017), data sizes (Bollmann et al., 2018) and single task loss curves (Bingel and Søgaard, 2017) have all been respective indicators for MTL performance. Different tasks, data sizes, and settings can all show different rel- ative performance gains (Adouane and Bernardy, 2020). Thus, it is still an open question under which circumstance MTL can be used to achieve max performance boosts over a single task system. In syntactic parsing, learning a closely related task (e.g. POS tagging) in a joint paradigm bene- fits overall performance (Bohnet and Nivre, 2012; Zhang and Weiss, 2016), and work has also ex- ploited MTL by leveraging two or more treebanks against each other (see section 2). We often assume simply increasing data and the sharing of syntac- tic information will inherently benefit all parsers, but this assumes that all syntactic sharing, specif- ically all annotation sharing, is positive and com- plementary. However, annotation decisions have been shown to favor parsing preferences (Rosa, 2015; Rehbein et al., 2017; Kohita et …

Content truncated to 500 words