Automatic Correction of Syntactic Dependency Annotation Differences

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 7106–7112 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 7106 Automatic Correction of Syntactic Dependency Annotation Differences Andrew Zupon, Andrew Carnie, Michael Hammond, Mihai Surdeanu University of Arizona Tucson, AZ, USA {zupon, carnie, hammond, msurdeanu}@email.arizona.edu Abstract Annotation inconsistencies between data sets can cause problems for low-resource NLP, where noisy or inconsistent data cannot be as easily replaced compared with resource-rich languages. In this paper, we propose a method for automatically detecting annotation mismatches between dependency parsing corpora, as well as three related methods for automatically converting the mismatches. All three methods rely on comparing an unseen example in a new corpus with similar examples in an existing corpus. These three methods include a simple lexical replacement using the most frequent tag of the example in the existing corpus, a GloVe embedding-based replacement that considers a wider pool of examples, and a BERT embedding-based replacement that uses contextualized embeddings to provide examples fine-tuned to our specific data. We then evaluate these conversions by retraining two dependency parsers—Stanza (Qi et al., 2020) and Parsing as Tagging (PaT) (Vacareanu et al., 2020)—on the converted and unconverted data. We find that applying our conversions yields significantly better performance in many cases. Some differences observed between the two parsers are observed. Stanza has a more complex architecture with a quadratic algorithm, so it takes longer to train, but it can generalize better with less data. The PaT parser has a simpler architecture with a linear algorithm, speeding up training time but requiring more training data to reach comparable or better performance. Keywords: dependency parsing, low-resource, data augmentation, data quality, data cleaning 1. Introduction Examples found in one data set that are not found in another can represent either unseen examples or inconsistencies in annotation. These annotation differences may not be a ma- jor problem for resource-rich languages—if a text or data set contains errors or inconsistencies, the researcher can re- move the faulty data and replace it with valid data—but this is not true for low-resource languages. Low-resource lan- guages have limited amounts of annotated data available for NLP tasks, so any data that is compromised by errors or inconsistent annotations cannot as easily be replaced. This project looks at reducing annotation differences be- tween two different corpora in order to augment training data in a more informed, consistent way. If successful, this automatic conversion can aid research on low-resource NLP by embiggening the pool of clean, usable data avail- able to researchers. The contributions of this work are: (1) We propose a simple approach for automatically iden- tifying mismatches between two Universal Dependencies (UD) dependency parsing data sets. First, we identify all the tokens in a head-dependent relation in each data set, along with the specific relations they occur with. Next, we identify the relations that occur between a head-dependent word pair in the second data set but not the first. This results in a set …

Content truncated to 500 words