Multiple Tasks Integration: Tagging, Syntactic and Semantic Parsing as a Single Task

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 783–794 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 783 Multiple Tasks Integration: Tagging, Syntactic and Semantic Parsing as a Single Task Timoth´ee Bernard∗ LLF, Universit´e de Paris, France timothee.bernard@ens-lyon.org Abstract Departing from both sequential pipelines and monotask systems, we propose Multiple Tasks Integration (MTI), a multitask paradigm or- thogonal to weight sharing. The essence of MTI is to process the input iteratively but con- currently at multiple levels of analysis, where each decision is based on all of the struc- tures that are already inferred and free from usual ordering constraints. We illustrate MTI with a system that performs part-of-speech tag- ging, syntactic dependency parsing and seman- tic dependency parsing. We observe that both the use of reinforcement learning and the re- lease from sequential constraints are benefi- cial to the quality of the syntactic and seman- tic parses. We also observe that our model adopts an easy-first strategy that consists, on average, of predicting shorter dependencies be- fore longer ones, but that syntax is not always tackled before semantics. 1 Introduction Historically, Natural Language Processing (NLP) systems have generally been built as sequential pipelines, where each module adds another layer of annotation, in order of (supposed) increasing com- plexity. Progress in neural networks has, however, led to the development of state-of-the-art systems that completely bypass intermediate levels of anal- ysis that were previously considered essential. For example, the system of Zhou and Xu (2015) per- form Semantic Role Labeling (SRL; Carreras and M`arquez, 2005) without referring to any explicit (morpho)syntactic information. A well known problem of sequential systems is error propagation, which happens when an incor- rect prediction at some point in the process leads to more incorrect predictions at a later stage. In tradi- tional pipelines, one of the roots of error propaga- ∗Work done while at AIST (Japan). tion lies in the fact that they feature a unidirectional flow of annotation between their different stages. End-to-end systems with no intermediate level of analysis, in contrast, are protected against one form of error propagation. Since such systems are not contingent on possibly faulty levels of analy- sis, they are free from the interference they could cause. However, the absence of any intermediate decisions — symbolic traces of the system’s com- putation — raises questions about generalisation ability and interpretability (Lipton, 2018). What is more, by reducing the information in the prediction and training signals at their disposal, they might not leverage the full power of the neural networks they are usually based on. Indeed, in their reintro- duction of syntactic dependencies to the training process, Strubell et al. (2018) managed to develop a state-of-the-art system for SRL that at the same time computes good-quality syntactic parses. Departing from these two kinds of architectures, we propose Multiple Tasks Integration (MTI). The principles of MTI are (i) to let the system take ac- tions pertaining to different levels of analysis with- …

Content truncated to 500 words