Does Character-level Information Always Improve DRS-based Semantic Parsing?

Proceedings of the The 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 249–258 July 13-14, 2023 ©2023 Association for Computational Linguistics Does Character-level Information Always Improve DRS-based Semantic Parsing? Tomoya Kurosawa and Hitomi Yanaka The University of Tokyo {kurosawa-tomoya, hyanaka}@is.s.u-tokyo.ac.jp Abstract Even in the era of massive language models, it has been suggested that character-level rep- resentations improve the performance of neu- ral models. The state-of-the-art neural seman- tic parser for Discourse Representation Struc- tures uses character-level representations, im- proving performance in the four languages (i.e., English, German, Dutch, and Italian) in the Parallel Meaning Bank dataset. However, how and why character-level information im- proves the parser’s performance remains un- clear. This study provides an in-depth analysis of performance changes by order of character sequences. In the experiments, we compare F1-scores by shuffling the order and random- izing character sequences after testing the per- formance of character-level information. Our results indicate that incorporating character- level information does not improve the perfor- mance in English and German. In addition, we find that the parser is not sensitive to correct character order in Dutch. Nevertheless, perfor- mance improvements are observed when using character-level information. 1 Introduction Character-level information is sometimes help- ful in grasping the meanings of words for hu- mans. Previous studies have suggested that character-level information helps to improve the performance of neural models on various NLP tasks (Cherry et al., 2018; Zhang et al., 2015). In multilingual NLP systems, character-level in- formation contributes to performance improve- ments on Named Entity Recognition tasks (Lam- ple et al., 2016; Yu et al., 2018) and semantic pars- ing tasks (van Noord et al., 2020). However, due to the black-box nature of neural models, it is still unclear how and why character-level information contributes to model performance. The rapid developments of neural models have led to a growing interest in investigating the extent to which these models understand natu- ral language. Recent works have indicated that pre-trained language models are insensitive to word order on permuted English datasets on lan- guage understanding tasks (Sinha et al., 2021a,b; Pham et al., 2021; Hessel and Schofield, 2021). Meanwhile, other works have shown controver- sial results regarding inductive biases for word or- der (Abdou et al., 2022), especially in different lan- guages (Ravfogel et al., 2019; White and Cotterell, 2021). In this work, we explore the extent to which neu- ral models capture character order. By focusing on character order rather than word order, we present an in-depth analysis of the capacity of models to capture syntactic structures across languages. To analyze whether the importance of character or- der information differs across languages, we in- vestigate multilingual Discourse Representation Structure (DRS; Kamp and Reyle (1993)) pars- ing models. Van Noord et al. (2020) proposed an encoder-decoder DRS parsing model incorpo- rating character-level representations. The study concluded that incorporating character-level rep- resentations contributes to performance improve- ments of the model across languages. However, the underlying mechanism remains unclear. We examine the influence of character-level …

Content truncated to 500 words