Character-level Representations Improve DRS-based Semantic Parsing Even in the Age of BERT

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4587–4603, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4587 Character-level Representations Improve DRS-based Semantic Parsing Even in the Age of BERT Rik van Noord CLCG University of Groningen The Netherlands rikvannoord@gmail.com Antonio Toral CLCG University of Groningen The Netherlands a.toral.ruiz@rug.nl Johan Bos CLCG University of Groningen The Netherlands johan.bos@rug.nl Abstract We combine character-level and contextual language model representations to improve performance on Discourse Representation Structure parsing. Character representations can easily be added in a sequence-to-sequence model in either one encoder or as a fully separate encoder, with improvements that are robust to different language models, lan- guages and data sets. For English, these im- provements are larger than adding individual sources of linguistic information or adding non-contextual embeddings. A new method of analysis based on semantic tags demon- strates that the character-level representations improve performance across a subset of se- lected semantic phenomena. 1 Introduction Character-level models have obtained impressive performance on a number of NLP tasks, rang- ing from the classic POS-tagging (Santos and Zadrozny, 2014) to complex tasks such as Dis- course Representation Structure (DRS) parsing (van Noord et al., 2018b). However, this was before the large pretrained language models (Peters et al., 2018; Devlin et al., 2019) took over the field, with the consequence that for most NLP tasks, state-of- the-art performance is now obtained by fine-tuning on one of these models (e.g., Conneau et al., 2020). Does this mean that, despite a long tradition of being used in language-related tasks (see Sec- tion 2.1), character-level representations are no longer useful? We try to answer this question by looking at semantic parsing, specifically DRS parsing (Abzianidze et al., 2017; van Noord et al., 2018a). We aim to answer the following research questions: 1. Do pretrained language models (LMs) outper- form character-level models for DRS parsing? 2. Can character and LM representations be com- bined to improve performance, and if so, what is the best method of combining them? 3. How do these improvements compare to adding linguistic features? 4. Are the improvements robust across different pretrained language models, languages, and data sets? 5. On what type of sentences do character-level representations specifically help? Why semantic parsing? Semantic parsing is the task of automatically mapping natural language ut- terances to interpretable meaning representations. The produced meaning representations can then potentially be used to improve downstream NLP applications (e.g., Issa et al., 2018; Song et al., 2019; Mihaylov and Frank, 2019), though the in- troduction of large pretrained language models has shown that explicit formal meaning representations might not be a necessary component to achieve high accuracy. However, it is now known that these models lack reasoning capabilities, often simply exploiting statistical artifacts in the data sets, in- stead of actually understanding language (Niven and Kao, 2019; McCoy et al., 2019). Moreover, Ettinger (2020) found that the popular BERT model (Devlin et al., 2019) completely failed to acquire a general understanding of negation. Related, Bender …

Content truncated to 500 words