Measuring Alignment Bias in Neural Seq2seq Semantic Parsers

Proceedings of the 11th Joint Conference on Lexical and Computational Semantics, pages 200 - 207 July 14-15, 2022 ©2022 Association for Computational Linguistics Measuring Alignment Bias in Neural Seq2Seq Semantic Parsers Davide Locatelli Universitat Polit`ecnica de Catalunya Campus Nord, Barcelona davide.locatelli@upc.edu Ariadna Quattoni Universitat Polit`ecnica de Catalunya Campus Nord, Barcelona aquattoni@cs.upc.edu Abstract Prior to deep learning the semantic pars- ing community has been interested in under- standing and modeling the range of possi- ble word alignments between natural language sentences and their corresponding meaning representations. Sequence-to-sequence mod- els changed the research landscape suggesting that we no longer need to worry about align- ments since they can be learned automatically by means of an attention mechanism. More recently, researchers have started to question such premise. In this work we investigate whether seq2seq models can handle both sim- ple and complex alignments. To answer this question we augment the popular GEO seman- tic parsing dataset with alignment annotations and create GEO-ALIGNED. We then study the performance of standard seq2seq models on the examples that can be aligned monotoni- cally versus examples that require more com- plex alignments. Our empirical study shows that performance is significantly better over monotonic alignments. 1 1 Introduction In semantic parsing, the goal is to map natural language (NL) sentences into machine-readable meaning representations (MR) which allow for automated reasoning. For example, consider the following pair: NL : What is the population of Georgia ? MR : answer (population (state (georgia) ) ) Prior to deep learning models, a popular ap- proach was to learn a grammar-based parser that explicitly models alignments between the NL and MR sequences (Wong and Mooney, 2006; Zettle- moyer and Collins, 2005, 2007; Lu et al., 2008; 1The code and data is publicly available at https:// github.com/interact-erc/geo-aligned Kwiatkowksi et al., 2010; Kwiatkowski et al., 2011). The emergence of sequence-to-sequence (seq2seq) semantic parsers with attention mecha- nisms changed the research landscape: one of the initial premises of seq2seq models is that align- ments no longer need to be explicitly modeled because the attention mechanisms will automat- ically learn them (Bahdanau et al., 2015). More re- cently, researchers started to question such premise, having observed that seq2seq models fail to make proper generalizations on out-of-distribution test sets on which traditional grammar-based models excel (Liu et al., 2020, 2021; Wang et al., 2021). In this paper we follow this line of research and ask the questions: Can standard seq2seq models handle arbitrary alignments? And if not, what kind of alignment bias do they have? To answer these questions, we augment the GEO semantic parsing benchmark (Zelle and Mooney, 1996) with align- ment annotations and create GEO-ALIGNED. We then compare the performance of seq2seq models on examples that can be easily aligned with simple monotonic alignments to the performance of these models on examples that require word reordering. Our empirical study shows that seq2seq parsers perform significantly better over examples that can be monotonically aligned. In other words, the flex- ibility of not having to explicitly model …

Content truncated to 500 words