Transformer-based Models for Long-Form Document Matching: Challenges and Empirical Analysis

Findings of the Association for Computational Linguistics: EACL 2023, pages 227–238 May 2-6, 2023 ©2023 Association for Computational Linguistics Translate First Reorder Later: Leveraging Monotonicity in Semantic Parsing Francesco Cazzaro∗ Davide Locatelli∗ Ariadna Quattoni Universitat Politècnica de Catalunya name.lastname@upc.edu Xavier Carreras dMetrics xavier.carreras@dmetrics.com Abstract Prior work in semantic parsing has shown that conventional seq2seq models fail at composi- tional generalization tasks. This limitation led to a resurgence of methods that model align- ments between sentences and their correspond- ing meaning representations, either implicitly through latent variables or explicitly by tak- ing advantage of alignment annotations. We take the second direction and propose TPOL, a two-step approach that first translates input sen- tences monotonically and then reorders them to obtain the correct output. This is achieved with a modular framework comprising a Trans- lator and a Reorderer component. We test our approach on two popular semantic pars- ing datasets. Our experiments show that by means of the monotonic translations, TPOL can learn reliable lexico-logical patterns from aligned data, significantly improving composi- tional generalization both over conventional seq2seq models, as well as over other ap- proaches that exploit gold alignments. Our code is publicly available at https://github. com/interact-erc/TPol.git 1 Introduction The goal of a semantic parser is to map natural language sentences (NLs) into meaning represen- tations (MRs). Most current semantic parsers are based on deep sequence-to-sequence (seq2seq) ap- proaches and presume that it is unnecessary to model token alignments between NLs and MRs because the attention mechanism can automatically learn the correspondences (Dong and Lapata, 2016; Jia and Liang, 2016). However, recent work has shown that such seq2seq models find compositional generalization challenging, i.e., they struggle to pre- dict unseen structures made up of components ob- served at training (Lake and Baroni, 2018; Finegan- Dollak et al., 2018). ∗Equal contribution Figure 1: Examples from the GEOALIGNED dataset. (a) is a monotonic alignment, (b) is non-monotonic. This limitation motivated the resurgence of ap- proaches that model alignments between NL sen- tences and their corresponding MRs more simi- larly to classical grammar and translation-based parsers (Herzig and Berant, 2021). Alignments can be modeled either implicitly through latent vari- ables (Wang et al., 2021), or explicitly by leverag- ing gold alignment annotations (Shi et al., 2020; Liu et al., 2021a). We take the second direction and exploit a recently released multilingual dataset for semantic parsing annotated with word align- ments: GEOALIGNED (Locatelli and Quattoni, 2022), which augments the popular GEO bench- mark (Zelle and Mooney, 1996). Figure 1 shows some examples of the annota- tions provided. One key observation is that a signif- icant percentage of the alignments are monotonic, i.e., they require no reordering of the target MR (Figure 1a), as opposed to non-monotonic align- ments (Figure 1b). This suggests that learning reli- able lexico-logical translation patterns from aligned data should be possible. If there are simple patterns, shouldn’t an ideal model be able to exploit them? With this in mind, we propose TPOL, a Two- step Parsing approach that leverages monotonic translations. …

Content truncated to 500 words