Prediction or Comparison: Toward Interpretable Qualitative Reasoning

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 57–62 August 1–6, 2021. ©2021 Association for Computational Linguistics 57 Diagnosing Transformers in Task-Oriented Semantic Parsing Shrey Desai Ahmed Aly Facebook {shreyd, ahhegazy}@fb.com Abstract Modern task-oriented semantic parsing ap- proaches typically use seq2seq transformers to map textual utterances to semantic frames comprised of intents and slots. While these models are empirically strong, their specific strengths and weaknesses have largely re- mained unexplored. In this work, we study BART (Lewis et al., 2020) and XLM-R (Con- neau et al., 2020), two state-of-the-art parsers, across both monolingual and multilingual set- tings. Our experiments yield several key re- sults: transformer-based parsers struggle not only with disambiguating intents/slots, but sur- prisingly also with producing syntactically- valid frames. Though pre-training imbues transformers with syntactic inductive biases, we find the ambiguity of copying utterance spans into frames often leads to tree invalid- ity, indicating span extraction is a major bot- tleneck for current parsers. However, as a sil- ver lining, we show transformer-based parsers give sufficient indicators for whether a frame is likely to be correct or incorrect, making them easier to deploy in production settings. 1 Introduction Task-oriented semantic parsing—mapping textual utterances to semantic frames—is a critical compo- nent of modern conversational AI systems (Gupta et al., 2018; Aghajanyan et al., 2020). Recent methodology casts parsing as transduction, using seq2seq pre-trained transformers to produce lin- earized parse trees (Aghajanyan et al., 2020; Chen et al., 2020; Li et al., 2021); here, each frame token is either copied from the utterance or generated from an ontology. Compared to explicit grammar- based approaches (Gupta et al., 2018), this plug- and-play of transformers simplifies the learning objective and scales to multilingual settings, but the lack of provenance makes it challenging to un- derstand model behavior “under the hood.” In this work, we investigate the strengths and weaknesses of transformer-based semantic parsers Figure 1: Example decoupled semantic frame represen- tation (Aghajanyan et al., 2020) for the utterance Direc- tions to the Warriors game. and provide modeling directions based on data- driven insights. Specifically, we study BART (Lewis et al., 2020) and XLM-R (Conneau et al., 2020), two state-of-the-art conversational semantic parsers, on both monolingual (TOP/TOPv2; (Gupta et al., 2018; Chen et al., 2020)) and multilingual (MTOP; (Li et al., 2021)) datasets. The compo- sitionality of utterances in these datasets provide a strong testbed for resolving both complex syn- tactic structure and semantic ambiguity, mirroring the types of challenges our parsers are likely to encounter in practice. We design our experiments around three main questions. First, broadly speaking, what types of errors do transformer-based parsers make? We be- gin by annotating 500+ predicted frames across 6 languages and categorize them with fine-grained types. We find transformer-based parsers struggle not only with classification (i.e., disambiguating intents/slots) but also planning (i.e., switching be- tween copying/generating). Planning errors are more egregious: misplacing close brackets, for ex- ample, can violate tree constraints, rendering the entire frame unusable. Next, we investigate transformer-based parsers’ abilities to …

Content truncated to 500 words