uChecker: Masked Pretrained Language Models as Unsupervised Chinese Spelling Checkers

Proceedings of the 29th International Conference on Computational Linguistics, pages 310–322 October 12–17, 2022. 310 Improving Top-K Decoding for Non-Autoregressive Semantic Parsing via Intent Conditioning Geunseob (GS) Oh, Rahul Goel, Chris Hidey, Shachi Paul, Aditya Gupta, Pararth Shah, Rushin Shah Google {ohgs,goelrahul}@google.com Abstract Semantic parsing (SP) is a core component of modern virtual assistants like Google As- sistant and Amazon Alexa. While sequence- to-sequence based auto-regressive (AR) ap- proaches are common for conversational SP, recent studies (Shrivastava et al., 2021) em- ploy non-autoregressive (NAR) decoders and reduce inference latency while maintaining competitive parsing quality. However, a ma- jor drawback of NAR decoders is the difficulty of generating top-k (i.e., k-best) outputs with approaches such as beam search. To address this challenge, we propose a novel NAR seman- tic parser that introduces intent conditioning on the decoder. Inspired by the traditional in- tent and slot tagging parsers, we decouple the top-level intent prediction from the rest of a parse. As the top-level intent largely governs the syntax and semantics of a parse, the in- tent conditioning allows the model to better control beam search and improves the quality and diversity of top-k outputs. We introduce a hybrid teacher-forcing approach to avoid train- ing and inference mismatch. We evaluate the proposed NAR on conversational SP datasets, TOP & TOPv2. Like the existing NAR mod- els, we maintain the O(1) decoding time com- plexity while generating more diverse outputs and improving top-3 exact match (EM) by 2.4 points. In comparison with AR models, our model speeds up beam search inference by 6.7 times on CPU with competitive top-k EM. 1 Introduction Neural sequence models are widely used for the task of conversational semantic parsing, which converts natural language utterances to machine- understandable meaning representations. Recent approaches (Chen et al., 2020; Rongali et al., 2020; Lialin et al., 2020; Aghajanyan et al., 2020a; Shri- vastava et al., 2021) combine Transformer-based se- quence models (Vaswani et al., 2017; Devlin et al., 2019) and Pointer Generator Networks (Vinyals Query: what is going on this weekend? Label: [in:get_event [sl:date_time this weekend ] ] (Failure 1: repeated tokens) [in:get_event [sl:date_time this weekend ] [sl:date_time this weekend ] ] (Parse with the repeated slot sl:data_time and leaf nodes this and weekend) (Failure 2: invalid syntax) [in:get_event [sl:date_time (Incomplete parse; missing ‘] ]’) Table 1: Examples of possible failures from the limited beam search of the existing NAR semantic parsers. et al., 2015; See et al., 2017). A vast majority of semantic parsing approaches (Gupta et al., 2018; Chen et al., 2020; Rongali et al., 2020; Lialin et al., 2020; Aghajanyan et al., 2020a; Yin et al., 2021) employ autoregressive (AR) decoders to generate structured output frames for the quality of output parses. During the AR decoding, output tokens are generated sequentially, conditioned on all pre- viously generated tokens. As a result, the decoding time (i.e. inference latency) increases linearly with the decoding length. This not only limits the ca- pacity to accommodate larger language models but may also degrade …

Content truncated to 500 words