Dialogue Graph Modeling for Conversational Machine Reading

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 314–329 August 1–6, 2021. ©2021 Association for Computational Linguistics 314 Empirical Error Modeling Improves Robustness of Noisy Neural Sequence Labeling Marcin Namysl1,2, Sven Behnke1,2, and Joachim K¨ohler1 1 Fraunhofer IAIS, Sankt Augustin, Germany 2 Autonomous Intelligent Systems, University of Bonn, Germany {Marcin.Namysl,Sven.Behnke,Joachim.Koehler}@iais.fraunhofer.de Abstract Despite recent advances, standard sequence labeling systems often fail when processing noisy user-generated text or consuming the output of an Optical Character Recognition (OCR) process. In this paper, we improve the noise-aware training method by proposing an empirical error generation approach that em- ploys a sequence-to-sequence model trained to perform translation from error-free to erro- neous text. Using an OCR engine, we gener- ated a large parallel text corpus for training and produced several real-world noisy sequence la- beling benchmarks for evaluation. Moreover, to overcome the data sparsity problem that ex- acerbates in the case of imperfect textual in- put, we learned noisy language model-based embeddings. Our approach outperformed the baseline noise generation and error correction techniques on the erroneous sequence labeling data sets. To facilitate future research on ro- bustness, we make our code, embeddings, and data conversion scripts publicly available. 1 Introduction Deep learning models have already surpassed human-level performance in many Natural Lan- guage Processing (NLP) tasks1. Sequence labeling systems have also reached extremely high accu- racy (Akbik et al., 2019; Heinzerling and Strube, 2019). Still, NLP models often fail in scenarios, where non-standard text is given as input (Heigold et al., 2018; Belinkov and Bisk, 2018). NLP algorithms are predominantly trained on error-free textual data but are also employed to pro- cess user-generated text (Baldwin et al., 2013; Der- czynski et al., 2013) or consume the output of prior Optical Character Recognition (OCR) or Auto- matic Speech Recognition (ASR) processes (Miller et al., 2000). Errors that occur in any upstream 1GLUE benchmark (Wang et al., 2018a): https:// gluebenchmark.com/leaderboard Training Loss Sailing is a passion. Sailing 1s o passion. Seq2Seq Model Figure 1: Our modification of the NAT approach (green boxes). We propose a learnable seq2seq-based error generator and re-train FLAIR embeddings using noisy text to improve the accuracy of noisy neural sequence labeling. Γ is a process that induces noise to the input x producing erroneous ˜x. E(x) is an embedding matrix. F(x) is a sequence labeling model. e(x) and e(˜x) are the embeddings of x and ˜x, respectively. y(x) and y(˜x) are the outputs of the model for x and ˜x, respectively. component of an NLP system deteriorate the accu- racy of the target downstream task (Alex and Burns, 2014). In this paper, we focus on the problem of per- forming sequence labeling on the text produced by an OCR engine. Moreover, we study the trans- ferability of the methods learned to model OCR noise to the distribution of the human-generated errors. Both misrecognized and mistyped text pose a challenge for the standard models trained using error-free data (Namysl et al., 2020). We make the following contributions (Figure 1): • We propose …

Content truncated to 500 words