READIN: A Chinese Multi-Task Benchmark with Realistic and Diverse Input Noises

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8272–8285 July 9-14, 2023 ©2023 Association for Computational Linguistics READIN: A Chinese Multi-Task Benchmark with Realistic and Diverse Input Noises Chenglei Si1,2∗ Zhengyan Zhang1∗ Yingfa Chen1∗ Xiaozhi Wang1 Zhiyuan Liu1,3† Maosong Sun1,3† 1 NLP Group, DCST, IAI, BNRIST, Tsinghua University, Beijing 2 University of Maryland, College Park, MD, USA 3 International Innovation Center of Tsinghua University, Shanghai clsi@terpmail.umd.edu, zy-z19@mails.tsinghua.edu.cn Abstract For many real-world applications, the user- generated inputs usually contain various noises due to speech recognition errors caused by lin- guistic variations1 or typographical errors (ty- pos). Thus, it is crucial to test model perfor- mance on data with realistic input noises to ensure robustness and fairness. However, little study has been done to construct such bench- marks for Chinese, where various language- specific input noises happen in the real world. In order to fill this important gap, we con- struct READIN: a Chinese multi-task bench- mark with REalistic And Diverse Input Noises. READIN contains four diverse tasks and re- quests annotators to re-enter the original test data with two commonly used Chinese in- put methods: Pinyin input and speech input. We designed our annotation pipeline to maxi- mize diversity, for example by instructing the annotators to use diverse input method edi- tors (IMEs) for keyboard noises and recruit- ing speakers from diverse dialectical groups for speech noises. We experiment with a se- ries of strong pretrained language models as well as robust training methods, we find that these models often suffer significant perfor- mance drops on READIN even with robustness methods like data augmentation. As the first large-scale attempt in creating a benchmark with noises geared towards user-generated in- puts, we believe that READIN serves as an im- portant complement to existing Chinese NLP benchmarks. The source code and dataset can be obtained from https://github.com/ thunlp/READIN. 1 Introduction User-generated inputs in real-world applications of- ten contain noises where wrong characters or words 1Note that linguistic variations themselves are not noises or errors, but they can lead to noises in the data processing for example due to failure of speech recognition. ∗ Equal contribution † Corresponding authors are used instead of the intended ones (Xu et al., 2021). This is especially true when users type fast or are using speech input in noisy environments or with less common accents that cause errors in post- processing systems. However, most benchmarks used in academic research do not explicitly try to capture such real-world input noises (Naplava et al., 2021), leaving the doubt whether models perform- ing well on standard clean test sets can transfer well onto real-world user-generated data. To evaluate the performance on noisy data for languages like English, existing work typically gen- erates typos via character-level perturbation such as randomly sampled or adversarial character swap or deletion (Belinkov and Bisk, 2018; Pruthi et al., 2019; Jones et al., 2020; Ma et al., 2020), automatic back-translation and speech conversion (Peskov et al., 2019; Ravichander …

Content truncated to 500 words