The Chinese Causative-Passive Homonymy Disambiguation: an adversarial Dataset for NLI and a Probing Task

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4316–4323 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4316 The Chinese Causative-Passive Homonymy Disambiguation: an Adversarial Dataset for NLI and a Probing Task Shanshan Xu1,2, Katja Markert3 1L3S Research Center, Germany 2Department of Informatics, Technical University of Munich, Germany 3Institute of Computational Linguistics, Heidelberg University, Germany shanshan.xu@tum.de, markert@cl.uni-heidelberg.de Abstract The disambiguation of causative-passive homonymy (CPH) is potentially tricky for machines, as the causative and the passive are not distinguished by the sentences’ syntactic structure. By transforming CPH disambiguation to a challenging natural language inference (NLI) task, we present the first Chinese Adversarial NLI challenge set (CANLI). We show that the pretrained transformer model RoBERTa, fine-tuned on an existing large-scale Chinese NLI benchmark dataset, performs poorly on CANLI. We also employ Word Sense Disambiguation as a probing task to investigate to what extent the CPH feature is captured in the model’s internal representation. We find that the model’s performance on CANLI does not correspond to its internal representation of CPH, which is the crucial linguistic ability central to the CANLI dataset. CANLI is available on Hugging Face Datasets (Lhoest et al., 2021) at https://huggingface.co/datasets/sxu/CANLI Keywords: natural language inference, causative-passive homonymy, Chinese, adversarial dataset 1. Introduction Pretrained Language Models (PLMs) have recently achieved significant progress in natural language un- derstanding tasks. Some models even claim to have sur- passed human performance (Devlin et al., 2019; Wang et al., 2019a). However, recent research (Bender and Koller, 2020) questions whether these models really understand the meaning of natural language. NLI is a canonical natural language understanding task, where a system must determine the relationship, such as entailment or contradiction, between a premise and a hypothesis. NLI performance is tested on large-scale datasets, of which several exist for English such as SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018). Some non-English NLI datasets exist but are mostly gen- erated either automatically or translated from existing English NLI datasets (Ham et al., 2020; Conneau et al., 2018). OCNLI (Hu et al., 2020) is a large-scale NLI dataset for Chinese that has not been generated by translation. State-of-the-Art PLMs achieve high performance for NLI tasks, reporting up to 90% accuracy of English RoBERTa on MNLI (Wang et al., 2019b; He et al., 2020) and around 78% of Chinese RoBERTa on OCNLI (Hu et al., 2020). However, some researchers (Ben- der and Koller, 2020) have recently challenged whether these benchmarks test natural language understanding in full and pointed out that the models might achieve their excellent performance by relying on spurious sta- tistical patterns in the data instead of by understanding meaning. This has been shown by feeding NLI systems with adversarial data sets where they might fail. These adversarial datasets are mostly limited to English, We are to the best of our knowledge the first to con- struct an adversarial NLI dataset for Chinese1. Our hypothesis is that certain linguistic phenomena can be exploited for …

Content truncated to 500 words