OCNLI: Original Chinese Natural Language Inference

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3512–3526 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3512 OCNLI: Original Chinese Natural Language Inference Hai Hu†⋄ Kyle Richardson‡ Liang Xu⋄ Lu Li⋄ Sandra K¨ubler† Lawrence S. Moss† †Indiana University, Bloomington, IN, USA ‡Allen Institute for Artificial Intelligence, Seattle, WA, USA ⋄Chinese Language Understanding Evaluation (CLUE) benchmark {huhai,skuebler,lmoss}@indiana.edu; kyler@allenai.org; xuliang@i-i.ai; ll000@mails.ccnu.edu.cn; CLUE@CLUEbenchmarks.com Abstract Despite the tremendous recent progress on nat- ural language inference (NLI), driven largely by large-scale investment in new datasets (e.g., SNLI, MNLI) and advances in modeling, most progress has been limited to English due to a lack of reliable datasets for most of the world’s languages. In this paper, we present the first large-scale NLI dataset (consisting of ∼56,000 annotated sentence pairs)1 for Chinese called the Original Chinese Natural Language In- ference dataset (OCNLI). Unlike recent at- tempts at extending NLI to other languages, our dataset does not rely on any automatic translation or non-expert annotation. Instead, we elicit annotations from native speakers spe- cializing in linguistics. We follow closely the annotation protocol used for MNLI, but create new strategies for eliciting diverse hypotheses. We establish several baseline results on our dataset using state-of-the-art pre-trained mod- els for Chinese, and find even the best perform- ing models to be far outpaced by human per- formance (∼12% absolute performance gap), making it a challenging new resource that we hope will help to accelerate progress in Chi- nese natural language understanding. To the best of our knowledge, this is the first human- elicited MNLI-style corpus for a non-English language. 1 Introduction In the last few years, natural language understand- ing has made considerable progress, driven largely by the availability of large-scale datasets and ad- vances in neural modeling (Peters et al., 2018; De- vlin et al., 2019). At the center of this progress has been natural language inference (NLI), which focuses on the problem of deciding whether two statements are connected via an entailment or a 1Our dataset and code are available at https:// github.com/CLUEbenchmark/OCNLI. contradiction. NLI profited immensely from new datasets such as the Stanford NLI (SNLI, Bow- man et al. (2015)) and Multi-Genre NLI (MNLI, Williams et al. (2018)) datasets. However, as of- ten the case, this progress has centered around the English language given that the most well-known datasets are limited to English. Efforts to build com- parable datasets for other languages have largely focused on (automatically) translating existing En- glish NLI datasets (Mehdad et al., 2011; Conneau et al., 2018). But this approach comes with its own issues (see section 2). To overcome these shortcomings and contribute to ongoing progress in Chinese NLU, we present the first large-scale NLI dataset for Chinese called the Original Chinese Natural Language Inference dataset (OCNLI). Unlike previous approaches, we rely entirely on original Chinese sources and use native speakers of Chinese with special expertise in linguistics and language studies for creating hy- potheses and for annotation. Our dataset contains ∼56,000 annotated premise-hypothesis pairs and follows a …

Content truncated to 500 words