ChID: A Large-scale Chinese IDiom Dataset for Cloze Test

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 778–787 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 778 ChID: A Large-scale Chinese IDiom Dataset for Cloze Test Chujie Zheng1,2, Minlie Huang1∗, Aixin Sun3 1Institute for Artificial Intelligence, Dept. of Computer Science, Tsinghua University 1Beijing National Research Center for Information Science and Technology, China 2Dept. of Physics, Tsinghua University, Beijing 100084, China 3School of Computer Science and Engineering, Nanyang Technological University, Singapore zcj16@mails.tsinghua.edu.cn, aihuang@tsinghua.edu.cn axsun@ntu.edu.sg Abstract Cloze-style reading comprehension in Chinese is still limited due to the lack of various cor- pora. In this paper we propose a large-scale Chinese cloze test dataset ChID, which stud- ies the comprehension of idiom, a unique lan- guage phenomenon in Chinese. In this corpus, the idioms in a passage are replaced by blank symbols and the correct answer needs to be chosen from well-designed candidate idioms. We carefully study how the design of candi- date idioms and the representation of idioms affect the performance of state-of-the-art mod- els. Results show that the machine accuracy is substantially worse than that of human, indi- cating a large space for further research. 1 Introduction Machine reading comprehension aims to assess the ability to comprehend natural language and answer questions from a given document or pas- sage. As a classical method of assessing language proficiency (Fotos, 1991; Jonz, 1991; Tremblay, 2011), cloze test (Taylor, 1953) has been widely employed due to its simplicity in form. Recently, a number of datasets for cloze test have been proposed for different languages. For instance, CNN/Daily Mail (Hermann et al., 2015) provides a benchmark for machine comprehension of En- glish text, while the People Daily and Children’s Fairy Tale dataset (Cui et al., 2016) and CMRC- 2017 (Cui et al., 2018) pioneer explorations in Chinese language. In this paper we explore idiom comprehension (Wray, 2002; Jackendoff and Jackendoff, 2002; Cacciari and Tabossi, 2014; Jiang et al., 2018) in cloze test. Idiom , which is called “成语” (chengyu) in Chinese, is an interesting linguistic phenomena in Chinese language, and this work ∗*Corresponding author: Minlie Huang. Idiom 亡羊补牢 Literal To mend the fence after sheep are lost Meaning Never be late to try Table 1: An example of metaphor in idiom. The sense of “亡羊补牢” should be inferred figuratively but not represented literally using the meanings of the four constituent characters. Idioms 侃侃而谈 口若悬河 Meanings To speak with fervour and assurance Fluently and eloquently Common Speak much and long Difference Describe the deme- anor of the speaker Describe the eloquence Table 2: An example of near-synonyms in idiom, where idioms share similar meanings but are different in lan- guage usage. is in parallel to several datasets (Hill et al., 2016; Xie et al., 2018) that have considered different lan- guage phenomena in English. Compared to other types of words, many idioms are unique for their non-compositionality and metaphorical meaning (see an example in Table 1). This feature requires a good representation of idiom. Meanwhile, …

Content truncated to 500 words