Japanese Realistic Textual Entailment Corpus

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6827–6834 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6827 Japanese Realistic Textual Entailment Corpus Yuta Hayashibe Megagon Labs, Tokyo, Japan, Recruit Co., Ltd. 7-3-5 Ginza Chuo-ku, Tokyo, 104-8227, Japan hayashibe@megagon.ai Abstract We perform the textual entailment (TE) corpus construction for the Japanese Language with the following three characteristics: First, the corpus consists of realistic sentences; that is, all sentences are spontaneous or almost equivalent. It does not need manual writing which causes hidden biases. Second, the corpus contains adversarial examples. We collect challenging examples that can not be solved by a recent pre-trained language model. Third, the corpus contains explanations for a part of non-entailment labels. We perform the reasoning annotation where annotators are asked to check which tokens in hypotheses are the reason why the relations are labeled. It makes easy to validate the annotation and analyze system errors. The resulting corpus consists of 48,000 realistic Japanese examples. It is the largest among publicly available Japanese TE corpora. Additionally, it is the first Japanese TE corpus that includes reasons for the annotation as we know. We are planning to distribute this corpus to the NLP community at the time of publication. Keywords: Textual Entailment, Language Model, BERT, Reasoning Annotation 1. Introduction In the era where massive texts are produced every day, ma- chines are indispensable to assist with daily life. However, to be truly helpful, machines must understand the meaning of texts (natural language understanding: NLU). An essen- tial task for NLU is Recognizing Textual Entailment (RTE), which is also known as Natural Language Inference (NLI). RTE predicts the relation between two statements, where one statement is called the “hypothesis” and the other is the “premise.” The goal is to predict whether the premise entails the hypothesis or not.1 Improving its performance is useful for natural language applications like Tatar et al. (2008) for summarization, and Harabagiu and Hickl (2006) for question answering. Many corpora have been created to train TE classifiers and evaluate RTE. In this paper, we perform the textual entailment (TE) cor- pus construction with the following three characteristics: First, the corpus consists of realistic2 sentences (Section 4). There are several methods to make hypothesis-premise pairs.3 In most previous works, human annotators are asked to compose new sentences for the given sentences to make examples in some previous works. However, such artifi- cial sentences lead hidden biases, because annotators un- consciously use particular words. Tsuchiya (2018) showed many TE labels in a corpus can be predicted without seeing premises. Hence we propose simple methods using seman- tic similarity and surface string similarity to collect natural occurring sentences to make examples. Second, the corpus contains adversarial examples (Section 5). The performances of recent pre-trained language models such as BERT (Devlin et al., 2019) far surpass those of previous models. the same performance as humans.They achieve almost However, there are still examples that cannot be solved by them. To …

Content truncated to 500 words