LIMIT-BERT : Linguistics Informed Multi-Task BERT

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 422–430 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 422 KorNLI and KorSTS: New Benchmark Datasets for Korean Natural Language Understanding Jiyeon Ham∗, Yo Joong Choe∗, Kyubyong Park∗, Ilji Choi, Hyungjoon Soh Kakao Brain {jiyeon.ham,yj.choe,kyubyong.park,ilji.choi,hj.soh}@kakaobrain.com Abstract Natural language inference (NLI) and seman- tic textual similarity (STS) are key tasks in natural language understanding (NLU). Al- though several benchmark datasets for those tasks have been released in English and a few other languages, there are no publicly available NLI or STS datasets in the Ko- rean language. Motivated by this, we con- struct and release new datasets for Korean NLI and STS, dubbed KorNLI and KorSTS, respectively. Following previous approaches, we machine-translate existing English train- ing sets and manually translate development and test sets into Korean. To accelerate re- search on Korean NLU, we also establish base- lines on KorNLI and KorSTS. Our datasets are publicly available at https://github.com/ kakaobrain/KorNLUDatasets. 1 Introduction Natural language inference (NLI) and semantic textual similarity (STS) are considered as two of the central tasks in natural language understanding (NLU). They are not only featured in GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019), which are two popular benchmarks for NLU, but also known to be useful for supplementary train- ing of pre-trained language models (Phang et al., 2018) as well as for building and evaluating fixed- size sentence embeddings (Reimers and Gurevych, 2019). Accordingly, several benchmark datasets have been released for both NLI (Bowman et al., 2015; Williams et al., 2018) and STS (Cer et al., 2017) in the English language. When it comes to the Korean language, however, benchmark datasets for NLI and STS do not exist. Popular benchmark datasets for Korean NLU typi- cally involve question answering12 and sentiment ∗Equal Contribution. 1https://korquad.github.io/ (Lim et al., 2019) 2http://www.aihub.or.kr/aidata/84 analysis3, but not NLI or STS. We believe that the lack of publicly available benchmark datasets for Korean NLI and STS has led to the lack of interest for building Korean NLU models suited for these key understanding tasks. Motivated by this, we construct and release Ko- rNLI and KorSTS, two new benchmark datasets for NLI and STS in the Korean language. Fol- lowing previous work (Conneau et al., 2018), we construct our datasets by machine-translating exist- ing English training sets and by translating English development and test sets via human translators. We then establish baselines for both KorNLI and KorSTS to facilitate research on Korean NLU. 2 Background 2.1 NLI and the {S,M,X}NLI Datasets In an NLI task, a system receives a pair of sen- tences, a premise and a hypothesis, and classifies their relationship into one out of three categories: entailment, contradiction, and neutral. There are several publicly available NLI datasets in English. Bowman et al. (2015) introduced the Stanford NLI (SNLI) dataset, which consists of 570K English sentence pairs based on image cap- tions. Williams et al. (2018) introduced the Multi- Genre NLI (MNLI) dataset, which consists of 455K …

Content truncated to 500 words