Compositional Evaluation on Japanese Textual Entailment and Similarity

Compositional Evaluation on Japanese Textual Entailment and Similarity Hitomi Yanaka1 and Koji Mineshima2 1The University of Tokyo, Japan 2Keio University, Japan hyanaka@is.s.u-tokyo.ac.jp, minesima@abelard.flet.keio.ac.jp Abstract Natural Language Inference (NLI) and Se- mantic Textual Similarity (STS) are widely used benchmark tasks for compositional eval- uation of pre-trained language models. Despite growing interest in linguistic universals, most NLI/STS studies have focused almost exclu- sively on English. In particular, there are no available multilingual NLI/STS datasets in Japanese, which is typologically differ- ent from English and can shed light on the currently controversial behavior of lan- guage models in matters such as sensitivity to word order and case particles. Against this background, we introduce JSICK, a Japanese NLI/STS dataset that was manually trans- lated from the English dataset SICK. We also present a stress-test dataset for compo- sitional inference, created by transforming syntactic structures of sentences in JSICK to investigate whether language models are sensitive to word order and case particles. We conduct baseline experiments on differ- ent pre-trained language models and compare the performance of multilingual models when applied to Japanese and other languages. The results of the stress-test experiments suggest that the current pre-trained language models are insensitive to word order and case marking. 1 Introduction Natural Language Inference (NLI) (Dagan et al., 2006; Bowman et al., 2015) and Semantic Tex- tual Similarity (STS) (Agirre et al., 2016) tasks are well positioned to serve as a basic bench- mark for natural language understanding. With the recent progress of deep neural networks, includ- ing pre-trained language models such as BERT (Devlin et al., 2019), the development of bench- mark datasets has centered on large crowdsourced English datasets, such as SNLI (Bowman et al., 2015) and MultiNLI (Williams et al., 2018). Since there has been an increasing need for benchmark datasets in linguistic universals (Linzen, 2020), general language understanding frameworks in- cluding NLI and STS for languages other than English have been provided (Conneau et al., 2018; Liang et al., 2020; Le et al., 2020; Shavrina et al., 2020; Xu et al., 2020; Seelawi et al., 2021; Park et al., 2021). Another recent line of work has investi- gated whether models are sensitive to shuffled word order, but the conclusions are controversial (Ravfogel et al., 2019; Sinha et al., 2021a; Sinha et al., 2021b; Gupta et al., 2021; Pham et al., 2021; White and Cotterell, 2021). One charac- teristic of human-like language understanding is that humans can understand sentences according to their word meanings and syntactic structures, then recognize their semantic relationships (Frege, 1963; Katz and Fodor, 1963; Montague, 1973; Janssen and Partee, 1997). Since previous work has demonstrated the usefulness of analyzing the generalization ability of models in challenging NLI in English (Naik et al., 2018; Glockner et al., 2018; McCoy et al., 2019; Rozen et al., 2019; Goodwin et al., 2020; Yanaka et al., 2021), we should continue this line of research in other languages. Against this background, we provide a Japanese NLI/STS dataset to analyze language models in compositional inference across …

Content truncated to 500 words