Fully Unsupervised Crosslingual Semantic Textual Similarity Metric Based on BERT for Identifying Parallel Data

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 206–215 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 206 Fully unsupervised crosslingual semantic textual similarity metric based on BERT for identifying parallel data Chi-kiu Lo and Michel Simard NRC-CNRC National Research Council Canada 1200 Montreal Road, Ottawa, Ontario K1A 0R6, Canada {Chikiu.Lo|Michel.Simard}@nrc-cnrc.gc.ca Abstract We present a fully unsupervised crosslin- gual semantic textual similarity (STS) met- ric, based on contextual embeddings extracted from BERT – Bidirectional Encoder Repre- sentations from Transformers (Devlin et al., 2019). The goal of crosslingual STS is to mea- sure to what degree two segments of text in different languages express the same mean- ing. Not only is it a key task in crosslingual natural language understanding (XLU), it is also particularly useful for identifying paral- lel resources for training and evaluating down- stream multilingual natural language process- ing (NLP) applications, such as machine trans- lation. Most previous crosslingual STS meth- ods relied heavily on existing parallel re- sources, thus leading to a circular dependency problem. With the advent of massively mul- tilingual context representation models such as BERT, which are trained on the concatena- tion of non-parallel data from each language, we show that the deadlock around parallel re- sources can be broken. We perform intrinsic evaluations on crosslingual STS data sets and extrinsic evaluations on parallel corpus filter- ing and human translation equivalence assess- ment tasks. Our results show that the unsu- pervised crosslingual STS metric using BERT without fine-tuning achieves performance on par with supervised or weakly supervised ap- proaches. 1 Introduction Crosslingual semantic textual similarity (STS) (Agirre et al., 2016a; Cer et al., 2017) aims at mea- suring the degree of meaning overlap between two texts written in different languages. It is a key task in crosslingual natural language understand- ing (XLU), with applications in crosslingual in- formation retrieval (Franco-Salvador et al., 2014; Vuli´c and Moens, 2015), crosslingual plagiarism detection (Franco-Salvador et al., 2016a,b), etc. It is also particularly useful for identifying parallel resources (Resnik and Smith, 2003; Aziz and Spe- cia, 2011) for training and evaluating downstream multilingual NLP applications, such as machine translation systems. Unlike in crosslingual textual entailment (Negri et al., 2013) or crosslingual natural language infer- ence (XNLI) (Conneau et al., 2018), which are di- rectional classification tasks, in crosslingual STS, continuous values are produced, to reflect a range of similarity that goes from complete semantic unrelatedness to complete semantic equivalence. Machine translation quality estimation (MTQE) (Specia et al., 2018) is perhaps the field of work that is the most related to crosslingual STS: in MTQE, one tries to estimate translation quality, by comparing an original source-language text with its machine translation. In contrast, in crosslin- gual STS, neither the direction nor the origin (hu- man or machine) of the translation is taken into account. Furthermore, MTQE also typically con- siders the fluency and grammaticality of the target text; these aspects are usually not perceived as rel- evant for crosslingual STS. Many previous crosslingual …

Content truncated to 500 words