Set-Theoretic Alignment for Comparable Corpora

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 2009–2018, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Set-Theoretic Alignment for Comparable Corpora Thierry Etchegoyhen and Andoni Azpeitia Vicomtech-IK4 Mikeletegi Pasalekua, 57 Donostia / San Sebasti´an, Gipuzkoa, Spain {tetchegoyhen, aazpeitia}@vicomtech.org Abstract We describe and evaluate a simple method to extract parallel sentences from com- parable corpora. The approach, termed STACC, is based on expanded lexical sets and the Jaccard similarity coefficient. We evaluate our system against state-of-the- art methods on a large range of datasets in different domains, for ten language pairs, showing that it either matches or outper- forms current methods across the board and gives significantly better results on the noisiest datasets. STACC is a portable method, requiring no particular adaptation for new domains or language pairs, thus enabling the efficient mining of parallel sentences in comparable corpora. 1 Introduction With the rise of data-driven machine translation, be it statistical (Brown et al., 1990), example- based (Nagao, 1984), or rooted in neural networks (Bahdanau et al., 2014), the need for large paral- lel corpora has increased accordingly. Although quality bitexts have been made available over the years (Tiedemann, 2012), creating parallel corpora is a resource-consuming effort involving profes- sional human translation of large volumes of texts in multiple languages. As a consequence, there is still a lack of parallel data to properly model trans- lation across languages and domains. To overcome this limitation, special emphasis has been placed in the last two decades on the exploitation of comparable corpora, with the de- velopment of a range of methods to mine paral- lel sentences from texts addressing similar topics in different languages. The work we present fol- lows this line of research, describing and evaluat- ing a simple method that allows parallel sentences to be efficiently mined in different languages and domains with minimal adaptation effort. The method we describe, termed STACC, is based on expanded lexical sets and the Jaccard similarity coefficient (Jaccard, 1901), which is computed as the ratio of set intersection over union. We evaluate this simple approach against state-of-the-art methods for comparable sentence alignment on a variety of datasets for ten dif- ferent language pairs, showing that STACC either matches or outperforms competing approaches. The paper is organised as follows: Section 2 de- scribes related work on parallel sentence mining in comparable corpora; Section 3 presents the STACC method; Section 4 describes the experiments in comparable sentence alignment, including the de- scription of test corpora and systems, and an anal- ysis of the results; Section 5 presents results ob- tained with an optimised version of the alignment process, beyond system comparison; finally, Sec- tion 6 draws conclusions from the work described in the paper. 2 Related work A large variety of techniques have been proposed to mine parallel sentences in comparable cor- pora. One of the first approaches was proposed by (Zhao and Vogel, 2002), who combined sen- tence length and bilingual lexicon models under a maximum likelihood criterion. …

Content truncated to 500 words