Exploiting a Large Strongly Comparable Corpus

Exploiting a Large Strongly Comparable Corpus Thierry Etchegoyhen, Andoni Azpeitia and Naiara P´erez Vicomtech-IK4 Donostia / San Sebasti´an, Spain {tetchegoyhen, aazpeitia, nperez}@vicomtech.org Abstract This article describes a large comparable corpus for Basque and Spanish and the methods employed to build a parallel resource from the original data. The EITB corpus, a strongly comparable corpus in the news domain, is to be shared with the research community, as an aid for the development and testing of methods in comparable corpora exploitation, and as basis for the improvement of data-driven machine translation systems for this language pair. Competing approaches were explored for the alignment of comparable segments in the corpus, resulting in the design of a simple method which outperformed a state-of-the-art method on the corpus test sets. The method we present is highly portable, computationally efficient, and significantly reduces deployment work, a welcome result for the exploitation of comparable corpora. Keywords: Comparable corpora, Alignment, Corpus exploitation, Basque, Statistical machine translation 1. Introduction Comparable corpora are a useful resource to overcome the issue of scarce parallel data in some language pairs and domains, providing statistical machine translation systems (Brown et al., 1990; Koehn, 2010) with the necessary data to increase overall MT quality (Munteanu and Marcu, 2005; Irvine and Callison-Burch, 2013). In this paper, we de- scribe a large comparable corpus for Basque and Spanish and the methods employed to build a parallel resource from the original data. Our first contribution is thus a description of the first large corpus for the Basque-Spanish language pair in the news domain, based on content professionally created by the EITB Basque public broadcasting services. This resource will be shared with the research community, as a basis for further advances in comparable corpora exploitation and data-driven machine translation. Additionally, the resource will contribute to advancing research on a minority lan- guage with scarce publically available resources. The second contribution of this paper is the evaluation of a simple method for the alignment of comparable seg- ments,1 which improves significantly over the state-of-the- art aproaches we tested on the corpus in terms of F1 mea- sure. The approach we describe, which is based on the Jac- card index and lexical set expansion, also improves over existing methods in terms of simplicity and deployment ef- fort. We first describe the EITB and IVAP corpora in Section 2, which feed the alignment methods described in the remain- der of the paper. Section 3 presents the alignment methods we used for document and segment aligment. In section 4, we discuss the results obtained on the EITB corpus. Finally, Section 5 presents concluding remarks and a description of future work. 1Hereafter, we use the term segment to refer to sentences as well as smaller independent linguistic units. 2. Corpora The EITB corpus is composed of news written by jour- nalists of the Basque Country’s public broadcast service.2 The news are written independently in Basque and Span- ish but refer to the same specific events. The corpus can thus be categorized as …

Content truncated to 500 words