The brWaC Corpus: A New Open Resource for Brazilian Portuguese

The brWaC Corpus: A New Open Resource for Brazilian Portuguese Jorge A. Wagner Filho1, Rodrigo Wilkens2, Marco Idiart3, Aline Villavicencio1,4 1 Institute of Informatics, Federal University of Rio Grande do Sul, Brazil 2 CENTAL, Universit´e catholique de Louvain - UCL 3 Institute of Physics, Federal University of Rio Grande do Sul, Brazil 4 School of Computer Science and Electronic Engineering, University of Essex, UK jawfilho@inf.ufrgs.br, rodrigo.wilkens@uclouvain.be, marco.idiart@gmail.com, avillavicencio@inf.ufrgs.br Abstract In this work, we present the construction process of a large Web corpus for Brazilian Portuguese, aiming to achieve a size comparable to the state of the art in other languages. We also discuss our updated sentence-level approach for the strict removal of duplicated content. Following the pipeline methodology, more than 60 million pages were crawled and filtered, with 3.5 million being selected. The obtained multi-domain corpus, named brWaC, is composed by 2.7 billion tokens, and has been annotated with tagging and parsing information. The incidence of non-unique long sentences, an indication of replicated content, which reaches 9% in other Web corpora, was reduced to only 0.5%. Domain diversity was also maximized, with 120,000 different websites contributing content. We are making our new resource freely available for the research community, both for querying and downloading, in the expectation of aiding in new advances for the processing of Brazilian Portuguese. Keywords: Web as Corpus, large corpus, Brazilian Portuguese 1. Introduction In recent years, initiatives for the construction of large cor- pora have attained ever-growing interest in the NLP com- munity. They are especially relevant for applications which demand large volumes of data, such as neural methods (Pennington et al., 2014; Bahdanau et al., 2014; Johnson et al., 2016), and also tend to produce more reliable re- sources for statistical models. In this paper, we build on previous work (Wagner Filho et al., 2016) and aim to con- struct a large and freely available Web corpus for Brazil- ian Portuguese, compatible with the state of the art in other languages. An example of the need for new, large corpora in this language was shown by Rodrigues et al. (2016), who had to combine 19 different corpora to obtain a 1.7 billion tokens corpus and create a distributional semantics model comparable to those available for English. Besides language models, the corpus presented here can also be used, for example, in dictionary creation (Kilgarriff et al., 2008), word similarity (Levy et al., 2015) and word sense induction (Navigli and Velardi, 2010; Di Marco and Nav- igli, 2013). A widely adopted approach for the construction of large corpora is the WaCky (Web-As-Corpus Kool Yinitiative) methodology (Bernardini et al., 2006; Baroni et al., 2009), which has been used to provide corpora in the scale of bil- lions of tokens for multiple languages, extracting text con- tent from the Web. It also enables the construction of cor- pora without domain biases, considering that a corpus will automatically get balanced after reaching a substantial size (Xiao, 2010). When constructing these corpora a real concern is to gather as representative …

Content truncated to 500 words