Using Diverse Information Sources to Retrieve Samples of Low Density Languages

Proceedings of the Australasian Language Technology Workshop 2005, pages 64–70, Sydney, Australia, December 2005. Using Diverse Information Sources to Retrieve Samples of Low-Density Languages Andrew MacKinlay Department of Computer Science and Software Engineering University of Melbourne Parkville VIC 3051 Australia Abstract Language samples are useful as an object of study for a diverse range of people. Samples of low-density languages in particular are often valuable in their own right, yet it is these sam- ples which are most difficult to locate, especially in a vast repository of information such as the World Wide Web. We identify here some short- comings to the more obvious approaches to lo- cating such samples and present an alternative technique based on a search query using pub- licly available wordlists augmented with geospa- tial evidence, and show that the technique is successful for a number of languages. 1 Introduction The utility of language samples to anyone with an interest in a given language is obvious: they can be valuable to linguists, language technolo- gists and speakers of the language, among oth- ers. The World Wide Web (WWW) is vast repository of information with potentially large numbers of samples1 of many languages, but locating these samples reliably is a non-trivial task. While language-specific search tools on search engines such as Google are useful for the languages they cover, for the vast majority of languages which have fewer speakers and a smaller online presence (low-density languages) they provide no information, and it is these lesser-known languages for which language re- sources are likely to be most useful to language researchers. There are existing online resources which pro- vide varying degrees of coverage for a large num- ber of languages but the amount of data stored in these for even the best covered languages is generally quite limited. It is likely that there is a large number of documents on the WWW 1In the context of this paper, we define a sample of a particular language as a webpage with a substantial proportion of content written in the language which for a variety of reasons have not made their way into these repositories. When we expand our focus to include the en- tire Web, it essentially becomes a problem of language classification, but instead of looking at just a handful of documents as is often the norm in language classification tasks, the num- ber of potential matches is all of the ∼8 billion documents on the WWW indexed by Google. This means the first stage of the classification will necessarily be a search to vastly cut down the number of potential matches. There are of course other alternatives to find- ing web-based data for these low-density lan- guages. The most obvious choice of a WWW query using the name of a language works in some cases, but there are of course complica- tions. Languages often have more than one name, such as Adamawe Fulfulde (Niger-Congo, West Africa) which has roughly 33 distinct names including vastly different names such …

Content truncated to 500 words