Active Learning and the Irish Treebank

Active Learning and the Irish Treebank Teresa Lynn1,2, Jennifer Foster1, Mark Dras2 and Elaine U´ı Dhonnchadha3 1NCLT/CNGL, Dublin City University, Ireland 2Department of Computing, Macquarie University, Sydney, Australia 3Centre for Language and Communication Studies, Trinity College, Dublin, Ireland 1{tlynn,jfoster}@computing.dcu.ie 2{teresa.lynn,mark.dras}@mq.edu.au, 3uidhonne@tcd.ie Abstract We report on our ongoing work in devel- oping the Irish Dependency Treebank, de- scribe the results of two Inter-annotator Agreement (IAA) studies, demonstrate improvements in annotation consistency which have a knock-on effect on parsing accuracy, and present the final set of de- pendency labels. We then go on to investi- gate the extent to which active learning can play a role in treebank and parser develop- ment by comparing an active learning boot- strapping approach to a passive approach in which sentences are chosen at random for manual revision. We show that active learn- ing outperforms passive learning, but when annotation effort is taken into account, it is not clear how much of an advantage the active learning approach has. Finally, we present results which suggest that adding automatic parses to the training data along with manually revised parses in an active learning setup does not greatly affect pars- ing accuracy. 1 Introduction The Irish language is an official language of the European Union and is the first national language of the Republic of Ireland. It is a Verb-Subject- Object language, belonging to the Celtic language group. Irish is considered a low-density language, lacking in sufficient resources for various natural language processing (NLP) applications. The de- velopment of a dependency treebank is part of a recent initiative to address this lack of resources, as has been the case for, for example, Danish (Kromann, 2003), Slovene (Dˇzeroski et al., 2006) and Finnish (Haverinen et al., 2010). Statisti- cal parsers are data-driven and require a sufficient number of parsed sentences to learn from. One of the expected uses of a treebank for Irish is to provide training data for the first Irish statistical dependency parser which will form the basis of useful NLP applications such as Machine Trans- lation or Computer Aided Language Learning. What counts as a sufficient number of trees for training an Irish statistical dependency parser re- mains an open question. However, what is clear is that the parser needs to have encountered a lin- guistic phenomenon in training in order to learn how to accurately analyse it. Creating a treebank is a resource-intensive process which requires ex- tensive linguistic research in order to design an appropriate labelling scheme, as well as consid- erable manual annotation (parsing). In general, manual annotation is desired to ensure high qual- ity treebank data. Yet, as is often encountered when working with language, the task of man- ually annotating text can become repetitive, in- volving frequent encounters with similar linguis- tic structures. In an effort to speed up the creation of tree- banks, there has been an increased focus towards automating, or at least, semi-automating the pro- cess using various bootstrapping techniques. A basic bootstrapping approach such as that out- …

Content truncated to 500 words