docrep: A lightweight and efficient document representation framework

Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 762–771, Dublin, Ireland, August 23-29 2014. docrep: A lightweight and efficient document representation framework Tim Dawborn and James R. Curran e -lab, School of Information Technologies University of Sydney NSW 2006, Australia {tim.dawborn,james.r.curran}@sydney.edu.au Abstract Modelling linguistic phenomena requires highly structured and complex data representations. Document representation frameworks (DRFs) provide an interface to store and retrieve multiple annotation layers over a document. Researchers face a difficult choice: using a heavy-weight DRF or implement a custom DRF. The cost is substantial, either learning a new complex system, or continually adding features to a home-grown system that risks overrunning its original scope. We introduce DOCREP, a lightweight and efficient DRF, and compare it against existing DRFs. We discuss our design goals and implementations in C++, Python, and Java. We transform the OntoNotes 5 corpus using DOCREP and UIMA, providing a quantitative comparison, as well as discussing modelling trade-offs. We conclude with qualitative feedback from researchers who have used DOCREP for their own projects. Ultimately, we hope DOCREP is useful for the busy researcher who wants the benefits of a DRF, but has better things to do than to write one. 1 Introduction Computational Linguistics (CL) is increasingly a data-driven research discipline with researchers us- ing diverse collections of large-scale corpora (Parker et al., 2011). Representing linguistic phenomena can require modelling intricate data structures, both flat and hierarchical, layered over the original text; e.g. tokens, sentences, parts-of-speech, named entities, coreference relations, and trees. The scale and complexity of the data demands efficient representations. A document representation framework (DRF) should support the creation, storage, and retrieval of different annotation layers over collections of hetero- geneous documents. DRFs typically store their annotations as stand-off annotations, treating the source document as immutable and annotations “stand-off” with offsets back into the document. Researchers may choose to use a heavy-weight DRF, for example GATE (Cunningham et al., 2002) or UIMA (G¨otz and Suhre, 2004), but this can require substantial investment to learn and apply the framework. Alternatively, researchers may “roll-their-own” framework for a particular project. While this is not inherently bad, our experience is that the scope of such smaller DRFs often creeps, without the benefits of the features and stability present in mature DRFs. Moreover, some DRFs are based on object serialisation, restricting the user to a specific language. In sum, while DRFs provide substantial benefits, they can come at an opportunity cost to valuable research time. DOCREP aims to solve this problem by proving a light-weight DRF that does not get in the way. Using a language-agnostic storage layer enables reuse across different tasks in whatever tools and programming languages are most appropriate. Efficiency is our primary goal, and we emphasise compact serialisation and lazy loading. Our streaming design is informed by the pipeline operation of UNIX commands. Section 2 compares existing DRFs and annotation schemes. We describe and introduce DOCREP in Section 3, outlining the design goals and the problems it aims …

Content truncated to 500 words