The xtsv Framework and the Twelve Virtues of Pipelines

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 7044–7052 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 7044 The xtsv Framework and the Twelve Virtues of Pipelines Bal´azs Indig, B´alint Sass, Iv´an Mittelholcz MTA Research Institute for Linguistics Bencz´ur u. 33., H-1068, Budapest, Hungary {LASTNAME.FIRSTNAME}@nytud.mta.hu Abstract We present xtsv, an abstract framework for building NLP pipelines. It covers several kinds of functionalities which can be implemented at an abstract level. We survey these features and argue that all are desired in a modern pipeline. The framework has a simple yet powerful internal communication format which is essentially tsv (tab separated values) with header plus some additional features. We put emphasis on the capabilities of the presented framework, for example its ability to allow new modules to be easily integrated or replaced, or the variety of its usage options. When a module is put into xtsv, all functionalities of the system are immediately available for that module, and the module can be be a part of an xtsv pipeline. The design also allows convenient investigation and manual correction of the data flow from one module to another. We demonstrate the power of our framework with a successful application: a concrete NLP pipeline for Hungarian called e-magyar text processing system (emtsv) which integrates Hungarian NLP tools in xtsv. All the advantages of the pipeline come from the inherent properties of the xtsv framework. Keywords: xtsv, general framework, NLP pipeline, NLP tools, tsv 1. Introduction Concerning NLP processing tools, there are big players and small ones as well. Numerous small independent tools ex- ist, their common value is that many of them implement brilliant ideas in an elegant manner, however they may not be acknowledged properly. The main reason for this is one of the followings: they handle some corner cases in their own way, they are language-dependent, they are im- plemented in a less-known programming language or they do not fit in a larger framework. When one starts to de- velop a new tool, the creative part of the work is to devise and realise one’s solution to the well-defined problem in order to showcase the idea by comparing it to the state-of- the-art alternative. However, in order to help this idea to spread widely, one must augment the implementation with a standard I/O interface to let others test it. In some cases, a demo service is also needed to draw attention to the so- lution, else the program – no matter how good it is – will become obsolete and finally not used by anybody. What is worse, it will probably be reinvented again, possibly in the same way. Usually, the developer concentrates on the inner-workings that results in neglecting “the interface boilerplate’. Nowa- days, the time left for the ideas to reach the market is shorter than ever before. The responsibility of the used framework is to reduce the number of steps needed for shipping a new idea. This is …

Content truncated to 500 words