Evaluation as a Language Technology Deployment Trigger

Evaluation as a Language Technology Deployment Trigger Rita Nübel and Jörg Schütz IAI Saarbrücken {rita,joerg}@iai.uni-sb.de Abstract Language technology (LT) plays an increasingly important role in the field of multilingual information content production and knowledge management. Companies are more and more faced with the problem of the multilingual bottleneck, and are looking for new technologies which bring in productivity gains in the localisation and globalisation process. On the other hand, the increasing number of LT products, especially machine translation systems (MT) calls for useful and practical assessment procedures for potential users in order to prove if a given system fits a user's needs. Static evaluations which test a system's fitness as-is, may be good for coarse-grained system comparisons, e.g. for best buy reports, or to make a system rating as the first step of more detailed system tests. Dynamic evaluations which consider also a system's adaptability and extensibility in a certain use case are more time-consuming, but they are the most realistic and constructive approaches to evaluation: they ideally combine both the assessment of the state-of-art of a system with the assessment of its extensibility and adaptability potential. The paper investigates the requirements for a useful evaluation of MT and presents practical guidelines to trigger the successful deployment of LT in an industrial context. The scenario is multilingual technical documentation in the automotive industry. 1 Introduction Deploying MT in a multilingual content production and management scenario is challenging because of the many well known problems of MT. On the other hand, jumping to "no go" conclusions when looking at bad MT output means ignoring the potential of MT. Today, MT deployment and LT deployment in general is complex not only because of the growing mar- ket of LT products which are still (or once again) offered as problem solvers -- though with a little less enthusiasm than in their first declared prime decades ago. Also, the situatedness of these technologies is becoming more and more complex. In the scenario of multilingual document production, MT has to be considered as part of a complex workflow and thus has to "communicate" or to interface with a couple of other processes. Hence, deploying language technology (LT) in a multilingual information production chain within an industrial setting requires a careful analysis of work-flows, including the anticipation of changes induced by LT technology, as well as detailed requirements analyses in various areas (business, user, qual- ity). Furthermore, a thorough evaluation of other existing LT products is essential in order to determine the range of products which shall support the multilingual document production. Taking these rather complex tasks into account, this paper presents some practical guidelines for a successful deployment of LT focusing on integrating MT into the multilingual document production process. The chosen scenario is the production process of technical documentation in the automotive industry. Section 2 starts from the definition of evaluation of LT / MT, and then identifies possible evaluation addressees, their reasons and some relevant criteria for in- dividual translation tasks. Section 3 …

Content truncated to 500 words