Semi-automatic evaluation of the grammatical coverage of machine translation systems

Semi-Automatic Evaluation of the Grammatical Coverage of Machine Translation Systems A. Guessoum & R. Zantout Dep. of Computer Science Dep. of Computer Science The University of Sharjah Faculty of Sciences P.O. Box 27272 The University of Balamand Sharjah, UAE P.O. Box 100, Tripoli, Lebanon e-mails: guessoum@sharjah.ac.ae rached.zantout@balamand.edu.lb Abstract In this paper we present a methodology for automating the evaluation of the grammatical coverage of machine translation (MT) systems. The methodology is based on the importance of unfolded grammatical structures, which represent the most basic syntactic pattern for a sentence in a given language. A database of unfolded grammatical structures is built to evaluate the parser of any NLP or MT system. The evaluation results in an overall measure called the grammatical coverage. The results of implementing the above approach on three English-to-Arabic commercial MT systems are presented. Keywords Machine Translation Evaluation; Translation Quality; Grammatical Coverage; Arabic Machine Translation; Parser Evaluation I. Introduction With the ever-growing interest in natural language processing (NLP) systems and machine translation (MT) systems, grows a need for developing adequate methodologies and tools for assessing their qualities. Such systems need to be assessed by system developers for any possible technological improvements and novel research ideas and by potential users for quality comparison purposes. The evaluation of NLP systems is classically divided into two main approaches: Glass-box and Black-box (Hutchins & Somers, 1992), (Nyberg & Mitamura & Carbonell, 1993), (Arnold, 1995). In black-box evaluation, the evaluator has access only to the input and output of the system under evaluation. In Glass-box evaluation, the evaluator also has access to the various workings of the system and can thus assess each sub-part of the system. Component-based evaluation and detailed error analyses are also important types of evaluation (Nyberg & Mitamura & Carbonell, 1993), (Arnold, 1995), and (Hedberg, 1995). In addition, a NLP system, just like any other software, needs to be evaluated for its cost, performance, stability, maintainability, and portability among other criteria. Subjectivity of NLP/MT system evaluation stems from the fact that evaluation methods rely heavily on humans. This makes the assessment depend on the evaluator’s background, skills, or even taste! The evaluation process is affected by the degree of proficiency of the human evaluator in the various facets of the languages that are involved in the translation. Even when strict and clear rules are introduced at the beginning of the evaluation process, the exact score given to a system component under evaluation would vary from one (human) evaluator to another. Hence more objective evaluation methodologies need to be developed. In Section II, a brief overview of some of the work on MT evaluation is given. In Section III, we introduce our methodology for evaluating the grammatical coverage of MT systems, which is similar to the methodology introduced in (Guessoum & Zantout, 2001) for MT system lexicon evaluation. This methodology minimizes the amount of subjectivity in the evaluation process by automating some evaluation tasks. Section IV presents and discusses the results of the implementation of the methodology for evaluating …

Content truncated to 500 words