Dictionary development workflow for MT: design and management

Dictionary Development Workflow for MT: Design and Management Mike Dillinger Logos Corporation 100 Enterprise Drive, Suite 501 Rockaway, NJ 07866 USA mdillinger@logos-usa.com Abstract An important part of the development of any machine translation system is the creation of lexical resources. We describe an analysis of the dictionary development workflow and supporting tools currently in use and under development at Logos. This workflow identifies the component processes of: setting goals, locating and acquiring lexical resources, transforming the resources to a common format, classifying and routing entries for special processing, importing entries, and verifying their adequacy in translation. Our approach has been to emphasize the tools necessary to support increased automation and use of resources available in electronic formats, in the context of a systematic workflow design. Keywords dictionary development; lexicon; machine translation; workflow Market forces are pressuring language engineering in general and machine translation in particular to evolve from cottage industries producing fine, handcrafted exemplars of custom systems to lower-cost, quickly- developed, open and adaptable components. There seems to be a clear consensus that language engineering technology is technically feasible; the problem now is to show that is economically viable. Since human intervention plays such an important role in the economics of developing natural language systems, management of human resources for such labor-intensive products will be an important factor in determining the future of the industry. A significant part of the development of a commercial machine translation (MT) system for a new language pair or a new domain is the development of the lexical resources that the system will use. However, there seems to be little literature available about efficient procedures or "best practices" for developing these resources. Even at the 1993 AAAI Symposium on "Building Lexicons for Machine Translation", there was more attention to theoretical issues than to establishing effective processes for dictionary development. Lexical resources are available in electronic format, techniques exist for extracting terms from large corpora, and standards are being developed for the exchange of lexical data, but it is clear that there are no de facto standards for format or content, no leading providers of lexical data, and no best practices to follow for someone who has to start building a new dictionary today. On the other hand, apocryphal accounts abound of bleary- eyed, underpaid and overworked interns and relatives "just typing the dictionary in". This is clearly not the method of choice for a company that has to deliver a quality product on time, under budget, meeting the customer's needs for lexical coverage, and then scale up to do the same for several languages and several different domains simultaneously. Since dictionary development can easily consume 30% or more of the total resources allocated for the development a language pair, in a commercial setting it has to be taken very seriously. A systematically designed and tested workflow for the development of lexical resources can lead to reduced expenses, higher quality products, quicker time to market and more reliable outsourcing of lexical development efforts. The present paper …

Content truncated to 500 words