Lexicons for Human Language Technology

Lexicons for Human Language Technology Mark Liberman Linguistic Data Consortium University of Pennsylvania Philadelphia, PA 19104-6305 myl@unagi, cis. upenn, edu ABSTRACT Information about words--their pronunciation, syntax and meaning--is a crucial and costly part of human language technology. Many questions remain about the best way to express and use such lexical information. Nevertheless, much of this information is common to all current approaches, and therefore the effort to collect it can usefully be shared. The Linguistic Data Consortium (LDC) has undertaken to pro- vide such common lexical information for the community of HLT researchers. The purpose of this paper is to sketch the various LDC lexical projects now underway or planned, and to solicit feedback from the community of HLT researchers. 1. Introduction This paper will give an overview of current LDC efforts to develop lexical resources and describe some efforts now in the planning stage. Readers are invited to join an on-going discussion of priorities, methods and even formats for our present and future efforts in this area. 1.1. Intellectual Property Rights Since lexicons, unlike text and speech databases, are likely to be incorporated (perhaps in derived form) in commercial HLT products, intellectual property rights come to center stage. The LDC's charter as a consor- tium requires us to leverage the U.S. government's in- vestment by sharing the cost of resource development among our members. This forces us to limit usage of such resources to consortium members, or others who have paid an appropriate fee. However, we also want to encourage rapid development and broad exploitation of commercial HLT technology. Therefore, we need to pro- tect our members' investment in research based on LDC resources by ensuring their rights to future commercial exploitation without additional license negotiations, roy- alty payments, or other intellectual property issues. This contrasts with the general practice for research use of machine-readable dictionaries, in which all rights to derived works are typically reserved to the publisher. For this reason, our lexicons will not be derived from exist- ing lexicons, except as permitted by normal provisions of copyright law, or in case we are able to purchase up- propriate rights from the owner of the existing resource. This also contrast with our practice with respect to text databases, where we have negotiated agreements to dis- tribute for research purposes many bodies of text whose copyright remains with the original owner. The differ- ence here is that the text corpora themselves will not typically be incorporated in future products, and our understanding of the applicable law (which we openly explain to information providers) is that language mod- els trained on such text are free of any IPR taint. We have worked hard, in consultation with our mem- bers, to develop an appropriate license for LDC lexi- cons. A copy of the draft license agreement for COM- LEX syntax will be furnished on request to the author, or ldc@unagi.cis.upenn.edu. 2. Current English Lexicon Efforts Our primary effort is to provide lexicons for English. We are funding a large, high-quality …

Content truncated to 500 words