Large-scale Controlled Vocabulary Indexing for Named Entities

Large-scale Controlled Vocabulary Indexing for Named Entities Mark Wasson LEXIS-NEXIS, a Division of Reed Elsevier plc 9443 Springboro Pike Miamisburg, Ohio 45342 USA mark.wasson@lexis-nexis.com Abstract A large-scale controlled vocabulary indexing system is described. The system currently covers almost 70,000 named entity topics, and applies to documents from thousands of news publications. Topic definitions are built through substantially automated knowledge engineering. 1 Introduction The goal of the Entity Indexing R&D program at LEXIS-NEXIS is to add controlled vocabulary indexing for named entities to searchable fields in appropriate news documents across thousands of news publications, where documents include both incoming news articles as well as news articles already in the LEXIS-NEXIS archives. A con- trolled vocabulary term (CVT) is a consistently specified topic indicator that users can incorporate into their queries in order to retrieve documents about the corresponding topic. When a CVT is added to an appropriate field in a document, it can be included in a Boolean query using field-name(controlled vocabulary term) The initial Entity Indexing release focused on companies as topics. For company indexing, the primary CFT is a standard form of the company name. When we add a company CVT to a docu- ment, we often also want to add secondary CFTs to the document that specify attributes of that com- pany. Attributes may include the ticker symbol, SIC product codes, industry codes and company headquarters information. Secondary CVTs allow customers to easily search on groups of companies that have one or more attributes in common, such as searching for documents about banks in our set of companies that are headquartered in Utah. It is generally easy to get high recall with Boolean queries when searching for documents about named entities. Typically the query will only need a short form of the entity's name. For example, the query American will retrieve virtually every document that men- tions American Airlines. Of course, this query re- sults in poor precision due to the ambiguity of American. The problem we wanted to address with controlled vocabulary indexing is to help online customers limit their search results to only those documents that contain a major reference to the topic, that is, to documents that are substantially about the topic. Because of the volume of news data we have, it is necessary that we fully automate the document categorization and indexing step in our data prepa- ration process. LEXIS-NEXIS adds 100,000 news articles daily to its collection of over 2 billion documents. For marketing and product positioning reasons, we want to provide indexing for tens of thousands of companies, where companies are targeted based on their presence on the New York, American and NASDAQ exchanges or on revenue-based criteria. Although such selection criteria help us explain the product feature to customers, it does not ensure that the targeted companies actually appear all that often in the news. In fact, for many targeted com- panies there is little training and test data available. Our company indexing system should address the following business product requirements: …

Content truncated to 500 words