Automatically generating hypertext in newspaper articles by computing semantic relatedness

I I I I I I I I I I I I I I I I I Automatically generating hypertext in newspaper articles by computing semantic relatedness Stephen J. Green Microsoft Research Institute School of Mathematics, Physics, Computing and Electronics* Macquarie University Sydney, NSW 2109 Australia sj green@mri.mq.edu.au Abstract We discuss an automatic method for the construction of hypertext links within and between newspaper articles. The method comprises three steps: determining the lexical chains in a text, building links between the paragraphs of articles, and building links between articles. Lexical chains capture the semantic relations between words that occur throughout a text. Each chain is a set of related words that captures a portion of the cohesive structure of a text. By considering the distribution of chains within an article, we can build links between the paragraphs. By computing the similarity of the chains contained in two different articles, we can decide whether or not to place a link between them. We also describe the results of an evaluation performed to test the methodology. 1 Introduction A survey, reported in Outing (1996), found that there were 1,115 commercial newspaper online services world- wide, 94% of which were on the World-Wide Web (WWW). Of these online newspapers, 73% are in North America. Outing predicted that the number of newspa- pers online would increase to more than 2,000 by the end of 1997. The problem is that these services are not making full use of the hypertext capabilities of the WWW. The user may be able to navigate to a particular article in the cur- rent edition of an online paper by using hypertext links, but they must then read the entire article to find the in- formation that interests them. These databases are "shal- low" hypertexts; the documents that are being retrieved are dead ends in the hypertext, rather than offering start- ing points for explorations. In order to truly reflect the hypertext nature of the Web, links should to be placed within and between the documents. As Westland (1991) has pointed out, manually creat- ing and maintaining the sets of links needed for a large- scale hypertext is prohibitively expensive. This is espe- cially true for newspapers, given the volume of articles Work done at the Department of Computer Science of the Univer- sity of Toronto produced every day. This could certainly account for the state of current WWW newspaper efforts. Aside from the time-and-money aspects of building such large hypertexts manually, humans are inconsistent in assigning hypertext links between the paragraphs of documents (Ellis et al., 1994; Green, 1997). That is, different linkers disagree with each other as to where to insert hypertext links into a document. The cost and inconsistency of manually constructed hypertexts does not necessarily mean that large-scale hy- pertexts can never be built. It is well known in the IR community that humans are inconsistent in assigning in- dex terms to documents, but this has not hindered the construction of automatic indexing systems intended to …

Content truncated to 500 words