Man vs. Machine: A Case Study in Base Noun Phrase Learning

Man* vs. Machine: A Case Study in Base Noun Phrase Learning Eric Brill and Grace Ngai Department of Computer Science The Johns Hopkins University Baltimore, MD 21218, USA Email: (brill,gyn}~cs. jhu. edu Abstract A great deal of work has been done demonstrat- ing the ability of machine learning algorithms to automatically extract linguistic knowledge from annotated corpora. Very little work has gone into quantifying the difference in ability at this task between a person and a machine. This pa- per is a first step in that direction. 1 Introduction Machine learning has been very successful at solving many problems in the field of natural language processing. It has been amply demon- strated that a wide assortment of machine learn- ing algorithms are quite effective at extracting linguistic information from manually annotated corpora. Among the machine learning algorithms stud- ied, rule based systems have proven effective on many natural language processing tasks, including part-of-speech tagging (Brill, 1995; Ramshaw and Marcus, 1994), spelling correc- tion (Mangu and Brill, 1997), word-sense dis- ambiguation (Gale et al., 1992), message un- derstanding (Day et al., 1997), discourse tag- ging (Samuel et al., 1998), accent restoration (Yarowsky, 1994), prepositional-phrase attach- ment (Brill and Resnik, 1994) and base noun phrase identification (Ramshaw and Marcus, In Press; Cardie and Pierce, 1998; Veenstra, 1998; Argamon et al., 1998). Many of these rule based systems learn a short list of simple rules (typ- ically on the order of 50-300) which are easily understood by humans. Since these rule-based systems achieve good performance while learning a small list of sim- ple rules, it raises the question of whether peo- *and Woman. 65 ple could also derive an effective rule list man- ually from an annotated corpus. In this pa- per we explore how quickly and effectively rel- atively untrained people can extract linguistic generalities from a corpus as compared to a ma- chine. There are a number of reasons for doing this. We would like to understand the relative strengths and weaknesses of humans versus ma- chines in hopes of marrying their con~plemen- tary strengths to create even more accurate sys- tems. Also, since people can use their meta- knowledge to generalize from a small number of examples, it is possible that a person could de- rive effective linguistic knowledge from a much smaller training corpus than that needed by a machine. A person could also potentially learn more powerful representations than a machine, thereby achieving higher accuracy. In this paper we describe experiments we per- formed to ascertain how well humans, given an annotated training set, can generate rules for base noun phrase chunking. Much previous work has been done on this problem and many different methods have been used: Church's PARTS (1988) program uses a Markov model; Bourigault (1992) uses heuristics along with a grammar; Voutilainen's NPTool (1993) uses a lexicon combined with a constraint grammar; Juteson and Katz (1995) use repeated phrases; Veenstra (1998), Argamon, Dagan & Kry- molowski(1998) and Daelemaus, van den Bosch & Zavrel (1999) use …

Content truncated to 500 words