Annotating 200 Million Words: The Bank of English Project

ANNOTATING 200 MILLION WORDS: THE BANK OF ENGLISH PROJECT Timo Jiirvinen Research Unit for Conq)ut;;Ltional l,inguistics Uniwersity of Itelsinki Abstract The llank of English is an international English hm- guage project sponsored by llarper-Collins Publish- ers, Glasgow, and conducl;ed by the COBUILD team at the University of Birrnhlgham, UK. The text hank will comprise some 200 million words of both written and spoken English. The whole 200 million word col pns is being annotated morphologically and syntacti- cally during 1993-94 at the Research Unit for Cor,,- Imtational Linguistics (IL/I(3L), University of Ilel- sinkl, using the Fmglish nmrphological analyser (ENC,- TW()I,) and English Constraint (:h'ammar (EN(:I(:.'(:~) parser. The first half of the texts (103 million words) has ah'eady been processed in 1993. The project is lead by Prof. 3ohn Sinchdr in Birmingham, and l'rof. Fred Karlsson in Ilelsinld. The present author is re- sponsible for conducting the annotation. In the introdnction of this paper the r,.:mtines Ibr dealing with htrge text corpora are presented and our analysing system outlined. Chapter 2 gives an' overlook how the texts are preprocessed. Chapter 3 descrihes the lexicon updating, which is a prelim- inary step to the analysis. The last part presents the li;N(:'C(~ parser and the ongoing developtnel,t of it.s syntactic cornponent. 1 INTRODUCTION Each rno,fl.h the (~OBUILI) tean-i supplies an apl)rox- imately 10 million word b;ttch of lw:trl~.tlp coded run- ning text (see Appendix A) in ASCII format. Every new batch is lh'st scanned by the EN(TI'WOI, lexi- eel and rnorphological analyser [Koskentdetni, 1!.183] in filtering rhode for the purpose of detecting words not inchaled in the present lexicon. This is followed hy a semi-autou-mtic tlpdal.illg of the lexicon. After these ad,iustments, the whole sysl.em is used for annol.at.ing; the data. ()ur mmlyslng system, which is presented in del.ail in [Karlsson, 19{),1], consists of the following successive sLages: • preprocessing • EN(V]'WOL lexical analysis • EN(:K~G morl~hological disambiguation • ENGC(.-I syntactic mapping and disamhiguation ']'he main routines performed on the rnonthly data, includitlg constant monitoring of hoth inconting texts and ~malysed output and management (doc- umentation, backul~S ) are closely linked to the hi)- dating of the preprocessing module and the EN(L TWOL lexicon. 2 PREPROCESSOR The preprocesshlg moduh,s stan,ihu.di:+e the runnh+g text and tokenise it into a, fc.z'm suitable for I,hc:~ EN(VI'WOI, lexica] aualyser. I'~N(~I(,'G has been developed so that it, takes lute ;tccotllll, Variolls Lexttl,'d codhlg cotwelfl;iofts [l(al']s- son, 1994]. "W'e In;ave develoF, ed i~reprocessing pro- cedures furthc:r to ca.ter %r the: dillbrc.nl, t.ypes of nmrkup c,w.tes systema+tica.lly. Since l.exl;s usually cofne frolft varJotls sollrces+ t]lel'<~ frilly be tlrldocll- rnented i(][osyllcra, cies or systelflat, ic cl'rops ill SOlfie S;II ill)leg. The inh)rmation cotweyed by the markup codes is lntilised iu the parshlg process. ,Ul-+lati,g the i~reprc.cesshtg module to achiew.~ t,he highest, possi- hie systematisat, iotl is therefore consldcred worth- while. The present system cau ,,teal with any code properly if it is used unambignously in either a selfl,encc~delirrtitlng function @4'; codes indicai,- ins headings, para.o'aph marke,'s), sentenlce-intertml function (e.g. I+cmt change …

Content truncated to 500 words