Corpus Lexicography in a Wider Context

Proceedings of Recent Advances in Natural Language Processing, pages 353–359, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_041 353 Abstract This paper describes a set of tools that offers comprehensive solutions for corpus lexicography. The tools perform a range of tasks, including construction of corpus lexicon, integrating information from external dictionaries, internal analysis of the lexicon, and lexical analysis of the corpus. The set of tools is particularly useful for creating dictionaries for under- resourced languages. The tools are integrated in a general-purpose software that includes additional tools for various research tasks, such as linguistic development analysis. Equipped with a user-friendly interface, the described system can be easily incorporated in research in a variety of fields. 1 Introduction Corpus lexicography, a key component in modern dictionary compilation, has become increasingly powerful and efficient due to the development of various tools such as Word Sketch (Kilgarriff and Tugwell, 2002) and TickBox Lexicography (Kilgarriff et al., 2010). While corpus lexicography deserves further development in its own right, it is worthwhile considering it as an integral part of wider scientific missions. This paper describes several tools for corpus lexicography, whose design takes into consideration their contribution to linguistic research. The tools are integrated in a general-purpose linguistic software, where they can be readily applied in language acquisition studies, psycholinguistics, and other fields of research. 1 Website: https://chengafni.wordpress.com/cpa/ 2 The code is written in Visual Basic for Applications for MS Excel and is available under the GNU General Public 2 Preliminaries The described system is implemented in the Child Phonology Analyzer software (CPA; Gafni, 2015)1, which was built in MS Excel due to its popularity and user-friendly interface.2 Nevertheless, the concepts behind the system are general and can be implemented in various environments. The software can analyze corpora stored in various file formats, including Excel and plain-text files, as well as several special formats used in linguistic research: Praat’s TextGrids, CHAT transcription files, EAF annotation files, and XML schema for TalkBank data. The software converts analyzed corpora into Excel format and adds all analysis products to the Excel file. 2.1 Organizing the Data The described tools (“macros”) require that the corpus text be stored in a vector format. The text can be converted to a vector format using CPA’s “Data preparation” macro with the “Corpus tokenization” option, which segments the text into words. Segmentation is performed on the basis of blank spaces and additional word-dividing characters, which can be defined in CPA’s “Word dividers” table (Figure 1). There are two types of word dividers, which can be used for separating words even at the absence of a blank space: punctuation marks (e.g., comma) are deleted during segmentation, while final letters are not (final letters are special letter forms appearing only at word endings. See some examples from Hebrew in Figure 1). License. A version of this system for LibreOffice Calc is planned to appear in the future in order to free it from dependency on proprietary software. Corpus Lexicography in a Wider Context Chen Gafni Bar-Ilan …

Content truncated to 500 words