More Than Words: Collocation Retokenization for Latent Dirichlet Allocation Models

Findings of the Association for Computational Linguistics: ACL 2022, pages 2696 - 2704 May 22-27, 2022 c⃝2022 Association for Computational Linguistics More Than Words: Collocation Retokenization for Latent Dirichlet Allocation Models Jin Cheevaprawatdomrong Chulalongkorn University jin236248@gmail.com Alexandra Schofield Harvey Mudd College xanda@cs.hmc.edu Attapol T. Rutherford ∗ Chulalongkorn University attapol.t@chula.ac.th Abstract Traditionally, Latent Dirichlet Allocation (LDA) ingests words in a collection of doc- uments to discover their latent topics us- ing word-document co-occurrences. Previous studies show that representing bigrams collo- cations in the input can improve topic coher- ence in English. However, it is unclear how to achieve the best results for languages with- out marked word boundaries such as Chinese and Thai. Here, we explore the use of reto- kenization based on chi-squared measures, t- statistics, and raw frequency to merge frequent token ngrams into collocations when prepar- ing input to the LDA model. Based on the goodness of fit and the coherence metric, we show that topics trained with merged tokens result in topic keys that are clearer, more coher- ent, and more effective at distinguishing topics than those of unmerged models. 1 Introduction Latent Dirichlet allocation (LDA) models provide useful insights into themes and trends in a large text collection through the unsupervised inference of topics, or probability distributions over unigram word types in the corpus (Blei et al., 2003). Topics from these models are often interpreted based on their highest-probability words, with documents expressed as vectors of proportions of each topic. Unfortunately, the context in which these tokens arise can be obscured in the bag-of-words render- ing of text as unigram counts in documents. For instance, a topic with high probabilities of both “coffee” and “table” is tempting to interpret as fo- cusing on the furniture item “coffee table”, but both words could be frequent in a discussion of cafes containing no coffee tables. This problem is ampli- fied in languages without marked word boundaries, such as Chinese and Thai: while existing tokeniz- ers in these languages can segment characters into ∗Corresponding author words, there is always a question about to what extent the tokenizers should group words together. Words that have been segmented by tokenizers may not express the concept of the original text if they were found as parts of collocations. Meaningful interpretation of topics can be lost without careful recombination of these words. We hypothesize that the morphology of the lan- guage should play an important role in determin- ing the suitable pre-processing steps that would improve the results of topic models. The main morphological types we consider are synthetic lan- guage and analytic language. Synthetic languages use many morphemes to compose a word and can be further divided into fusional and agglutinative languages. Fusional languages such as German differ from agglutinative languages such as Ko- rean and Japanese: a single morpheme in fusional languages can code for many morphosyntactic fea- tures. On the other hand, analytic languages such as Thai and Chinese convey meanings by relating many words together, and morphological devices are …

Content truncated to 500 words