Data-driven Model Generalizability in Crosslinguistic Low-resource Morphological Segmentation

Data-driven Model Generalizability in Crosslinguistic Low-resource Morphological Segmentation Zoey Liu Department of Computer Science Boston College, USA zoey.liu@bc.edu Emily Prud’hommeaux Department of Computer Science Boston College, USA prudhome@bc.edu Abstract Common designs of model evaluation typi- cally focus on monolingual settings, where different models are compared according to their performance on a single data set that is assumed to be representative of all possible data for the task at hand. While this may be reasonable for a large data set, this assump- tion is difficult to maintain in low-resource scenarios, where artifacts of the data col- lection can yield data sets that are outliers, potentially making conclusions about model performance coincidental. To address these concerns, we investigate model generalizabil- ity in crosslinguistic low-resource scenarios. Using morphological segmentation as the test case, we compare three broad classes of models with different parameterizations, taking data from 11 languages across 6 language fami- lies. In each experimental setting, we evaluate all models on a first data set, then examine their performance consistency when introduc- ing new randomly sampled data sets with the same size and when applying the trained mod- els to unseen test sets of varying sizes. The results demonstrate that the extent of model generalization depends on the characteristics of the data set, and does not necessarily rely heavily on the data set size. Among the charac- teristics that we studied, the ratio of morpheme overlap and that of the average number of morphemes per word between the training and test sets are the two most prominent factors. Our findings suggest that future work should adopt random sampling to construct data sets with different sizes in order to make more responsible claims about model evaluation. 1 Introduction In various natural language processing (NLP) studies, when evaluating or comparing the per- formance of several models for a specific task, the current common practice tends to examine these models with one particular data set that is deemed representative. The models are usually trained and evaluated using a predefined split of training/test tests (with or without a develop- ment set), or using cross-validation. Given the results from the test set, authors typically con- clude that certain models are better than others (Devlin et al., 2019), report that one model is gen- erally suitable for particular tasks (Rajpurkar et al., 2016; Wang et al., 2018; Kondratyuk and Straka, 2019), or state that large pretrained models have ‘‘mastered’’ linguistic knowledge at human levels (Hu et al., 2020). Consequently, the ‘‘best’’ models, while not always surpassing other models by a large mar- gin, are subsequently cited as the new baseline or adopted for state-of-the-art comparisons in follow-up experiments. These new experiments might not repeat the comparisons that were carried out in the work that identified the best model(s). This model might be extended to similar tasks in different domains or languages, regardless of whether these new tasks have characteris- tics comparable to those where the model was demonstrated to be effective. That being said, just because a self-proclaimed …

Content truncated to 500 words