Unsupervised Preference-Aware Language Identification

Findings of the Association for Computational Linguistics: ACL 2022, pages 3847 - 3852 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Unsupervised Preference-Aware Language Identification Xingzhang Ren Baosong Yang∗ Dayiheng Liu Haibo Zhang Xiaoyu Lv Liang Yao Jun Xie DAMO Academy, Alibaba Group {xingzhang.rxz, yangbaosong.ybs, liudayiheng.ldyh, zhanhui.zhb, anzhi.lxy, yaoliang.yl, qingjing.xj}@alibaba-inc.com Abstract Recognizing the language of ambiguous texts has become a main challenge in language iden- tification (LID). When using multilingual ap- plications, users have their own language pref- erences, which can be regarded as external knowledge for LID. Nevertheless, current stud- ies do not consider the inter-personal varia- tions due to the lack of user annotated training data. To fill this gap, we introduce preference- aware LID and propose a novel unsupervised learning strategy. Concretely, we construct pseudo training set for each user by extracting training samples from a standard LID corpus according to his/her historical language distri- bution. Besides, we contribute the first user la- beled LID test set called “U-LID”. Experimen- tal results reveal that our model can incarnate user traits and significantly outperforms exist- ing LID systems on handling ambiguous texts. Our code and benchmark have been released.1 1 Introduction Language identification (LID) is widely applied in a range of web services where a multitude of languages may be presented, such as translation systems, search engines, and social media (Yao et al., 2020a; Sun et al., 2020; Li et al., 2020; Bi et al., 2020; Xu et al., 2021). It predicts the natural language that a text is written in, and decides which language-specific model to invoke in downstream natural language processing (NLP) tasks (Lui et al., 2014; Yao et al., 2020b; Tambi et al., 2020). Several recent studies have well tackled LID by designing a feature set for a traditional or neural classifier (Kocmi and Bojar, 2017; Vo and Khoury, 2020; Jauhiainen et al., 2021). However, these researches merely explore textual information re- gardless of external knowledge about the user. In a real-world scenario, there exists large amount of ∗Baosong Yang is the corresponding author. 1https://github.com/xzhren/PreferenceAwareLID User Input Text Label Prefer. Baseline Ours velo es (veil) es en es velo fr (bike) fr en fr fundas huawei y7 es (huawei y7 cases) es en es kello kitty en (hello kitty) de it en Table 1: Examples of ambiguous text that are difficult to be accurately recognized. “Label” shows the lan- guage label that is annotated by a user and conforms to his/her input intention. “Prefer.” denotes the lan- guage most frequently used by the corresponding user. “Baseline” and “Ours” indicate the predictions of base- line LID system and the proposed model, respectively. ambiguous user inputs, such as texts with false- friend, code-switching, and misspelling, as shown in Table 1. On the one hand, the languages of these texts are difficult (even impossible) to be explic- itly identified without external knowledge. On the other hand, for different users, a good LID should flexibly give different results to the same ambigu- ous input, thus conforming to users’ intention …

Content truncated to 500 words