CanVEC - the Canberra Vietnamese-English Code-switching Natural Speech Corpus

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4121–4129 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 4121 CanVEC – the Canberra Vietnamese-English Code-switching Natural Speech Corpus Li Nguyen† and Christopher Bryant‡ Theoretical and Applied Linguistics† Department of Computer Science and Technology‡ University of Cambridge, Cambridge, United Kingdom {nhbn2, cjb255}@cam.ac.uk Abstract This paper introduces the Canberra Vietnamese-English Code-switching corpus (CanVEC), an original corpus of natural mixed speech that we semi-automatically annotated with language information, part of speech (POS) tags and Vietnamese translations. The corpus, which was built to inform a sociolinguistic study on language variation and code-switching, consists of 10 hours of recorded speech (87k tokens) between 45 Vietnamese-English bilinguals living in Canberra, Australia. We describe how we collected and annotated the corpus by pipelining several monolingual toolkits to considerably speed up the annotation process. We also describe how we evaluated the automatic annotations to ensure corpus reliability. We make the corpus available for research purposes. Keywords: Codeswitching, Vietnamese, Natural Speech, Automatic Annotation 1. Introduction Code-switching is the linguistic phenomenon where a speaker uses two or more languages in a single conversa- tion or utterance; for example: (1) mỗi each group phải must có have a different focus ‘Each group must have a different focus’ Although this language-mixing phenomenon has been stud- ied extensively in linguistics (Poplack, 1980; Myers- Scotton, 1997; Muysken, 2000), it has received much less attention in the Natural Language Processing (NLP) com- munity, where it remains a serious challenge (Solorio and Liu, 2008; Nguyen and Do˘gru¨oz, 2013; Li and Fung, 2014; Molina et al., 2016; C¸ etino˘glu et al., 2016). This is largely because of a lack of availability of large, high-quality code- switching corpora, and the reality that monolingual tech- niques often struggle with input from multiple languages (Hamed et al., 2017; Soto and Hirschberg, 2018). A related issue is that, of the code-switching corpora that are available, the majority are textual and consist only of web documents (Hamed et al., 2017) or social media posts from platforms such as Twitter (Maharjan et al., 2015; Jurgens et al., 2014; Mave et al., 2018) and Facebook (Bali et al., 2014; Barman et al., 2014). Given that code- switching occurs most frequently and naturally in infor- mal speech however (Labov, 2004; Cacoullos and Travis, 2018; Deuchar et al., 2018; Nguyen, 2018), the faithful- ness of such code-switching utterances is somewhat ques- tionable. In particular, although text is often considered a more canonical form of language use than speech, Adouane et al. (2018) points out that a heavy reliance on written input as training data is highly problematic, as it implic- itly assumes all forms of languages are uniform and mono- lingual. This is especially not the case for code-switching however, where language use encompasses many forms of variation and speakers may use different phonological, lexi- cal and syntactic inputs from multiple languages (Cacoullos and Travis, 2018; Deuchar et al., 2018). Although the amount of time and effort …

Content truncated to 500 words