A Human Subject Study of Named Entity Recognition in Conversational Music Recommendation Queries

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1281–1296 May 2-6, 2023 ©2023 Association for Computational Linguistics A Human Subject Study of Named Entity Recognition (NER) in Conversational Music Recommendation Queries Elena V. Epure and Romain Hennequin Deezer Research, Paris, France research@deezer.com Abstract We conducted a human subject study of named entity recognition on a noisy corpus of conver- sational music recommendation queries, with many irregular and novel named entities. We evaluated the human NER linguistic behaviour in these challenging conditions and compared it with the most common NER systems nowa- days, fine-tuned transformers. Our goal was to learn about the task to guide the design of better evaluation methods and NER algorithms. The results showed that NER in our context was quite hard for both human and algorithms under a strict evaluation schema; humans had higher precision, while the model higher recall because of entity exposure especially during pre-training; and entity types had different error patterns (e.g. frequent typing errors for artists). The released corpus goes beyond predefined frames of interaction and can support future work in conversational music recommendation. 1 Introduction Music recommendation systems (RSs), fundamen- tal to streaming services nowadays, learn from user listening history or music content which artists or tracks to suggest next (Schedl et al., 2018). Most of these algorithms provide personalized music con- tent to the users when logging in the streaming apps or websites, or when triggered with pre-defined ut- terances via voice assistants (Ammari et al., 2019; Bontempelli et al., 2022). More recent conversa- tional RSs aim to help users to express their rec- ommendation needs by supporting interactions via queries in natural language (Jannach et al., 2021). However, despite existing in the scientific literature, such conversational RSs are not widely deployed because of multiple issues, one being NER. The processing of recommendation queries en- tails the extraction of named entity mentions (Moon et al., 2019; Rongali et al., 2020). This sub-task faces multiple challenges, even when queries are framed as pre-defined utterances. The transcrip- tions of the voice queries results in lower-case noisy text, often with misspellings (Muralidharan et al., 2021). The lack of capitalisation in entities and misspelled words are often present in text-based queries too (Cheng et al., 2021). Music entities, or those coming from the creative content domains, are highly irregular: they do not follow inherent patterns as it is the case with people’s names, and there is little to no separation between the vocab- ularies of entity and context words, especially for creative works (Derczynski et al., 2017) (e.g. com- mon words like "I" or "love" in track titles). Also, new music entities appear all the time. Major mu- sic streaming services ingest one new track almost every second (Ingham, 2021). Previous works have already shown that NER systems struggle with the aforementioned chal- lenges (Augenstein et al., 2017; Lin et al., 2020b; Epure and Hennequin, 2022). Thus, multiple ap- proaches have been proposed to address them, ei- …

Content truncated to 500 words