LITE: Intent-based Task Representation Learning Using Weak Supervision

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2410 - 2424 July 10-15, 2022 ©2022 Association for Computational Linguistics LITE: Intent-based Task Representation Learning Using Weak Supervision Naoki Otani†∗, Michael Gamon‡, Sujay Kumar Jauhar‡, Mei Yang‡, Sri Raghu Malireddi‡, Oriana Riva‡ †Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA, USA ‡Microsoft Research, Redmond, WA, USA †notani@cs.cmu.edu ‡{mgamon,sjauhar,meiyang,sri.malireddi,oriana.riva}@microsoft.com Abstract Users write to-dos as personal notes to them- selves, about things they need to complete, re- member or organize. To-do texts are usually short and under-specified, which poses a chal- lenge for current text representation models. Yet, understanding and representing their mean- ing is the first step towards providing intelli- gent assistance for to-do management. We ad- dress this problem by proposing a neural multi- task learning framework, LITE, which extracts representations of English to-do tasks with a multi-head attention mechanism on top of a pre-trained text encoder. To adapt representa- tion models to to-do texts, we collect weak- supervision labels from semantically rich ex- ternal resources (e.g., dynamic commonsense knowledge bases), following the principle that to-do tasks with similar intents have similar labels. We then train the model on multiple generative/predictive training objectives jointly. We evaluate our representation model on four downstream tasks and show that our approach consistently improves performance over base- line models, achieving error reduction of up to 38.7%. 1 Introduction Task management tools are widely used to orga- nize tasks and keep track of progress in work and daily lives. Examples include Microsoft To-do, Todoist, Trello, and digital assistants such as Ama- zon Alexa and Google Assistant. Machine learn- ing techniques can automate various aspects of task management such as task creation (Mukherjee et al., 2020), organization (Landes and Di Eugenio, 2018), prioritization, and decomposition of com- plex tasks (Nouri et al., 2020; Zhang et al., 2021). The goal of this work is to develop a single, general-purpose encoding system that converts to- do task texts into real-valued vector representa- ∗This work was done while the first author was an intern at Microsoft Research. Vector representations of to-do tasks Groceries milk eggs carrots Today Pay credit card Call Mom Pick up Chris at 4 Description List name Intelligent assistance task detection, organization, recommendation, … User-generated to-do texts Figure 1: Our aim is to encode user-generated to-do texts (list names and descriptions) into vector represen- tations so that machine learning systems can provide various kinds of intelligent assistance. tions (Fig. 1). Using one encoding system for mul- tiple task functions (task detection, organization, recommendation, etc.) as opposed to having mul- tiple dedicated encoders saves the computational costs of updating models regularly and encoding texts from millions of users. Representation learning has been extensively studied in natural language processing (Camacho- Collados and Pilehvar, 2018). Adapting models pre-trained on massive amounts of raw texts to a target domain or a task has become common prac- tice (Qiu et al., 2020), with many publicly available pre-trained models such as …

Content truncated to 500 words