Improving Segmentation for Technical Support Problems

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3125–3134 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3125 Improving Segmentation for Technical Support Problems Kushal Chauhan ∗ ABV-IIITM, Gwalior kushalchauhan98@gmail.com Abhirut Gupta † IBM Research AI abhirut.91@gmail.com Abstract Technical support problems are often long and complex. They typically contain user descrip- tions of the problem, the setup, and steps for attempted resolution. Often they also contain various non-natural language text ele- ments like outputs of commands, snippets of code, error messages or stack traces. These elements contain potentially crucial informa- tion for problem resolution. However, they cannot be correctly parsed by tools designed for natural language. In this paper, we ad- dress the problem of segmentation for techni- cal support questions. We formulate the prob- lem as a sequence labelling task, and study the performance of state of the art approaches. We compare this against an intuitive contex- tual sentence-level classification baseline, and a state of the art supervised text-segmentation approach. We also introduce a novel com- ponent of combining contextual embeddings from multiple language models pre-trained on different data sources, which achieves a marked improvement over using embeddings from a single pre-trained language model. Fi- nally, we also demonstrate the usefulness of such segmentation with improvements on the downstream task of answer retrieval. 1 Introduction Problems, reported by users of software or hard- ware products - called tickets or cases, are often long and complex. Along with a description of the problem, users often report the setup, steps they have tried at mitigating the problem, and explicit requests. These problems also contain various non- natural language elements like snippets of code or commands tried, outputs of commands or software tools, error messages or stack traces, contents of log files or configuration files, and lists of key-value ∗Work done at IBM Research during a summer internship †Now at Google Figure 1: Various non-natural language segments la- belled from a problem on AskUbuntu pairs. Figure 1 shows a sample support problem from AskUbuntu 1, where all such segments are labeled. While these segments are important sources of information for the human reader, they are difficult to handle for systems built to automatically answer support problems. As noted in Gupta et al. (2018), the non-natural language segments lead to parsing mistakes, and errors in the understanding of support problems. Correctly identifying these segments can also augment problem understanding. For instance, a retrieval engine with error messages and their solutions indexed in distinct fields would return better results with a fielded query containing just the error message from the ticket. Specialized tools for log analysis (He et al., 2016) could also be 1https://askubuntu.com/ 3126 run specifically on the identified log segment of problems. In this paper, we aim to address the problem of identifying and extracting these non-natural lan- guage segments from support tickets. In particular, we choose to focus on the following six segment labels which appear often in support tickets (also shown in Figure …

Content truncated to 500 words