Automatic Discovery of Heterogeneous Machine Learning Pipelines: An Application to Natural Language Processing

Proceedings of the 28th International Conference on Computational Linguistics, pages 3558–3568 Barcelona, Spain (Online), December 8-13, 2020 3558 Automatic Discovery of Heterogeneous Machine Learning Pipelines: An Application to Natural Language Processing Suilan Estevez-Velarde1, Yoan Guti´errez2,3, Andr´es Montoyo2,3, and Yudivi´an Almeida-Cruz1 1School of Math and Computer Science, University of Havana, Cuba {sestevez,yudy}@matcom.uh.cu 2University Institute for Computing Research (IUII), University of Alicante, Spain 3Department of Languages and Computing Systems, University of Alicante, Spain {ygutierrez,montoyo}@dlsi.ua.es Abstract This paper presents AutoGOAL, a system for automatic machine learning (AutoML) that uses heterogeneous techniques. In contrast with existing AutoML approaches, our contribution can automatically build machine learning pipelines that combine techniques and algorithms from different frameworks, including shallow classifiers, natural language processing tools, and neural networks. We define the heterogeneous AutoML optimization problem as the search for the best sequence of algorithms that transforms specific input data into the desired output. This provides a novel theoretical and practical approach to AutoML. Our proposal is experimentally evaluated in diverse machine learning problems and compared with alternative approaches, showing that it is competitive with other AutoML alternatives in standard benchmarks. Furthermore, it can be applied to novel scenarios, such as several NLP tasks, where existing alternatives cannot be directly deployed. The system is freely available and includes in-built compatibility with a large number of popular machine learning frameworks, which makes our approach useful for solving practical problems with relative ease and effort. 1 Introduction The continued development of new machine learning algorithms and techniques, and the widely available tools and datasets have brought new opportunities and challenges for researchers and practitioners in both academia and industry. Selecting the best possible strategy to solve a machine learning problem is increasingly difficult partly because it requires long experimentation times and deep technical knowledge. In this scenario, Automatic Machine Learning (AutoML) has risen to prominence as it provides tools based on specific technologies to efficiently search large spaces of machine learning pipelines, such as Auto-Weka (Thornton et al., 2013), Auto-Sklearn (Feurer et al., 2015) and Auto-Keras (Jin et al., 2018). However, practical problems often require combining and comparing heterogeneous algorithms implemented with different underlying technologies. Natural language processing is one scenario where the space of possible techniques to apply varies widely between different tasks, from preprocessing to representation and actual classification. Performing AutoML in an heterogeneous scenario like this is complex because the necessary solution could comprise non-compatible tools and libraries. This would require all algorithms to agree on a common protocol that enables the output of an algorithm to be shared as inputs to any other. Table 1 contrasts several existing AutoML systems with the system proposed in this research in terms of their capabilities of dealing with heterogeneous scenarios. This evaluation does not attempt to compare AutoML systems in terms of their overall performance, capacity or applicability, but rather with respect to the specific issue of dealing with multiple, heterogeneous algorithms directly. Several popular AutoML systems are based on specific machine learning libraries, such as Auto-Sklearn (Feurer et al., …

Content truncated to 500 words