Dynamic autoselection and autotuning of machine learning models for cloud network analytics

Rupesh Raj Karn, Prabhakar Kudva, Ibrahim Abe M. Elfadel

Research output: Contribution to journalArticlepeer-review

18 Scopus citations

Abstract

Cloud network monitoring data is dynamic and distributed. Signals to monitor the cloud can appear, disappear or change their importance and clarity over time. Machine learning (ML) models tuned to a given data set can therefore quickly become inadequate. A model might be highly accurate at one point in time but may lose its accuracy at a later time due to changes in input data and their features. Distributed learning with dynamic model selection is therefore often required. Under such selection, poorly performing models (although aggressively tuned for the prior data) are retired or put on standby while new or standby models are brought in. The well-known method of Ensemble ML (EML) may potentially be applied to improve the overall accuracy of a family of ML models. Unfortunately, EML has several disadvantages, including the need for continuous training, excessive computational resources, requirement for large training datasets, high risks of overfitting, and a time-consuming model-building process. In this paper, we propose a novel cloud methodology for automatic ML model selection and tuning that automates model building and selection and is competitive with existing methods. We use unsupervised learning to better explore the data space before the generation of targeted supervised learning models in an automated fashion. In particular, we create a Cloud DevOps architecture for autotuning and selection based on container orchestration and messaging between containers, and take advantage of a new autoscaling method to dynamically create and evaluate instantiations of ML algorithms. The proposed methodology and tool are demonstrated on cloud network security datasets.

Original languageBritish English
Article number8500348
Pages (from-to)1052-1064
Number of pages13
JournalIEEE Transactions on Parallel and Distributed Systems
Volume30
Issue number5
DOIs
StatePublished - 1 May 2019

Keywords

  • autoselection
  • autotuning
  • classification
  • Cloud analytics
  • cloud DevOps
  • clustering
  • containers
  • decision feedback
  • distributed learning
  • docker
  • ensemble learning
  • Kafka
  • machine learning

Fingerprint

Dive into the research topics of 'Dynamic autoselection and autotuning of machine learning models for cloud network analytics'. Together they form a unique fingerprint.

Cite this