Anomaly Detection and Preprocessing

Abstract

In sustainable environments, efficient anomaly (outlier) detection is essential to help monitor and control the system with the decision making process. Anomaly detection is an inherently difficult problem due to its decisions of what is normal and what is unusual, and the ability to distinguish between the two. Another serious difficulty is that the definition of normal can change. Sensor nodes in wireless sensor networks have limited energy resources and this hinders the dissemination of the gathered data to a central location. This stimulated our research to make use of the limited computational capabilities of these sensor nodes to build a normal model of the data gathered. In our research, our goal is to determine what is normal and what is abnormal and to distinguish between Normal & abnormal. We developed an algorithm called "Two-layered Data Capture Anomaly Detection". Our algorithm sends anomalies (2%) as well as roughly (2% or 4%) of normal data for further data processing and classification purposes. For testing purposes we also deployed three different machine learning and data mining tools. Three separate data sets were also used to validate the system. The performance of the proposed method is evaluated and compared with results obtained from the application of state of the art methods on the same data sets. In these tests our method provided very promising results.

Date of Award	2014
Original language	American English
Supervisor	U Zeyar Aung (Supervisor)