Weather Data Analysis and Sensor Fault Detection Using An Extended IoT Framework with Semantics, Big Data, and Machine Learning

Abstract

In recent years, big data and Internet of Things (IoT) implementations started getting more attention. Researchers focused on developing big data analytics solutions using machine learning models. Machine learning is a rising trend in this field due to its ability to extract hidden features and patterns even in highly complex datasets. In this study, we used our Big Data IoT Framework in a weather data analysis use case. We implemented weather clustering and sensor anomaly detection using a publicly available dataset. We provided the implementation details of each framework layer (acquisition, ETL, data processing, learning and decision) for this particular use case. Our chosen learning model within the library is Scikit-Learn based k-means clustering. The data analysis results indicate that it is possible to extract meaningful information from a relatively complex dataset using our framework.