Towards Prediction with Partial Data in Sensor-based Big Data Applications

Many emerging big data applications such as in smart electric grids, transportation, avionics, manufacturing, and remote medical and environment monitoring involve sensors for tracking, monitoring, and control. These sensors are generally located at geographically dispersed locations and expected to periodically send back acquired information to centrally located nodes or processing centers. In many cases, the data from sensors is not available at central nodes at a frequency that is required for fast and real-time modeling and decision-making. For example, while many of these sensors are capable of collecting information at a high speed, logging data every minute or so, the physical limitations, specially latency, of the transmission networks limit the frequency at which data from sensors can be transmitted back to the central nodes. Also, consumers may limit frequent transmission of information from sensors located at their premises for security and privacy concerns. Finally, the data may not reach the central nodes due to faults in the sensors or transmission systems. All these scenarios raise the issue of data veracity in big data applications. While volume, variety, and velocity aspects of big data have been the focus of much recent research, veracity has received less attention. In this paper, we propose a novel solution to the problem of making short term predictions (up to a few hours ahead) in absence of real-time data from sensors. A key implication of our work is that by using real-time data from only a small subset of influential sensors, we are able to make predictions for all sensors. We thus reduce unnecessary transmissions from sensors and provide a practical solution to data veracity in many sensor based big data applications. We use real-world electricity consumption data from smart meters to empirically demonstrate the usefulness of our method. Keywords—data veracity, short-term prediction, prediction