Distributed and Real-Time Query Framework for Processing Participatory Sensing Data Streams

Detecting emergent events and monitoring civil infrastructures in modern metropolitan cities by participatory sensing have recently been identified as a critical part of the public service management. With fast information distribution, multimedia messages (e.g., sound, images, videos, and texts) collected from citizens' smart devices can provide useful information to infer such emergencies by processing application level queries initialized from end terminals. This requires to establish an efficient, real-time processing systems for participatory sensing that can cope with both the dynamic queries and a variety of information with diverse attributes. To this end, in this paper, we first design a distributed and real-time query framework for event-based stream processing in participatory sensing, including a Storm-based real-time query engine, a messaging queue on Kafka, and a data persistence module based on HBase. Second, a dynamic indexing division method that is aware of the change of query attributes and volume is proposed. Third, we implement an application for civil infrastructure monitoring, and finally we evaluate the performance of proposed framework compared with existing approaches, simulation results of which show its advantages.