Edge Video Analytics for Public Safety: A Review

With the installation of enormous public safety and transportation infrastructure cameras, video analytics has come to play an essential part in public safety. Typically, video analytics is to collectively leverage the advanced computer vision (CV) and artificial intelligence (AI) to solve the four-W problem. That is to identify Who has done something (What) at a specific place (Where) at some time (When). According to the difference of latency requirements, video analytics can be applied to postevent retrospective analysis, such as archive management, search, forensic investigation and real-time live video stream analysis, such as situation awareness, alerting, and interested object (criminal suspect/missing vehicle) detection. The latter is characterized as having higher requirements on hardware resources as the sophisticated image processing algorithms under the hood. However, analyzing large-scale live video streams on the Cloud is impractical as the edge solution that conducts the video analytics on (or close to) the camera provides a silvering light. Analyzing live video streams on the edge is not trivial due to the constrained hardware resources on edge. The AI-dominated video analytics requires higher bandwidth, consumes considerable CPU/GPU resources for processing, and demands larger memory for caching. In this paper, we review the applications, algorithms, and solutions that have been proposed recently to facilitate edge video analytics for public safety.

[1]  Michael Felsberg,et al.  The Visual Object Tracking VOT2015 Challenge Results , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[2]  Youmin Zhang,et al.  Fire detection using infrared images for UAV-based forest fire surveillance , 2017, 2017 International Conference on Unmanned Aircraft Systems (ICUAS).

[3]  A. Bayen,et al.  Intelligent Transportation Systems and Infrastructure A Series of Briefs for Smart Investments , 2017 .

[4]  Meng Joo Er,et al.  A Novel Approach for Vehicle Detection Using an AND–OR-Graph-Based Multiscale Model , 2015, IEEE Transactions on Intelligent Transportation Systems.

[5]  Yoav Freund,et al.  Boosting a weak learning algorithm by majority , 1995, COLT '90.

[6]  Z. Zivkovic.  Improved adaptive Gaussian mixture model for background subtraction , 2004, ICPR 2004.

[7]  Xin Wang,et al.  Deep Reinforcement Learning for Visual Object Tracking in Videos , 2017, ArXiv.

[8]  Jian Sun,et al.  Blessing of Dimensionality: High-Dimensional Feature and Its Efficient Compression for Face Verification , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[9]  Xiaogang Wang,et al.  Deep Learning Face Representation from Predicting 10,000 Classes , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  Xiaopei Wu,et al.  Edge computing enabled smart firefighting: opportunities and challenges , 2017, HotWeb.

[11]  Paramvir Bahl,et al.  VideoEdge: Processing Camera Streams using Hierarchical Clusters , 2018, 2018 IEEE/ACM Symposium on Edge Computing (SEC).

[12]  Nikolaos Grammalidis,et al.  Spatio-Temporal Flame Modeling and Dynamic Texture Analysis for Automatic Video-Based Fire Detection , 2015, IEEE Transactions on Circuits and Systems for Video Technology.

[13]  Sameer Qazi,et al.  UAV based real time video surveillance over 4G LTE , 2015, 2015 International Conference on Open Source Systems & Technologies (ICOSST).

[14]  Paramvir Bahl,et al.  Real-Time Video Analytics: The Killer App for Edge Computing , 2017, Computer.

[15]  Ion Stoica,et al.  Chameleon: scalable adaptation of video analytics , 2018, SIGCOMM.

[16]  Alessandro Verri,et al.  Motion Segmentation from Optical Flow , 1989, Alvey Vision Conference.

[17]  Quan Zhang,et al.  Firework: Data Processing and Sharing for Hybrid Cloud-Edge Analytics , 2018, IEEE Transactions on Parallel and Distributed Systems.

[18]  Adel M. Alimi,et al.  Deep Learning System for Automatic License Plate Detection and Recognition , 2017, 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR).

[19]  Luca Bertinetto,et al.  Fully-Convolutional Siamese Networks for Object Tracking , 2016, ECCV Workshops.

[20]  Christian Jutten,et al.  Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects , 2015, Proceedings of the IEEE.

[21]  Gang Hua,et al.  A convolutional neural network cascade for face detection , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Yang Yu,et al.  Real-time vehicle detection using histograms of oriented gradients and AdaBoost classification , 2016 .

[23]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Hao Wang,et al.  Face R-CNN , 2017, ArXiv.

[25]  O. Muensterer,et al.  Google Glass in pediatric surgery: an exploratory study. , 2014, International journal of surgery.

[26]  Hussein Zedan,et al.  The implementation of an intelligent and video-based fall detection system using a neural network , 2014, Appl. Soft Comput..

[27]  Antoine Widmer,et al.  Using Google Glass to enhance pre-hospital care , 2014 .

[28]  Takeo Kanade,et al.  Neural Network-Based Face Detection , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[29]  Bo Chen,et al.  Teleconsultant: Communication and Analysis of Wearable Videos in Emergency Medical Environments , 2017, ACM Multimedia.

[30]  Symeon Papavassiliou,et al.  Edge Computing in IoT Ecosystems for UAV-Enabled Early Fire Detection , 2018, 2018 IEEE International Conference on Smart Computing (SMARTCOMP).

[31]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Zhiming Cui,et al.  A Survey on Video-based Vehicle Behavior Analysis Algorithms , 2012, J. Multim..

[33]  Fei-Yue Wang,et al.  Data-Driven Intelligent Transportation Systems: A Survey , 2011, IEEE Transactions on Intelligent Transportation Systems.

[34]  Zhuo Chen,et al.  Bandwidth-Efficient Live Video Analytics for Drones Via Edge Computing , 2018, 2018 IEEE/ACM Symposium on Edge Computing (SEC).

[35]  Paramvir Bahl,et al.  Live Video Analytics at Scale with Approximation and Delay-Tolerance , 2017, NSDI.

[36]  Kazuhiko Sumi,et al.  A robust background subtraction method for changing background , 2000, Proceedings Fifth IEEE Workshop on Applications of Computer Vision.

[37]  Marjorie Skubic,et al.  Fall Detection in Homes of Older Adults Using the Microsoft Kinect , 2015, IEEE Journal of Biomedical and Health Informatics.

[38]  Trevor N. Mudge,et al.  Neurosurgeon: Collaborative Intelligence Between the Cloud and Mobile Edge , 2017, ASPLOS.

[39]  Zhihai He,et al.  A real-time system for in-home activity monitoring of elders , 2009, 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[40]  Quan Zhang,et al.  Distributed Collaborative Execution on the Edges and Its Application to AMBER Alerts , 2018, IEEE Internet of Things Journal.

[41]  Sung Wook Baik,et al.  Early fire detection using convolutional neural networks during surveillance for effective disaster management , 2017, Neurocomputing.

[42]  Henning Müller,et al.  Facilitating medical information search using Google Glass connected to a content-based medical image retrieval system , 2014, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[43]  Luis Miguel Bergasa,et al.  Expert video-surveillance system for real-time detection of suspicious behaviors in shopping malls , 2015, Expert Syst. Appl..

[44]  T. Poggio,et al.  A Contour-Based Moving Object Detection and Tracking , 2005, 2005 IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance.

[45]  B. Yogameena,et al.  Synoptic video based human crowd behavior analysis for forensic video surveillance , 2015, 2015 Eighth International Conference on Advances in Pattern Recognition (ICAPR).

[46]  James Philbin,et al.  FaceNet: A unified embedding for face recognition and clustering , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[47]  Xiaopei Wu,et al.  STREMS: A Smart Real-Time Solution toward Enhancing EMS Prehospital Quality , 2017, 2017 IEEE/ACM International Conference on Connected Health: Applications, Systems and Engineering Technologies (CHASE).

[48]  Shuming Tang,et al.  Traffic-incident detection-algorithm based on nonparametric regression , 2005, IEEE Transactions on Intelligent Transportation Systems.

[49]  Silvio Savarese,et al.  Learning to Track at 100 FPS with Deep Regression Networks , 2016, ECCV.

[50]  Henning Müller,et al.  Using smart glasses in medical emergency situations, a qualitative pilot study , 2016, 2016 IEEE Wireless Health (WH).

[51]  Liusheng Huang,et al.  A robust and efficient method for license plate recognition , 2018, 2018 24th International Conference on Pattern Recognition (ICPR).

[52]  Jing Xin,et al.  An early forest fire detection method based on unmanned aerial vehicle vision , 2018, 2018 Chinese Control And Decision Conference (CCDC).

[53]  Juan López,et al.  Architecture for a helicopter-based unmanned aerial systems wildfire surveillance system , 2011 .

[54]  Paramvir Bahl,et al.  The Case for VM-Based Cloudlets in Mobile Computing , 2009, IEEE Pervasive Computing.

[55]  Bing-Fei Wu,et al.  Adaptive Vehicle Detector Approach for Complex Environments , 2012, IEEE Transactions on Intelligent Transportation Systems.

[56]  Jun Fang,et al.  A Saliency-Based Method for Early Smoke Detection in Video Sequences , 2016 .

[57]  Daijin Kim,et al.  Moving object detection under free-moving camera , 2010, 2010 IEEE International Conference on Image Processing.

[58]  Brian Y. Lattimer,et al.  Real-time probabilistic classification of fire and smoke using thermal imagery for intelligent firefighting robot , 2015 .

[59]  M. Amaç Güvensan,et al.  Driver Behavior Analysis for Safe Driving: A Survey , 2015, IEEE Transactions on Intelligent Transportation Systems.

[60]  Aakanksha Chowdhery,et al.  Networked Drone Cameras for Sports Streaming , 2017, 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS).

[61]  Ye Li,et al.  Vehicle detection based on And-Or Graph and Hybrid Image Templates for complex urban traffic conditions , 2015 .

[62]  Hui Lin,et al.  Image-based seat belt detection , 2011, Proceedings of 2011 IEEE International Conference on Vehicular Electronics and Safety.

[63]  Aníbal Ollero,et al.  Journal of Intelligent & Robotic Systems manuscript No. (will be inserted by the editor) An Unmanned Aircraft System for Automatic Forest Fire Monitoring and Measurement , 2022 .

[64]  Youmin Zhang,et al.  Vision-based forest fire detection in aerial images for firefighting using UAVs , 2016, 2016 International Conference on Unmanned Aircraft Systems (ICUAS).

[65]  Xiaofei Wang,et al.  Fast moving object detection with non-stationary background , 2012, Multimedia Tools and Applications.

[66]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[67]  Zhongyuan Wang,et al.  Smart Monitoring Cameras Driven Intelligent Processing to Big Surveillance Video Data , 2018, IEEE Transactions on Big Data.

[68]  Lisa M. Brown,et al.  IBM smart surveillance system (S3): event based video surveillance system with an open and extensible framework , 2008, Machine Vision and Applications.

[69]  Feiniu Yuan,et al.  A Deep Normalization and Convolutional Neural Network for Image Smoke Detection , 2017, IEEE Access.

[70]  Guodong Guo,et al.  Face recognition by support vector machines , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[71]  Sari I. Lakkis,et al.  IoT based emergency and operational services in medical care systems , 2017, 2017 Internet of Things Business Models, Users, and Networks.

[72]  Srdjan M. Lukic,et al.  Review of driving conditions prediction and driving style recognition based control algorithms for hybrid electric vehicles , 2011, 2011 IEEE Vehicle Power and Propulsion Conference.

[73]  Brian Y. Lattimer,et al.  Robotic Fire Suppression Through Autonomous Feedback Control , 2017 .

[74]  Xiaogang Wang,et al.  Deep Learning Face Representation by Joint Identification-Verification , 2014, NIPS.

[75]  Tarik Taleb,et al.  UAV-Based IoT Platform: A Crowd Surveillance Use Case , 2017, IEEE Communications Magazine.

[76]  Gajendra Sharma,et al.  Performance Analysis of Vehicle Number Plate Recognition System Using Template Matching Techniques , 2018 .

[77]  K. A. Joshi,et al.  A Survey on Moving Object Detection and Tracking in Video Surveillance System , 2012 .

[78]  Nick Antonopoulos,et al.  Video Stream Analysis in Clouds: An Object Detection and Classification Framework for High Performance Video Analytics , 2019, IEEE Transactions on Cloud Computing.

[79]  Bin Yang,et al.  Person Recognition Based on Micro-Doppler and Thermal Infrared Camera Fusion for Firefighting , 2018, 2018 21st International Conference on Information Fusion (FUSION).

[80]  Max Mignotte,et al.  Fall Detection from Depth Map Video Sequences , 2011, ICOST.

[81]  Aura Conci,et al.  Video-Based Distance Traffic Analysis: Application to Vehicle Tracking and Counting , 2011, Computing in Science & Engineering.

[82]  Limin Wang,et al.  Temporal Action Detection with Structured Segment Networks , 2017, International Journal of Computer Vision.

[83]  Yu Qiao,et al.  Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks , 2016, IEEE Signal Processing Letters.

[84]  James Nightingale,et al.  A UAV-Cloud System for Disaster Sensing Applications , 2015, 2015 IEEE 81st Vehicular Technology Conference (VTC Spring).

[85]  Ming Yang,et al.  DeepFace: Closing the Gap to Human-Level Performance in Face Verification , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[86]  Jian Sun,et al.  Bayesian Face Revisited: A Joint Formulation , 2012, ECCV.

[87]  Yong-Kul Ki,et al.  A Traffic Accident Recording and Reporting Model at Intersections , 2007, IEEE Transactions on Intelligent Transportation Systems.

[88]  Rita Cucchiara,et al.  Intelligent Video Surveillance as a Service , 2013, Intelligent Multimedia Surveillance.

[89]  Pan Wang,et al.  Smoke Detection Based on Deep Convolutional Neural Networks , 2016, 2016 International Conference on Industrial Informatics - Computing Technology, Intelligent Technology, Industrial Information Integration (ICIICII).

[90]  M. Turk,et al.  Eigenfaces for Recognition , 1991, Journal of Cognitive Neuroscience.

[91]  Brian Y. Lattimer,et al.  Bayesian estimation based real-time fire-heading in smoke-filled indoor environments using thermal imagery , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[92]  Farhat Fnaiech,et al.  Convolutional neural network for video fire and smoke detection , 2016, IECON 2016 - 42nd Annual Conference of the IEEE Industrial Electronics Society.

[93]  Jean Meunier,et al.  Robust Video Surveillance for Fall Detection Based on Human Shape Deformation , 2011, IEEE Transactions on Circuits and Systems for Video Technology.

[94]  Brian Y. Lattimer,et al.  Firefighting Robot Stereo Infrared Vision and Radar Sensor Fusion for Imaging through Smoke , 2015 .

[95]  Xiaogang Wang,et al.  DeepID3: Face Recognition with Very Deep Neural Networks , 2015, ArXiv.

[96]  Dan Roth,et al.  Learning to detect objects in images via a sparse, part-based representation , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[97]  Ergun Erçelebi,et al.  Automatic Vehicle Identification by Plate Recognition , 2007 .

[98]  David J. Kriegman,et al.  Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection , 1996, ECCV.

[99]  Jean Meunier,et al.  Elderly fall detection system based on multiple shape features and motion analysis , 2018, 2018 International Conference on Intelligent Systems and Computer Vision (ISCV).

[100]  Dimitrios Makris,et al.  Fall detection system using Kinect’s infrared sensor , 2014, Journal of Real-Time Image Processing.

[101]  Weisong Shi,et al.  The Promise of Edge Computing , 2016, Computer.

[102]  Giuseppe Andreoni,et al.  SMARTA: Smart Ambiente and Wearable Home Monitoring for Elderly , 2015, IoT 360.

[103]  Kyung-Sup Kwak,et al.  The Internet of Things for Health Care: A Comprehensive Survey , 2015, IEEE Access.

[104]  Weisong Shi,et al.  Edge-based personal computing services: fall detection as a pilot study , 2018, Computing.

[105]  A. Enis Çetin,et al.  Covariance matrix-based fire and flame detection method in video , 2012, Machine Vision and Applications.

[106]  Hong Zhong,et al.  Demo Abstract: EVAPS: Edge Video Analysis for Public Safety , 2016, 2016 IEEE/ACM Symposium on Edge Computing (SEC).

[107]  Weisong Shi,et al.  SafeShareRide: Edge-Based Attack Detection in Ridesharing Services , 2018, 2018 IEEE/ACM Symposium on Edge Computing (SEC).

[108]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[109]  Manish Kumar,et al.  SIERRA Team Flight of Zephyr UAS at West Virginia Wild Land Fire Burn , 2012, Infotech@Aerospace.

[110]  Youmin Zhang,et al.  UAV-based forest fire detection and tracking using image processing techniques , 2015, 2015 International Conference on Unmanned Aircraft Systems (ICUAS).

[111]  Tie Qiu,et al.  Fog Computing Based Face Identification and Resolution Scheme in Internet of Things , 2017, IEEE Transactions on Industrial Informatics.

[112]  Aakanksha Chowdhery,et al.  The Design and Implementation of a Wireless Video Surveillance System , 2015, MobiCom.

[113]  Cristina P. Santos,et al.  Vision based automatic traffic condition interpretation , 2010, 2010 8th IEEE International Conference on Industrial Informatics.

[114]  Alex Mihailidis,et al.  An intelligent emergency response system: preliminary development and testing of automated fall detection , 2005, Journal of telemedicine and telecare.

[115]  Weisong Shi,et al.  Edge Computing: Vision and Challenges , 2016, IEEE Internet of Things Journal.

[116]  Hong Zhong,et al.  How Edge Computing and Initial Congestion Window Affect Latency of Web-Based Services: Early Experiences with Baidu? , 2018, 2018 IEEE/ACM Symposium on Edge Computing (SEC).

[117]  Marios Savvides,et al.  Driver cell phone usage detection on Strategic Highway Research Program (SHRP2) face view videos , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[118]  Matti Pietikäinen,et al.  Face Description with Local Binary Patterns: Application to Face Recognition , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[119]  Patrick Pérez,et al.  Color-Based Probabilistic Tracking , 2002, ECCV.

[120]  Weisong Shi,et al.  LAVEA: latency-aware video analytics on edge computing platform , 2017, SEC.

[121]  Orhan Bulan,et al.  Passenger Compartment Violation Detection in HOV/HOT Lanes , 2016, IEEE Transactions on Intelligent Transportation Systems.

[122]  Brian Y. Lattimer,et al.  Feature Selection for Intelligent Firefighting Robot Classification of Fire, Smoke, and Thermal Reflections Using Thermal Infrared Images , 2016, J. Sensors.

[123]  Kuo-Chin Fan,et al.  Vehicle Detection Using Normalized Color and Edge Map , 2007, IEEE Transactions on Image Processing.

[124]  Yaser Sheikh,et al.  Video Analysis for Body-worn Cameras in Law Enforcement , 2016, ArXiv.

[125]  Avideh Zakhor,et al.  Applications of Video-Content Analysis and Retrieval , 2002, IEEE Multim..

[126]  Feiniu Yuan,et al.  An integrated fire detection and suppression system based on widely available video surveillance , 2010, Machine Vision and Applications.

[127]  O. Muensterer,et al.  Google Glass for Documentation of Medical Findings: Evaluation in Forensic Medicine , 2014, Journal of medical Internet research.

[128]  Takeshi Arikuma,et al.  Intelligent multimedia surveillance system for safer cities , 2016, APSIPA Transactions on Signal and Information Processing.

[129]  Alessandro Perina,et al.  Person re-identification by symmetry-driven accumulation of local features , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[130]  Manish Kumar,et al.  Genetic Fuzzy based Target Geo-localization using Unmanned Aerial Systems for Firefighting Applications , 2018 .