The Emergence of Visual Crowdsensing: Challenges and Opportunities

Visual crowdsensing (VCS), which leverages built-in cameras of smart devices to attain informative and comprehensive sensing of interesting targets, has become a predominant sensing paradigm of mobile crowdsensing (MCS). Compared to MCS tasks using other sensing modalities, VCS faces numerous unique issues, such as multi-dimensional coverage needs, data redundancy identification and elimination, low-cost transmission, as well as high data processing cost. This paper characterizes the concepts, unique features, and novel application areas of VCS, and investigates its challenges and key techniques. A generic framework for VCS systems is then presented, followed by discussions about the future directions of crowdsourced picture transmission and the experimental setup in VCS system evaluation.

[1]  Michael Goesele,et al.  Scene Reconstruction and Visualization From Community Photo Collections , 2010, Proceedings of the IEEE.

[2]  Hong-Hsu Yen,et al.  A Survey on Sensor Coverage and Visual Data Capturing/Processing/Transmission in Wireless Visual Sensor Networks , 2014, Sensors.

[3]  Samuel D. Gosling,et al.  Like Partying? Your Face Says It All. Predicting the Ambiance of Places with Profile Pictures , 2015, ICWSM.

[4]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[5]  Jiannong Cao,et al.  CROWD-PAN-360: Crowdsourcing Based Context-Aware Panoramic Map Generation for Smartphone Users , 2015, IEEE Transactions on Parallel and Distributed Systems.

[6]  B. S. Manjunath,et al.  Color and texture descriptors , 2001, IEEE Trans. Circuits Syst. Video Technol..

[7]  Kevin R. Fall,et al.  A delay-tolerant network architecture for challenged internets , 2003, SIGCOMM '03.

[8]  Yan Zhang,et al.  Optimal Incentive Design for Cloud-Enabled Multimedia Crowdsourcing , 2016, IEEE Transactions on Multimedia.

[9]  Douglas C. Schmidt,et al.  WreckWatch: Automatic Traffic Accident Detection and Notification with Smartphones , 2011, Mob. Networks Appl..

[10]  Deborah Estrin,et al.  Image browsing, processing, and clustering for participatory sensing: lessons from a DietSense prototype , 2007, EmNets '07.

[11]  Zhang Meng,et al.  A Robust and Discriminative Image Perceptual Hash Algorithm , 2010, 2010 Fourth International Conference on Genetic and Evolutionary Computing.

[12]  Zoran Popovic,et al.  PhotoCity: training experts at large-scale image acquisition through a competitive game , 2011, CHI.

[13]  Kaigui Bian,et al.  Jigsaw: indoor floor plan reconstruction via mobile crowdsensing , 2014, MobiCom.

[14]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Daqing Zhang,et al.  Participatory Cultural Mapping Based on Collective Behavior Data in Location-Based Social Networks , 2016, ACM Trans. Intell. Syst. Technol..

[16]  Liming Chen,et al.  A Generic Framework for Constraint-Driven Data Selection in Mobile Crowd Photographing , 2017, IEEE Internet of Things Journal.

[17]  Andrew Raij,et al.  A Survey of Incentive Techniques for Mobile Crowd Sensing , 2015, IEEE Internet of Things Journal.

[18]  Chunming Qiao,et al.  Crowd Map: Accurate Reconstruction of Indoor Floor Plans from Crowdsourced Sensor-Rich Videos , 2015, 2015 IEEE 35th International Conference on Distributed Computing Systems.

[19]  Masamichi Shimosaka,et al.  Steered crowdsensing: incentive design towards quality-oriented place-centric crowdsensing , 2014, UbiComp.

[20]  James Lee Hafner,et al.  Efficient Color Histogram Indexing for Quadratic Form Distance Functions , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  Cyrus Shahabi,et al.  Key Frame Selection Algorithms for Automatic Generation of Panoramic Images from Crowdsourced Geo-tagged Videos , 2014, W2GIS.

[22]  Mo Li,et al.  Travi-Navi: Self-Deployable Indoor Navigation System , 2017, TNET.

[23]  Margaret Martonosi,et al.  SignalGuru: leveraging mobile phones for collaborative traffic signal schedule advisory , 2011, MobiSys '11.

[24]  Andreas Uhl,et al.  Image similarity measurement by Kullback-Leibler divergences between complex wavelet subband statistics for texture retrieval , 2008, 2008 15th IEEE International Conference on Image Processing.

[25]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[26]  Nenghai Yu,et al.  MoVieUp: Automatic Mobile Video Mashup , 2015, IEEE Transactions on Circuits and Systems for Video Technology.

[27]  Deborah Estrin,et al.  Examining micro-payments for participatory sensing data collections , 2010, UbiComp.

[28]  Mianxiong Dong,et al.  RMER: Reliable and Energy-Efficient Data Collection for Large-Scale Wireless Sensor Networks , 2016, IEEE Internet of Things Journal.

[29]  Jörg Ott,et al.  The ONE simulator for DTN protocol evaluation , 2009, SIMUTools 2009.

[30]  Fan Ye,et al.  Mobile crowdsensing: current state and future challenges , 2011, IEEE Communications Magazine.

[31]  Gang Zhou,et al.  CrowdWatch: pedestrian safety assistance with mobile crowd sensing , 2016, UbiComp Adjunct.

[32]  Guohong Cao,et al.  VideoMec: A Metadata-Enhanced Crowdsourcing System for Mobile Videos , 2017, 2017 16th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN).

[33]  Xing Xie,et al.  TaskMe: Toward a dynamic and quality-enhanced incentive mechanism for mobile crowd sensing , 2017, Int. J. Hum. Comput. Stud..

[34]  Xiaohui Xie,et al.  An energy-efficient data transmission protocol for mobile crowd sensing , 2017, Peer Peer Netw. Appl..

[35]  Eben M. Haber,et al.  Creek watch: pairing usefulness and usability for successful citizen science , 2011, CHI.

[36]  Jacky C. K. Chow.  Multi-Sensor Integration for Indoor 3D Reconstruction , 2014, ArXiv.

[37]  Ramesh Govindan,et al.  MediaScope: Selective on-demand media retrieval from mobile devices , 2013, 2013 ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN).

[38]  Bin Guo,et al.  Public Sense: Refined Urban Sensing and Public Facility Management with Crowdsourced Data , 2015, 2015 IEEE 12th Intl Conf on Ubiquitous Intelligence and Computing and 2015 IEEE 12th Intl Conf on Autonomic and Trusted Computing and 2015 IEEE 15th Intl Conf on Scalable Computing and Communications and Its Associated Workshops (UIC-ATC-ScalCom).

[39]  Yan Liu,et al.  ActiveCrowd: A Framework for Optimized Multitask Allocation in Mobile Crowdsensing Systems , 2016, IEEE Transactions on Human-Machine Systems.

[40]  Amit K. Roy-Chowdhury,et al.  Opportunistic Image Acquisition of Individual and Group Activities in a Distributed Camera Network , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[41]  Junjie Chen,et al.  A novel rumour propagation model on social networks , 2017, Int. J. Sens. Networks.

[42]  Ian F. Akyildiz,et al.  A survey on wireless multimedia sensor networks , 2007, Comput. Networks.

[43]  Zhaohui Wu,et al.  Container Port Performance Measurement and Comparison Leveraging Ship GPS Traces and Maritime Open Data , 2016, IEEE Transactions on Intelligent Transportation Systems.

[44]  Nael B. Abu-Ghazaleh,et al.  Coverage algorithms for visual sensor networks , 2013, TOSN.

[45]  Michael R. Souryal,et al.  RFID-based localization and tracking technologies , 2011, IEEE Wireless Communications.

[46]  Yutaka Matsuo,et al.  Earthquake shakes Twitter users: real-time event detection by social sensors , 2010, WWW '10.

[47]  Tao Wang,et al.  QoE-ensured price competition model for emerging mobile networks , 2015, IEEE Wireless Communications.

[48]  Qi Han,et al.  Toward real-time and cooperative mobile visual sensing and sharing , 2016, IEEE INFOCOM 2016 - The 35th Annual IEEE International Conference on Computer Communications.

[49]  Ming Yang,et al.  Large-scale image classification: Fast feature extraction and SVM training , 2011, CVPR 2011.

[50]  Kaigui Bian,et al.  Sextant: Towards Ubiquitous Indoor Localization Service by Photo-Taking of the Environment , 2016, IEEE Transactions on Mobile Computing.

[51]  Cyrill Stachniss,et al.  Simultaneous Localization and Mapping , 2016, Springer Handbook of Robotics, 2nd Ed..

[52]  Di Wu,et al.  Efficient data dissemination by crowdsensing in vehicular networks , 2014, 2014 IEEE 22nd International Symposium of Quality of Service (IWQoS).

[53]  Hojung Cha,et al.  Piggyback CrowdSensing (PCS): energy efficient crowdsourcing of mobile sensor data by exploiting smartphone app opportunities , 2013, SenSys '13.

[54]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[55]  Yunhao Liu,et al.  On Reliable Task Assignment for Spatial Crowdsourcing , 2019, IEEE Transactions on Emerging Topics in Computing.

[56]  K. Atukorala,et al.  SmartEye Integrated Solution to Home Automation, Security and Monitoring through Mobile Phones , 2009, 2009 Third International Conference on Next Generation Mobile Applications, Services and Technologies.

[57]  Steven M. Seitz,et al.  Capturing indoor scenes with smartphones , 2012, UIST.

[58]  Delphine Reinhardt.  Privacy in mobile participatory sensing: Current trends and future challenges , 2016, J. Syst. Softw..

[59]  Xing Zhang,et al.  A Survey on Mobile Edge Networks: Convergence of Computing, Caching and Communications , 2017, IEEE Access.

[60]  Minho Shin,et al.  Anonysense: privacy-aware people-centric sensing , 2008, MobiSys '08.

[61]  Bin Guo,et al.  CooperSense: A Cooperative and Selective Picture Forwarding Framework Based on Tree Fusion , 2016, Int. J. Distributed Sens. Networks.

[62]  Xing Zhang,et al.  Jam Eyes: A Traffic Jam Awareness and Observation System Using Mobile Phones , 2012, Int. J. Distributed Sens. Networks.

[63]  Xingshe Zhou,et al.  A Data-Centric Framework for Cyber-Physical-Social Systems , 2015, IT Prof..

[64]  Daqing Zhang,et al.  The Emergence of Social and Community Intelligence , 2011, Computer.

[65]  Daqing Zhang,et al.  Sparse mobile crowdsensing: challenges and opportunities , 2016, IEEE Communications Magazine.

[66]  John P. Rula,et al.  Crowdsensing Under (Soft) Control , 2015, 2015 IEEE Conference on Computer Communications (INFOCOM).

[67]  Daqing Zhang,et al.  CrowdRecruiter: selecting participants for piggyback crowdsensing under probabilistic coverage constraint , 2014, UbiComp.

[68]  Srinivasan Seshan,et al.  CARE: Content Aware Redundancy Elimination for Disaster Communications on Damaged Networks , 2012, ArXiv.

[69]  Jizhong Zhao,et al.  Reliable Diversity-Based Spatial Crowdsourcing by Moving Workers , 2014, Proc. VLDB Endow..

[70]  Yi Wang,et al.  SmartPhoto: A Resource-Aware Crowdsourcing Approach for Image Sensing with Smartphones , 2014, IEEE Transactions on Mobile Computing.

[71]  Landon P. Cox,et al.  LiveCompare: grocery bargain hunting through participatory sensing , 2009, HotMobile '09.

[72]  Mahadev Satyanarayanan,et al.  The Emergence of Edge Computing , 2017, Computer.

[73]  Daqing Zhang,et al.  EnUp: Energy-Efficient Data Uploading for Mobile Crowd Sensing Applications , 2016, 2016 Intl IEEE Conferences on Ubiquitous Intelligence & Computing, Advanced and Trusted Computing, Scalable Computing and Communications, Cloud and Big Data Computing, Internet of People, and Smart World Congress (UIC/ATC/ScalCom/CBDCom/IoP/SmartWorld).

[74]  Tianrui Li,et al.  Urban sensing based on human mobility , 2016, UbiComp.

[75]  Romit Roy Choudhury,et al.  MoVi: mobile phone based video highlights via collaborative sensing , 2010, MobiSys '10.

[76]  Daqing Zhang,et al.  iCrowd: Near-Optimal Task Allocation for Piggyback Crowdsensing , 2016, IEEE Transactions on Mobile Computing.

[77]  Cordelia Schmid,et al.  Good Practice in Large-Scale Learning for Image Classification , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[78]  S. Govindarajulu,et al.  A Comparison of SIFT, PCA-SIFT and SURF , 2012 .

[79]  Fei-Fei Li,et al.  Large-Scale Video Classification with Convolutional Neural Networks , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[80]  Zhuo Chen,et al.  Edge Analytics in the Internet of Things , 2015, IEEE Pervasive Computing.

[81]  Qi Han,et al.  Worker-Contributed Data Utility Measurement for Visual Crowdsensing Systems , 2017, IEEE Transactions on Mobile Computing.

[82]  Yonatan Wexler,et al.  Detecting text in natural scenes with stroke width transform , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[83]  Bin Guo,et al.  Personalized Travel Package With Multi-Point-of-Interest Recommendation Based on Crowdsourced User Footprints , 2016, IEEE Transactions on Human-Machine Systems.

[84]  David W. Mizell,et al.  Using gravity to estimate accelerometer orientation , 2003, Seventh IEEE International Symposium on Wearable Computers, 2003. Proceedings..

[85]  Daqing Zhang,et al.  crowddeliver: Planning City-Wide Package Delivery Paths Leveraging the Crowd of Taxis , 2017, IEEE Transactions on Intelligent Transportation Systems.

[86]  Xiaoying Gan,et al.  Critical sensing range for mobile heterogeneous camera sensor networks , 2014, IEEE INFOCOM 2014 - IEEE Conference on Computer Communications.

[87]  Richard Szeliski,et al.  Building Rome in a day , 2009, ICCV.

[88]  Md. Yusuf Sarwar Uddin,et al.  PhotoNet: A Similarity-Aware Picture Delivery Service for Situation Awareness , 2011, 2011 IEEE 32nd Real-Time Systems Symposium.

[89]  Mahadev Satyanarayanan,et al.  Scalable crowd-sourcing of video from mobile devices , 2013, MobiSys '13.

[90]  Bernhard Rinner,et al.  Security and Privacy Protection in Visual Sensor Networks , 2014, ACM Comput. Surv..

[91]  Hojung Cha,et al.  Automatically characterizing places with opportunistic crowdsensing using smartphones , 2012, UbiComp.

[92]  育久 満上,et al.  Bundler: Structure from Motion for Unordered Image Collections , 2011 .

[93]  Srinivasan Seshan,et al.  CARE: content aware redundancy elimination for challenged networks , 2012, HotNets-XI.

[94]  Li Xiao,et al.  SADV: Static-Node-Assisted Adaptive Data Dissemination in Vehicular Networks , 2010, IEEE Transactions on Vehicular Technology.

[95]  K. Beaty,et al.  FOCUS: clustering crowdsourced videos by line-of-sight , 2013, SenSys '13.

[96]  Ayan Banerjee,et al.  MT-Diet: Automated smartphone based diet assessment with infrared images , 2016, 2016 IEEE International Conference on Pervasive Computing and Communications (PerCom).

[97]  Xiaomei Zhang,et al.  Resource-Aware Photo Crowdsourcing Through Disruption Tolerant Networks , 2016, 2016 IEEE 36th International Conference on Distributed Computing Systems (ICDCS).

[98]  ZhangDaqing,et al.  The Emergence of Social and Community Intelligence , 2011 .

[99]  Antonio Torralba,et al.  Ieee Transactions on Pattern Analysis and Machine Intelligence 1 80 Million Tiny Images: a Large Dataset for Non-parametric Object and Scene Recognition , 2022 .

[100]  Yunhao Liu,et al.  Enhancing wifi-based localization with visual clues , 2015, UbiComp.

[101]  Yi Wang,et al.  Photo crowdsourcing for area coverage in resource constrained environments , 2017, IEEE INFOCOM 2017 - IEEE Conference on Computer Communications.

[102]  Jiajia Liu,et al.  Congestion-Aware Communication Paradigm for Sustainable Dense Mobile Crowdsensing , 2017, IEEE Communications Magazine.

[103]  Jim R. Parker,et al.  Algorithms for image processing and computer vision , 1996 .

[104]  Jeffrey P. Bigham,et al.  VizWiz: nearly real-time answers to visual questions , 2010, W4A.

[105]  Zhu Wang,et al.  Mobile Crowd Sensing and Computing , 2015, ACM Comput. Surv..

[106]  Huadong Ma,et al.  Opportunities in mobile crowd sensing , 2014, IEEE Communications Magazine.

[107]  Xing Xie,et al.  FlierMeet: A Mobile Crowdsensing System for Cross-Space Public Information Reposting, Tagging, and Sharing , 2015, IEEE Transactions on Mobile Computing.

[108]  Ramesh Govindan,et al.  Medusa: a programming framework for crowd-sensing applications , 2012, MobiSys '12.

[109]  Luo Juan,et al.  A comparison of SIFT, PCA-SIFT and SURF , 2009 .

[110]  Aniket Kittur,et al.  Crowdsourcing user studies with Mechanical Turk , 2008, CHI.

[111]  Yunhao Liu,et al.  From RSSI to CSI , 2013, ACM Comput. Surv..

[112]  Dario Pompili,et al.  Collaborative Mobile Edge Computing in 5G Networks: New Paradigms, Scenarios, and Challenges , 2016, IEEE Communications Magazine.

[113]  Delphine Christin,et al.  Privacy in mobile participatory sensing , 2016 .

[114]  Daqing Zhang,et al.  EMC3: Energy-efficient data transfer in mobile crowdsensing under full coverage constraint , 2015, IEEE Transactions on Mobile Computing.

[115]  Wei Tsang Ooi,et al.  MoViMash: online mobile video mashup , 2012, ACM Multimedia.

[116]  Keiichi Yasumoto,et al.  SakuraSensor: quasi-realtime cherry-lined roads detection through participatory video sensing by cars , 2015, UbiComp.

[117]  Maria E. Niessen,et al.  NoiseTube: Measuring and mapping noise pollution with mobile phones , 2009, ITEE.

[118]  Heng Ji,et al.  FaitCrowd: Fine Grained Truth Discovery for Crowdsourced Data Aggregation , 2015, KDD.

[119]  Yang Wang,et al.  TaskMe: multi-task allocation in mobile crowd sensing , 2016, UbiComp.

[120]  Matthew Antone,et al.  Grouping Crowd-Sourced Mobile Videos for Cross-Camera Tracking , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[121]  Jian Tang,et al.  Sensing as a Service: Challenges, Solutions and Future Directions , 2013, IEEE Sensors Journal.

[122]  Yi Wang,et al.  Barrier coverage in camera sensor networks , 2011, MobiHoc '11.

[123]  Chunming Qiao,et al.  Rise of the Indoor Crowd: Reconstruction of Building Interior View via Mobile Crowdsourcing , 2015, SenSys.