Unveiling Real-Life Effects of Online Photo Sharing

Social networks give free access to their services in exchange for the right to exploit their users’ data. Data sharing is done in an initial context which is chosen by the users. However, data are used by social networks and third parties in different contexts which are often not transparent. In order to unveil such usages, we propose an approach which focuses on the effects of data sharing in impactful real-life situations. Focus is put on visual content because of its strong influence in shaping online user profiles. The approach relies on three components: (1) a set of visual objects with associated situation impact ratings obtained by crowdsourcing, (2) a corresponding set of object detectors for mining users’ photos and (3) a ground truth dataset made of 500 visual user profiles which are manually rated per situation. These components are combined in LERV UP, a method which learns to rate visual user profiles in each situation. LERV UP exploits a new image descriptor which aggregates object ratings and object detections at user level and an attention mechanism which boosts highly-rated objects to prevent them from being overwhelmed by low-rated ones. Performance is evaluated per situation by measuring the correlation between the automatic ranking of profile ratings and a manual ground truth. Results indicate that LERV UP is effective since a strong correlation of the two rankings is obtained. A practical implementation of the approach in a mobile app which raises user awareness about shared data usage is also discussed.

[1]  G. F. Hughes,et al.  On the mean accuracy of statistical pattern recognizers , 1968, IEEE Trans. Inf. Theory.

[2]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[3]  Michael Smyth,et al.  Minimising Conceptual Baggage: Making Choices about Metaphor , 1994, BCS HCI.

[4]  M. Banaji,et al.  Implicit social cognition: attitudes, self-esteem, and stereotypes. , 1995, Psychological review.

[5]  Helen Nissenbaum,et al.  Bias in computer systems , 1996, TOIS.

[6]  Bernhard Schölkopf,et al.  Kernel Principal Component Analysis , 1997, ICANN.

[7]  Michael J. Burke,et al.  On Average Deviation Indices for Estimating Interrater Agreement , 1999 .

[8]  Michael J. Burke,et al.  Estimating Interrater Agreement with the Average Deviation Index: A User’s Guide , 2002 .

[9]  Robert P. Sheridan,et al.  Random Forest: A Classification and Regression Tool for Compound Classification and QSAR Modeling , 2003, J. Chem. Inf. Comput. Sci..

[10]  Zhen Lin,et al.  Choosing SNPs using feature selection , 2005, 2005 IEEE Computational Systems Bioinformatics Conference (CSB'05).

[11]  Mor Naaman,et al.  Over-exposed?: privacy patterns and considerations in online and mobile photo sharing , 2007, CHI.

[12]  Heng Tao Shen,et al.  Principal Component Analysis , 2009, Encyclopedia of Biometrics.

[13]  Jaeyoung Choi,et al.  Semantic Computing and Privacy: a Case Study Using Inferred Geo-Location , 2011, Int. J. Semantic Comput..

[14]  K. Curran,et al.  Advertising on Facebook , 2011 .

[15]  Jonathon S. Hare,et al.  Privacy-aware image classification and search , 2012, SIGIR '12.

[16]  Jürgen Schmidhuber,et al.  Multi-column deep neural network for traffic sign classification , 2012, Neural Networks.

[17]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[18]  T. Graepel,et al.  Private traits and attributes are predictable from digital records of human behavior , 2013, Proceedings of the National Academy of Sciences.

[19]  David Lyon,et al.  Surveillance, Snowden, and Big Data: Capacities, consequences, critique , 2014, Big Data Soc..

[20]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[21]  Joan C. Williams.  Double Jeopardy? An Empirical Study with Implications for the Debates over Implicit Bias and Intersectionality, , 2014 .

[22]  Yiannis Kompatsiaris,et al.  PScore: A Framework for Enhancing Privacy Awareness in Online Social Networks , 2015, 2015 10th International Conference on Availability, Reliability and Security.

[23]  Gregory J. Park,et al.  Automatic personality assessment through social media language. , 2015, Journal of personality and social psychology.

[24]  David Martens,et al.  Who cares about your Facebook friends? Credit scoring for microfinance , 2015 .

[25]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Jason Yosinski,et al.  Deep neural networks are easily fooled: High confidence predictions for unrecognizable images , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Barbara Caputo,et al.  A Deeper Look at Dataset Bias , 2015, Domain Adaptation in Computer Vision Applications.

[28]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[29]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  David A. Shamma,et al.  YFCC100M , 2015, Commun. ACM.

[31]  Yiannis Kompatsiaris,et al.  Personalized Privacy-aware Image Classification , 2016, ICMR.

[32]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[33]  Chen Huang,et al.  Learning Deep Representation for Imbalanced Classification , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Rogério Schmidt Feris,et al.  A Unified Multi-scale Deep Convolutional Neural Network for Fast Object Detection , 2016, ECCV.

[35]  Maritta Heisel,et al.  Online Self-disclosure: From Users' Regrets to Instructional Awareness , 2017, CD-MAKE.

[36]  Tribhuvanesh Orekondy,et al.  Towards a Visual Privacy Advisor: Understanding and Predicting Privacy Risks in Images , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[37]  Michael Luca,et al.  Racial Discrimination in the Sharing Economy: Evidence from a Field Experiment , 2016 .

[38]  Kaiming He,et al.  Focal Loss for Dense Object Detection , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[39]  Ali Farhadi,et al.  YOLO9000: Better, Faster, Stronger , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Muhsin Tunay Gencoglu,et al.  The performance comparison of Multiple Linear Regression, Random Forest and Artificial Neural Network by using photovoltaic and atmospheric data , 2017, 2017 14th International Conference on Engineering of Modern Electric Systems (EMES).

[41]  Sergey Ioffe,et al.  Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning , 2016, AAAI.

[42]  M. Hussain,et al.  Constructing a Data-Driven Society: China's Social Credit System as a State Surveillance Infrastructure , 2018, Policy & Internet.

[43]  Matthieu Manant,et al.  Can Social Media Lead to Labor Market Discrimination? Evidence from a Field Experiment , 2017, Journal of Economics & Management Strategy.

[44]  Wenguan Wang,et al.  Deep Visual Attention Prediction , 2017, IEEE Transactions on Image Processing.

[45]  Mark Sandler,et al.  MobileNetV2: Inverted Residuals and Linear Bottlenecks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[46]  Ali Farhadi,et al.  YOLOv3: An Incremental Improvement , 2018, ArXiv.

[47]  Christine T. Wolf.  Explainability scenarios: towards scenario-based XAI design , 2019, IUI.

[48]  Cornelia Caragea,et al.  Dynamic Deep Multi-modal Fusion for Image Privacy Prediction , 2019, WWW.

[49]  Quoc V. Le,et al.  NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detection , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[50]  Yuxing Peng,et al.  ThunderNet: Towards Real-Time Generic Object Detection on Mobile Devices , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[51]  Ross B. Girshick,et al.  Mask R-CNN , 2017, 1703.06870.

[52]  Justin Cheng,et al.  Social Comparison and Facebook: Feedback, Positivity, and Opportunities for Comparison , 2020, CHI.

[53]  Ross B. Girshick,et al.  Focal Loss for Dense Object Detection , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[54]  A. Acquisti,et al.  An Experiment in Hiring Discrimination via Online Social Networks , 2020, Manag. Sci..