Connecting Pixels to Privacy and Utility: Automatic Redaction of Private Information in Images

Images convey a broad spectrum of personal information. If such images are shared on social media platforms, this personal information is leaked which conflicts with the privacy of depicted persons. Therefore, we aim for automated approaches to redact such private information and thereby protect privacy of the individual. By conducting a user study we find that obfuscating the image regions related to the private information leads to privacy while retaining utility of the images. Moreover, by varying the size of the regions different privacy-utility trade-offs can be achieved. Our findings argue for a "redaction by segmentation" paradigm. Hence, we propose the first sizable dataset of private images "in the wild" annotated with pixel and instance level labels across a broad range of privacy classes. We present the first model for automatic redaction of diverse private information. It is effective at achieving various privacy-utility trade-offs within 83% of the performance of redactions based on ground-truth annotation.

[1]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, International Journal of Computer Vision.

[2]  Touradj Ebrahimi,et al.  Using warping for privacy protection in video surveillance , 2013, 2013 18th International Conference on Digital Signal Processing (DSP).

[3]  Iasonas Kokkinos,et al.  Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs , 2014, ICLR.

[4]  Wei Xu,et al.  Bidirectional LSTM-CRF Models for Sequence Tagging , 2015, ArXiv.

[5]  David Sánchez,et al.  Toward sensitive document release with privacy guarantees , 2017, Eng. Appl. Artif. Intell..

[6]  Qiang Wu,et al.  Learning-Based License Plate Detection Using Global and Local Features , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[7]  Hongxin Hu,et al.  Blur vs. Block: Investigating the Effectiveness of Privacy-Enhancing Obfuscation for Images , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[8]  David J. Crandall,et al.  Viewer Experience of Obscuring Scene Elements in Photos to Enhance Privacy , 2018, CHI.

[9]  Massimo Mauro,et al.  Multi-class semantic segmentation of faces , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[10]  Cornelia Caragea,et al.  Privacy Prediction of Images Shared on Social Media Sites Using Deep Features , 2015, ArXiv.

[11]  Seong Joon Oh,et al.  Faceless Person Recognition: Privacy Implications in Social Media , 2016, ECCV.

[12]  David Sánchez,et al.  Automatic General-Purpose Sanitization of Textual Documents , 2013, IEEE Transactions on Information Forensics and Security.

[13]  Thomas Brox,et al.  Universal Adversarial Perturbations Against Semantic Image Segmentation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[14]  Christopher D. Manning,et al.  Incorporating Non-local Information into Information Extraction Systems by Gibbs Sampling , 2005, ACL.

[15]  Jessica Staddon,et al.  Sanitization's slippery slope: the design and study of a text revision assistant , 2009, SOUPS.

[16]  Honglak Lee,et al.  Augmenting CRFs with Boltzmann Machine Shape Priors for Image Labeling , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  Touradj Ebrahimi,et al.  Subjective study of privacy filters in video surveillance , 2012, 2012 IEEE 14th International Workshop on Multimedia Signal Processing (MMSP).

[18]  Seong Joon Oh,et al.  Adversarial Image Perturbation for Privacy Protection A Game Theory Perspective , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[19]  Alessandro Acquisti,et al.  Imagined Communities: Awareness, Information Sharing, and Privacy on the Facebook , 2006, Privacy Enhancing Technologies.

[20]  Sebastian Ramos,et al.  The Cityscapes Dataset for Semantic Urban Scene Understanding , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Eduard H. Hovy,et al.  End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF , 2016, ACL.

[22]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[23]  Guillaume Lample,et al.  Neural Architectures for Named Entity Recognition , 2016, NAACL.

[24]  Steven C. H. Hoi,et al.  Face Detection using Deep Learning: An Improved Faster RCNN Approach , 2017, Neurocomputing.

[25]  Ashwin Machanavajjhala,et al.  Protecting Visual Secrets Using Adversarial Nets , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[26]  Qi Tian,et al.  Principal Visual Word Discovery for Automatic License Plate Detection , 2012, IEEE Transactions on Image Processing.

[27]  Jonathon S. Hare,et al.  I know what you did last summer! - privacy-aware image classification and search , 2012 .

[28]  Ivan Sikiric,et al.  I Know That Person: Generative Full Body and Face De-identification of People in Images , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[29]  Yiannis Kompatsiaris,et al.  Personalized Privacy-aware Image Classification , 2016, ICMR.

[30]  Yoon Kim,et al.  Convolutional Neural Networks for Sentence Classification , 2014, EMNLP.

[31]  Pascal Fua,et al.  SLIC Superpixels Compared to State-of-the-Art Superpixel Methods , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Jonathon Shlens,et al.  Explaining and Harnessing Adversarial Examples , 2014, ICLR.

[33]  Lujo Bauer,et al.  Accessorize to a Crime: Real and Stealthy Attacks on State-of-the-Art Face Recognition , 2016, CCS.

[34]  Seong Joon Oh,et al.  Exploiting Saliency for Object Segmentation from Image Level Labels , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Jessica Staddon,et al.  Detecting privacy leaks using corpus-based association rules , 2008, KDD.

[36]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[38]  Bernt Schiele,et al.  A Domain Based Approach to Social Relation Recognition , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Christian Bauckhage,et al.  Age Recognition in the Wild , 2010, 2010 20th International Conference on Pattern Recognition.

[40]  Sei-Wang Chen,et al.  Automatic license plate recognition , 2004, IEEE Transactions on Intelligent Transportation Systems.

[41]  Touradj Ebrahimi,et al.  PEViD: privacy evaluation video dataset , 2013, Optics & Photonics - Optical Engineering + Applications.

[42]  Edoardo M. Airoldi,et al.  Integrating Utility into Face De-identification , 2005, Privacy Enhancing Technologies.

[43]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Horst Bischof,et al.  Annotated Facial Landmarks in the Wild: A large-scale, real-world database for facial landmark localization , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[45]  Seyed-Mohsen Moosavi-Dezfooli,et al.  Universal Adversarial Perturbations , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  Christopher Edwards,et al.  The effects of filtered video on awareness and privacy , 2000, CSCW '00.

[47]  Bernt Schiele,et al.  Speaking the Same Language: Matching Machine to Human Captions by Adversarial Training , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[48]  Yi Li,et al.  Fully Convolutional Instance-Aware Semantic Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[49]  David Sánchez,et al.  Detecting Sensitive Information from Textual Documents: An Information-Theoretic Approach , 2012, MDAI.

[50]  Gang Wang,et al.  Seeing People in Social Context: Recognizing People and Social Relationships , 2010, ECCV.

[51]  Vladlen Koltun,et al.  Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials , 2011, NIPS.

[52]  Bernhard Debatin,et al.  Facebook and Online Privacy: Attitudes, Behaviors, and Unintended Consequences , 2009, J. Comput. Mediat. Commun..

[53]  David J. Crandall,et al.  Cartooning for Enhanced Privacy in Lifelogging and Streaming Videos , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[54]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[55]  Mukesh K. Mohania,et al.  Efficient techniques for document sanitization , 2008, CIKM '08.

[56]  Abhishek Dutta,et al.  The VGG Image Annotator (VIA) , 2019, ArXiv.

[57]  Ming Shao,et al.  What Do You Do? Occupation Recognition in a Photo via Social Context , 2013, 2013 IEEE International Conference on Computer Vision.

[58]  Tribhuvanesh Orekondy,et al.  Towards a Visual Privacy Advisor: Understanding and Predicting Privacy Risks in Images , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).