GANmouflage: 3D Object Nondetection with Texture Fields

We propose a method that learns to camouflage 3D objects within scenes. Given an object's shape and a distribution of viewpoints from which it will be seen, we estimate a texture that will make it difficult to detect. Successfully solving this task requires a model that can accurately reproduce textures from the scene, while simultaneously dealing with the highly conflicting constraints imposed by each viewpoint. We address these challenges with a model based on texture fields and adversarial learning. Our model learns to camouflage a variety of object shapes from randomly sampled locations and viewpoints within the input scene, and is the first to address the problem of hiding complex object shapes. Using a human visual search study, we find that our estimated textures conceal objects significantly better than previous methods. Project site: https://rrrrrguo.github.io/ganmouflage/

[1]  Leon L. Xu,et al.  ABO: Dataset and Benchmarks for Real-World 3D Object Understanding , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Victor Lempitsky,et al.  Resolution-robust Large Mask Inpainting with Fourier Convolutions , 2021, 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV).

[3]  Erika Lu,et al.  Self-supervised Video Object Segmentation by Motion Grouping , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[4]  Hao Su,et al.  MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[5]  Angjoo Kanazawa,et al.  pixelNeRF: Neural Radiance Fields from One or Few Images , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Tam V. Nguyen,et al.  MirrorNet: Bio-Inspired Camouflaged Object Segmentation , 2020, IEEE Access.

[7]  Andrew Zisserman,et al.  Betrayed by Motion: Camouflaged Object Discovery via Motion Segmentation , 2020, ACCV.

[8]  Orcun Goksel,et al.  GramGAN: Deep 3D Texture Synthesis From 2D Exemplars , 2020, NeurIPS.

[9]  Ling Shao,et al.  Camouflaged Object Detection , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Yongwei Nie,et al.  Deep Camouflage Images , 2020, AAAI.

[11]  Pratul P. Srinivasan,et al.  NeRF , 2020, ECCV.

[12]  N. Mitra,et al.  Learning a Neural 3D Texture Space From 2D Exemplars , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Wan-Yen Lo,et al.  Accelerating 3D deep learning with PyTorch3D , 2019, SIGGRAPH Asia 2020 Courses.

[14]  Nicholas E. Scott-Samuel,et al.  CamoGAN: Evolving optimum camouflage with Generative Adversarial Networks , 2019, Methods in Ecology and Evolution.

[15]  Gordon Wetzstein,et al.  Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations , 2019, NeurIPS.

[16]  Andreas Geiger,et al.  Texture Fields: Learning Texture Representations in Function Space , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[17]  Hao Li,et al.  PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[18]  Tali Dekel,et al.  SinGAN: Learning a Generative Model From a Single Natural Image , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[19]  Thomas S. Huang,et al.  Free-Form Image Inpainting With Gated Convolution , 2018, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[20]  Thomas S. Huang,et al.  Generative Image Inpainting with Contextual Attention , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[21]  Alexei A. Efros,et al.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[22]  Logan Engstrom,et al.  Synthesizing Robust Adversarial Examples , 2017, ICML.

[23]  Ravi Ramamoorthi,et al.  Patch-based optimization for image-based texture mapping , 2017, ACM Trans. Graph..

[24]  S. Merilaita,et al.  How camouflage works , 2017, Philosophical Transactions of the Royal Society B: Biological Sciences.

[25]  Martin Stevens,et al.  Relative advantages of dichromatic and trichromatic color vision in camouflage breaking , 2017, Behavioral ecology : official journal of the International Society for Behavioral Ecology.

[26]  Michael J. Black,et al.  3D Menagerie: Modeling the 3D Shape and Pose of Animals , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Leon A. Gatys,et al.  Image Style Transfer Using Convolutional Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Alexei A. Efros,et al.  Context Encoders: Feature Learning by Inpainting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[31]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Luca Konig,et al.  The Blind Watchmaker Why The Evidence Of Evolution Reveals A Universe Without Design , 2016 .

[33]  Stephanie Boehm,et al.  Animal Camouflage Mechanisms And Function , 2016 .

[34]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[35]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[36]  Jonathon Shlens,et al.  Explaining and Harnessing Adversarial Examples , 2014, ICLR.

[37]  Jitendra Malik,et al.  Hypercolumns for object segmentation and fine-grained localization , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[39]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[40]  Hanumant Singh,et al.  Camouflaging an Object from Many Viewpoints , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[41]  Joan Bruna,et al.  Intriguing properties of neural networks , 2013, ICLR.

[42]  Sha.  Public Art Projects , 2012 .

[43]  Craig W. Reynolds.  Interactive Evolution of Camouflage , 2011, Artificial Life.

[44]  T. Wong,et al.  Camouflage images , 2010, ACM Trans. Graph..

[45]  Yael Pritch,et al.  Shift-map image editing , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[46]  Eli Shechtman,et al.  PatchMatch: a randomized correspondence algorithm for structural image editing , 2009, ACM Trans. Graph..

[47]  Alexei A. Efros,et al.  Scene completion using millions of photographs , 2008, Commun. ACM.

[48]  Roger Hanlon,et al.  Cephalopod dynamic camouflage , 2007, Current Biology.

[49]  Steven M. Seitz,et al.  Photo tourism: exploring photo collections in 3D , 2006, ACM Trans. Graph..

[50]  David Salesin,et al.  Interactive digital photomontage , 2004, ACM Trans. Graph..

[51]  William T. Freeman,et al.  Example-Based Super-Resolution , 2002, IEEE Computer Graphics and Applications.

[52]  Guillermo Sapiro,et al.  Image inpainting , 2000, SIGGRAPH.

[53]  Alexei A. Efros,et al.  Texture synthesis by non-parametric sampling , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[54]  Karl Sims,et al.  Evolving 3D Morphology and Behavior by Competition , 1994, Artificial Life.

[55]  E. Poulton.  Adaptive Coloration in Animals , 1940, Nature.