A 4D Light-Field Dataset and CNN Architectures for Material Recognition

We introduce a new light-field dataset of materials, and take advantage of the recent success of deep learning to perform material recognition on the 4D light-field. Our dataset contains 12 material categories, each with 100 images taken with a Lytro Illum, from which we extract about 30,000 patches in total. To the best of our knowledge, this is the first mid-size dataset for light-field images. Our main goal is to investigate whether the additional information in a light-field (such as multiple sub-aperture views and view-dependent reflectance effects) can aid material recognition. Since recognition networks have not been trained on 4D images before, we propose and compare several novel CNN architectures to train on light-field images. In our experiments, the best performing CNN architecture achieves a 7 % boost compared with 2D image classification (\(70\,\%\rightarrow 77\,\%\)). These results constitute important baselines that can spur further research in the use of CNNs for light-field applications. Upon publication, our dataset also enables other novel applications of light-fields, including object detection, image segmentation and view interpolation.

[1]  Iasonas Kokkinos,et al.  Describing Textures in the Wild , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  Sven Wanner,et al.  Datasets and Benchmarks for Densely Sampled 4D Light Fields , 2013, VMV.

[3]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[4]  Noah Snavely,et al.  Material recognition in the wild with the Materials in Context Database , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Yael Pritch,et al.  Scene reconstruction from high spatio-angular resolution light fields , 2013, ACM Trans. Graph..

[6]  Noah Snavely,et al.  OpenSurfaces , 2013, ACM Trans. Graph..

[7]  Hang Zhang,et al.  Reflectance hashing for material recognition , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Rong Xiao,et al.  Pairwise Rotation Invariant Co-Occurrence Local Binary Pattern , 2014, IEEE Trans. Pattern Anal. Mach. Intell..

[9]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[10]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[11]  Xiaofeng Ren,et al.  Toward Robust Material Recognition for Everyday Objects , 2011, BMVC.

[12]  In-So Kweon,et al.  Learning a Deep Convolutional Network for Light-Field Image Super-Resolution , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[13]  Subhransu Maji,et al.  Multi-view Convolutional Neural Networks for 3D Shape Recognition , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[14]  Shree K. Nayar,et al.  Reflectance and texture of real-world surfaces , 1999, TOGS.

[15]  Jian Sun,et al.  Guided Image Filtering , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Edward H. Adelson,et al.  Material perception: What can you see in a brief glance? , 2010 .

[17]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Ko Nishino,et al.  Visual Material Traits: Recognizing Per-Pixel Material Context , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[19]  Camille Couprie,et al.  Learning Hierarchical Features for Scene Labeling , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Alexei A. Efros,et al.  Occlusion-Aware Depth Estimation Using Light-Field Cameras , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[21]  Mario Fritz,et al.  On the Significance of Real-World Conditions for Material Classification , 2004, ECCV.

[22]  Kiran B. Raja,et al.  Exploring the Usefulness of Light Field Cameras for Biometrics: An Empirical Study on Face and Iris Recognition , 2016, IEEE Transactions on Information Forensics and Security.

[23]  Subhransu Maji,et al.  Deep filter banks for texture recognition and segmentation , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Ko Nishino,et al.  Single image multimaterial estimation , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[25]  Mario Fritz,et al.  Recognizing Materials from Virtual Examples , 2012, ECCV.

[26]  Chao Liu,et al.  Discriminative illumination: Per-pixel classification of raw materials based on optimal projections of spectral BRDF , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[27]  Haibin Ling,et al.  Saliency Detection on Light Field , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  F. E. Nicodemus,et al.  Geometrical considerations and nomenclature for reflectance , 1977 .

[29]  Shree K. Nayar,et al.  Reflectance and texture of real-world surfaces , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[30]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Barbara Caputo,et al.  Class-Specific Material Categorisation , 2005, ICCV.

[32]  Kristin J. Dana,et al.  3D Texture Recognition Using Bidirectional Feature Histograms , 2004, International Journal of Computer Vision.

[33]  Adrien Bousseau,et al.  How do people edit light fields? , 2014, ACM Trans. Graph..

[34]  Edward H. Adelson,et al.  Exploring features in a Bayesian framework for material recognition , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[35]  Gordon Wetzstein,et al.  Compressive light field photography using overcomplete dictionaries and optimized projections , 2013, ACM Trans. Graph..

[36]  Ivan Laptev,et al.  Learning and Transferring Mid-level Image Representations Using Convolutional Neural Networks , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[37]  Michael Weinmann,et al.  Material Classification Based on Training Data Synthesized Using a BTF Database , 2014, ECCV.

[38]  Jitendra Malik,et al.  Depth from Combining Defocus and Correspondence Using Light-Field Cameras , 2013, 2013 IEEE International Conference on Computer Vision.