Joint segmentation of color and depth data based on splitting and merging driven by surface fitting

Abstract This paper proposes a segmentation scheme based on the joint usage of color and depth data together with a 3D surface estimation scheme. Firstly a set of multi-dimensional vectors is built from color, geometry and surface orientation information. Normalized cuts spectral clustering is then applied in order to recursively segment the scene in two parts thus obtaining an over-segmentation. This procedure is followed by a recursive merging stage where close segments belonging to the same object are joined together. At each step of both procedures a NURBS model is fitted on the computed segments and the accuracy of the fitting is used as a measure of the plausibility that a segment represents a single surface or object. By comparing the accuracy to the one at the previous step, it is possible to determine if each splitting or merging operation leads to a better scene representation and consequently whether to perform it or not. Experimental results show how the proposed method provides an accurate and reliable segmentation.

[1]  W. F. Clocksin,et al.  Joint Optimization for Object Class Segmentation and Dense Stereo Reconstruction , 2011, International Journal of Computer Vision.

[2]  Li-Hong Juang,et al.  A dynamic portrait segmentation by merging colors and depth information , 2015 .

[3]  Anthony Cowley,et al.  Parsing Indoor Scenes Using RGB-D Imagery , 2012, Robotics: Science and Systems.

[4]  Ruigang Yang,et al.  TofCut: Towards Robust Real-time Foreground Extraction using Time-of-flight Camera , 2016 .

[5]  Rae-Hong Park,et al.  Segmentation with saliency map using colour and depth images , 2015, IET Image Process..

[6]  Pushmeet Kohli,et al.  Object stereo — Joint stereo matching and object segmentation , 2011, CVPR 2011.

[7]  Marc Van Droogenbroeck,et al.  Combining Color, Depth, and Motion for Video Segmentation , 2009, ICVS.

[8]  Michael Felsberg,et al.  Channel Coding for Joint Colour and Depth Segmentation , 2011, DAGM-Symposium.

[9]  Pietro Zanuttigh,et al.  Scene segmentation from depth and color data driven by surface fitting , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[10]  Guido M. Cortelazzo,et al.  Fusion of Geometry and Color Information for Scene Segmentation , 2012, IEEE Journal of Selected Topics in Signal Processing.

[11]  Andrew Blake,et al.  Bi-layer segmentation of binocular stereo video , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[12]  Michael Harville,et al.  Foreground segmentation using adaptive mixture models in color and depth , 2001, Proceedings IEEE Workshop on Detection and Recognition of Events in Video.

[13]  Mahfuzur Rahman Khan,et al.  Unsupervised RGB-D image segmentation by multi-layer clustering , 2016, 2016 5th International Conference on Informatics, Electronics and Vision (ICIEV).

[14]  Pietro Zanuttigh,et al.  Joint Color and Depth Segmentation based on Region Merging and Surface Fitting , 2016, VISIGRAPP.

[15]  Montse Pardàs,et al.  Region based foreground segmentation combining color and depth sensors via logarithmic opinion pool decision , 2014, J. Vis. Commun. Image Represent..

[16]  Frank Dellaert,et al.  A Rao-Blackwellized MCMC algorithm for recovering piecewise planar 3D models from multiple view RGBD images , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[17]  Alain Trémeau,et al.  Unsupervised RGB-D image segmentation using joint clustering and region merging , 2014, BMVC.

[18]  Daniel P. Huttenlocher,et al.  Efficient Graph-Based Image Segmentation , 2004, International Journal of Computer Vision.

[19]  Huangang Wang,et al.  Parameter Selection of Gaussian Kernel for One-Class SVM , 2015, IEEE Transactions on Cybernetics.

[20]  Frank Dellaert,et al.  Planar Segmentation of RGBD Images Using Fast Linear Fitting and Markov Chain Monte Carlo , 2012, 2012 Ninth Conference on Computer and Robot Vision.

[21]  Jitendra Malik,et al.  Perceptual Organization and Recognition of Indoor Scenes from RGB-D Images , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  Florentin Wörgötter,et al.  Object Partitioning Using Local Convexity , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[23]  Longin Jan Latecki,et al.  Unsupervised Segmentation of RGB-D Images , 2014, ACCV.

[24]  Dieter Fox,et al.  RGB-(D) scene labeling: Features and algorithms , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[25]  Guido M. Cortelazzo,et al.  Time-of-Flight Cameras and Microsoft Kinect™ , 2012, Springer Briefs in Electrical and Computer Engineering.

[26]  Michael Werman,et al.  Fusing Time-of-Flight Depth and Color for Real-Time Segmentation and Tracking , 2009, Dyn3D.

[27]  Nassir Navab,et al.  Adaptive neighborhood selection for real-time surface normal estimation from organized point cloud data using integral images , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[28]  Derek Hoiem,et al.  Indoor Segmentation and Support Inference from RGBD Images , 2012, ECCV.

[29]  Alain Trémeau,et al.  Joint Color-Spatial-Directional Clustering and Region Merging (JCSD-RM) for Unsupervised RGB-D Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Jitendra Malik,et al.  Indoor Scene Understanding with RGB-D Images: Bottom-up Segmentation, Object Detection and Semantic Segmentation , 2015, International Journal of Computer Vision.

[31]  Guido M. Cortelazzo,et al.  A Probabilistic Approach to ToF and Stereo Data Fusion , 2010 .

[32]  Daniel Herrera C,et al.  Joint depth and color camera calibration with distortion correction. , 2012, IEEE transactions on pattern analysis and machine intelligence.

[33]  Charless C. Fowlkes,et al.  Contour Detection and Hierarchical Image Segmentation , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  Daniel Cohen-Or,et al.  Combining color and depth for enhanced image segmentation and retargeting , 2011, The Visual Computer.

[35]  Ludovico Minto,et al.  Time-of-Flight and Structured Light Depth Cameras , 2016, Springer International Publishing.

[36]  Ferran Marqués,et al.  Hierarchical fusion of color and depth information at partition level by cooperative region merging , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[37]  Richard Szeliski,et al.  Computer Vision - Algorithms and Applications , 2011, Texts in Computer Science.

[38]  Jitendra Malik,et al.  Normalized cuts and image segmentation , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[39]  Pietro Zanuttigh,et al.  Scene Segmentation Based on NURBS Surface Fitting Metrics , 2015, STAG.

[40]  Stefano Mattoccia,et al.  Scene Segmentation Assisted by Stereo Vision , 2011, 2011 International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission.

[41]  Jitendra Malik,et al.  Spectral grouping using the Nystrom method , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Kun Li,et al.  Graph-Based Segmentation for RGB-D Data Using 3-D Geometry Enhanced Superpixels , 2015, IEEE Transactions on Cybernetics.

[43]  Les A. Piegl,et al.  The NURBS book (2nd ed.) , 1997 .

[44]  Dorin Comaniciu,et al.  Mean Shift: A Robust Approach Toward Feature Space Analysis , 2002, IEEE Trans. Pattern Anal. Mach. Intell..