Joint Color and Depth Segmentation based on Region Merging and Surface Fitting

The recent introduction of consumer depth cameras has opened the way to novel segmentation approaches exploiting depth data together with the color information. This paper proposes a region merging segmentation scheme that jointly exploits the two clues. Firstly a set of multi-dimensional vectors is built considering the 3D spatial position, the surface orientation and the color data associated to each scene sample. Normalized cuts spectral clustering is applied to the obtained vectors in order to over-segment the scene into a large number of small segments. Then an iterative merging procedure is used to recombine the segments into the regions corresponding to the various objects and surfaces. The proposed algorithm tries to combine close compatible segments and uses a NURBS surface fitting scheme on the considered segments in order to understand if the regions candidate for the merging correspond to a single surface. The comparison with state-of-the-art methods shows how the proposed method provides an accurate and reliable scene segmentation.

[1]  Alain Trémeau,et al.  Unsupervised RGB-D image segmentation using joint clustering and region merging , 2014, BMVC.

[2]  Jerry Potter.  Scene segmentation , 1975, SGAR.

[3]  Jitendra Malik,et al.  Perceptual Organization and Recognition of Indoor Scenes from RGB-D Images , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[4]  Pietro Zanuttigh,et al.  Scene segmentation from depth and color data driven by surface fitting , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[5]  Jitendra Malik,et al.  Normalized cuts and image segmentation , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[6]  Derek Hoiem,et al.  Indoor Segmentation and Support Inference from RGBD Images , 2012, ECCV.

[7]  Michael Felsberg,et al.  Channel Coding for Joint Colour and Depth Segmentation , 2011, DAGM-Symposium.

[8]  Stefano Mattoccia,et al.  Scene Segmentation Assisted by Stereo Vision , 2011, 2011 International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission.

[9]  Jitendra Malik,et al.  Spectral grouping using the Nystrom method , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Pietro Zanuttigh,et al.  Scene Segmentation Based on NURBS Surface Fitting Metrics , 2015, STAG.

[11]  Dieter Fox,et al.  RGB-(D) scene labeling: Features and algorithms , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[12]  Guido M. Cortelazzo,et al.  Time-of-Flight Cameras and Microsoft Kinect™ , 2012, Springer Briefs in Electrical and Computer Engineering.

[13]  Michael Werman,et al.  Fusing Time-of-Flight Depth and Color for Real-Time Segmentation and Tracking , 2009, Dyn3D.

[14]  Jitendra Malik,et al.  Indoor Scene Understanding with RGB-D Images: Bottom-up Segmentation, Object Detection and Semantic Segmentation , 2015, International Journal of Computer Vision.

[15]  Frank Dellaert,et al.  A Rao-Blackwellized MCMC algorithm for recovering piecewise planar 3D models from multiple view RGBD images , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[16]  Daniel P. Huttenlocher,et al.  Efficient Graph-Based Image Segmentation , 2004, International Journal of Computer Vision.

[17]  Frank Dellaert,et al.  Planar Segmentation of RGBD Images Using Fast Linear Fitting and Markov Chain Monte Carlo , 2012, 2012 Ninth Conference on Computer and Robot Vision.

[18]  Anthony Cowley,et al.  Parsing Indoor Scenes Using RGB-D Imagery , 2012, Robotics: Science and Systems.

[19]  Les A. Piegl,et al.  The NURBS book (2nd ed.) , 1997 .

[20]  Guido M. Cortelazzo,et al.  Fusion of Geometry and Color Information for Scene Segmentation , 2012, IEEE Journal of Selected Topics in Signal Processing.

[21]  Charless C. Fowlkes,et al.  Contour Detection and Hierarchical Image Segmentation , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.