Segmentation Based Depth Extraction for Stereo Image and Video Sequence

3D representation nowadays has attracted much more public attention than ever before. One of the most important techniques in this field is depth extraction. In this thesis, we first introduce a well-known stereo matching method using color segmentation and belief propagation, and make an implementation of this framework. The color-segmentation based stereo matching method performs well recently, since this method can keep the object boundaries accurate, which is very important to depth map. Based on the implemented framework of segmentation based stereo matching, we proposed a color segmentation based 2D-to-3D video conversion method using high quality motion information. In our proposed scheme, the original depth map is generated from motion parallax by optical flow calculation. After that we employ color segmentation and plane estimation to optimize the original depth map to get an improved depth map with sharp object boundaries. We also make some adjustments for optical flow calculation to improve its efficiency and accuracy. By using the motion vectors extracted from compressed video as initial values for optical flow calculation, the calculated motion vectors are more accurate within a shorter time compared with the same process without initial values. The experimental results shows that our proposed method indeed gives much more accurate depth maps with high quality edge information. Optical flow with initial values provides good original depth map, and color segmentation with plane estimation further improves the depth map by sharpening its boundaries.

[1]  Ju-Hyun Park,et al.  The virtual time domain depth estimation of stereo-scopic sequence using optical flow , 2002, 2002 IEEE Region 10 Conference on Computers, Communications, Control and Power Engineering. TENCOM '02. Proceedings..

[2]  Chengming Zou,et al.  Recovering Depth from a Single Image Using Dark Channel Prior , 2010, 2010 11th ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing.

[3]  Xiaoou Tang,et al.  Single Image Haze Removal Using Dark Channel Prior , 2011 .

[4]  R. Hariharan,et al.  Shape-From-Focus by Tensor Voting , 2012, IEEE Transactions on Image Processing.

[5]  Wei-Chih Chen,et al.  A 2D to 3D conversion scheme based on depth cues analysis for MPEG videos , 2010, 2010 IEEE International Conference on Multimedia and Expo.

[6]  Panos Nasiopoulos,et al.  Random forests-based 2D-to-3D video conversion , 2010, 2010 17th IEEE International Conference on Electronics, Circuits and Systems.

[7]  Yoonsik Choe,et al.  High-quality 2D to 3D video conversion based on robust MRF-based object tracking and reliable graph-cut-based contour refinement , 2011, ICTC 2011.

[8]  Ju Liu,et al.  Depth generation method for 2D to 3D conversion , 2011, 2011 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON).

[9]  Philippe Salembier,et al.  Occlusion-based depth ordering on monocular images with Binary Partition Tree , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[10]  Wenhui Zhou,et al.  Combining dark channel prior and color cues for road following in outdoor environments , 2010, 2010 IEEE International Conference on Image Processing.

[11]  Yong Ju Jung,et al.  A novel 2D-to-3D conversion technique based on relative height-depth cue , 2009, Electronic Imaging.

[12]  Chao Liu,et al.  Depth map estimation from motion for 2D to 3D conversion , 2012, 2012 IEEE International Conference on Electro/Information Technology.

[13]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[14]  Michael J. Black,et al.  Secrets of optical flow estimation and their principles , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[15]  Margrit Gelautz,et al.  Graph-based surface reconstruction from stereo pairs using image segmentation , 2005 .

[16]  Peter Lawrence,et al.  An Investigation of Methods for Determining Depth from Focus , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[17]  Tien-Ying Kuo,et al.  Depth estimation from a monocular view of the outdoors , 2011, IEEE Transactions on Consumer Electronics.

[18]  Zhaozheng Yin,et al.  Improving depth perception with motion parallax and its application in teleconferencing , 2009, 2009 IEEE International Workshop on Multimedia Signal Processing.

[19]  Dae-Woo Lee,et al.  3D depth estimation for target region using optical flow and mean-shift algorithm , 2008, 2008 International Conference on Control, Automation and Systems.

[20]  Dan Schonfeld,et al.  Virtual Focus and Depth Estimation From Defocused Video Sequences , 2010, IEEE Transactions on Image Processing.

[21]  Yuanyuan Ding,et al.  Multiperspective stereo matching and volumetric reconstruction , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[22]  Thomas Brox,et al.  High Accuracy Optical Flow Estimation Based on a Theory for Warping , 2004, ECCV.

[23]  Wen Gao,et al.  Visual pertinent 2D-to-3D video conversion by multi-cue fusion , 2011, 2011 18th IEEE International Conference on Image Processing.

[24]  Dorin Comaniciu,et al.  Mean Shift: A Robust Approach Toward Feature Space Analysis , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[25]  Filippo Speranza,et al.  Depth image based rendering for multiview stereoscopic displays: role of information at object boundaries , 2005, SPIE Optics East.

[26]  Zhaoyang Zhang,et al.  A New Improved Depth Estimation Algorithm Based on Gradient Feature for Preserving Consistency among Views , 2011, 2011 Third International Conference on Multimedia Information Networking and Security.

[27]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[28]  Larry D. Hostetler,et al.  The estimation of the gradient of a density function, with applications in pattern recognition , 1975, IEEE Trans. Inf. Theory.