On the impact of the error measure selection in evaluating disparity maps

A quantitative evaluation methodology for disparity maps includes the selection of an error measure. Among existing measures, the percentage of bad matched pixels is commonly used. Nevertheless, it requires an error threshold. Thus, a score of zero bad matched pixels does not necessarily imply that a disparity map is free of errors. On the other hand, we have not found publications on the evaluation process where different error measures are applied. In this paper, error measures are characterised in order to provide the bases to select a measure during the evaluation process. An analysis of the impact on results of selecting different error measures on the evaluation of disparity maps is conducted based on the presented characterisation. The evaluation results showed that there is a lack of consistency on the results achieved by considering different error measures. It has an impact on interpreting the accuracy of stereo correspondence algorithms.

[1]  Ruigang Yang,et al.  Stereo Matching with Color-Weighted Correlation, Hierarchical Belief Propagation and Occlusion Handling , 2006, CVPR.

[2]  María P. Trujillo,et al.  A Non-linear Quantitative Evaluation Approach for Disparity Estimation - Pareto Dominance Applied in Stereo Vision , 2011, VISAPP.

[3]  Heiko Hirschmüller,et al.  Stereo camera based navigation of mobile robots on rough terrain , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[4]  Margaret Florian,et al.  An Evaluation Methodology for Stereo Correspondence Algorithms , 2012, VISAPP.

[5]  Dariu Gavrila,et al.  Real-time dense stereo for intelligent vehicles , 2006, IEEE Transactions on Intelligent Transportation Systems.

[6]  D. Nistér,et al.  Stereo Matching with Color-Weighted Correlation, Hierarchical Belief Propagation, and Occlusion Handling , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  D. Scharstein,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[8]  Xing Mei,et al.  Stereo Matching with Reliable Disparity Propagation , 2011, 2011 International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission.

[9]  Richard Szeliski,et al.  An Experimental Comparison of Stereo Algorithms , 1999, Workshop on Vision Algorithms.

[10]  Abdul Rehman,et al.  Reduced-reference SSIM estimation , 2010, 2010 IEEE International Conference on Image Processing.

[11]  In-So Kweon,et al.  Stereo Matching with the Distinctive Similarity Measure , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[12]  Gary B. Lamont,et al.  Considerations in engineering parallel multiobjective evolutionary algorithms , 2003, IEEE Trans. Evol. Comput..

[13]  Richard Szeliski,et al.  High-accuracy stereo depth maps using structured light , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[14]  Zhi-Gang Zheng,et al.  A region based stereo matching algorithm using cooperative optimization , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Federico Tombari,et al.  Stereo for robots: Quantitative evaluation of efficient and low-memory dense stereo algorithms , 2010, 2010 11th International Conference on Control Automation Robotics & Vision.

[16]  Xing Mei,et al.  On building an accurate stereo matching system on graphics hardware , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[17]  María P. Trujillo,et al.  A Measure for Accuracy Disparity Maps Evaluation , 2011, CIARP.

[18]  Radim Sára,et al.  Feasibility Boundary in Dense and Semi-Dense Stereo Matching , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Carsten Rother,et al.  PatchMatch Stereo - Stereo Matching with Slanted Support Windows , 2011, BMVC.

[20]  Heiko Hirschmüller,et al.  Evaluation of Stereo Matching Costs on Images with Radiometric Differences , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Nicolas Papadakis,et al.  Multi-label Depth Estimation for Graph Cuts Stereo Problems , 2010, Journal of Mathematical Imaging and Vision.

[22]  Ruigang Yang,et al.  Spatial-Depth Super Resolution for Range Images , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[23]  Alan C. Bovik,et al.  Range image quality assessment by Structural Similarity , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[24]  Jan-Michael Frahm,et al.  Variable baseline/resolution stereo , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[25]  Pushmeet Kohli,et al.  Object stereo — Joint stereo matching and object segmentation , 2011, CVPR 2011.

[26]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[27]  Yee-Hong Yang,et al.  Evaluation of constructable match cost measures for stereo correspondence using cluster ranking , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[28]  Margrit Gelautz,et al.  A layered stereo algorithm using image segmentation and global visibility constraints , 2004, 2004 International Conference on Image Processing, 2004. ICIP '04..

[29]  Vladimir Kolmogorov,et al.  Multi-camera Scene Reconstruction via Graph Cuts , 2002, ECCV.

[30]  Richard Szeliski,et al.  Prediction error as a quality metric for motion and stereo , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[31]  N. Grammalidis,et al.  3D content generation for autostereoscopic displays , 2009, 2009 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video.